Lesson 8.4: Rsync - File Synchronization

Rsync is the power tool for file synchronization. It's faster and smarter than SCP, only transferring what's changed. Perfect for backups, deployments, and keeping directories in sync.

Think of it like this: While SCP copies everything every time (like photocopying all pages), rsync is smart enough to only copy the pages that changed. Much faster for updates!

What is Rsync?

Rsync (remote sync) is an intelligent file synchronization tool:

Efficient: Only transfers differences between files
Fast: Delta-transfer algorithm saves time and bandwidth
Versatile: Works locally and over SSH
Reliable: Preserves permissions, ownership, timestamps
Smart: Resume interrupted transfers, skip unchanged files

Why Use Rsync Over SCP?

Scenario	SCP	Rsync
First-time copy	Fast	Similar speed
Update existing files	Copies everything	Only copies changes!
Large directories	Slow every time	Fast updates
Interrupted transfer	Start over	Resume where it stopped

Basic Rsync Syntax

The general format is similar to SCP:

rsync [options] source destination

Remote locations use the same SSH format:

username@hostname:/path/to/directory

The Magic Three: -avz

Most rsync commands start with these three essential flags:

rsync -avz source/ destination/

What do these flags mean?

-a (archive) - Preserve permissions, timestamps, symbolic links, etc.
-v (verbose) - Show what's being transferred
-z (compress) - Compress data during transfer

Remember -avz: These three flags together are so common, they're practically rsync's default. Start every rsync command with them!

Local Rsync (Practice)

Before using rsync remotely, understand it locally:

# Copy a directory locally
rsync -av source-directory/ destination-directory/

# Backup your documents
rsync -av ~/Documents/ ~/Backup/Documents/

# Keep two local directories synchronized
rsync -av ~/my-website/ ~/website-backup/

Understanding Trailing Slashes

Trailing slashes matter in rsync - they change behavior:

# WITH trailing slash: copies CONTENTS of source-dir
rsync -av source-dir/ destination/
# Result: destination/ contains the files from source-dir

# WITHOUT trailing slash: copies the DIRECTORY itself
rsync -av source-dir destination/
# Result: destination/source-dir/ contains the files

Example:
rsync -av website/ /var/www/html/ → copies index.html, about.html directly
rsync -av website /var/www/html/ → creates /var/www/html/website/ first

Uploading to Remote Server

Upload files from local machine to remote server:

Basic Upload

# Upload directory to remote server
rsync -avz my-website/ valente@myserver.com:/var/www/html/

# Upload with progress indicator
rsync -avz --progress my-website/ valente@myserver.com:/var/www/html/

# Upload specific files
rsync -avz *.html valente@myserver.com:/var/www/html/

Deploy Website Example

# First deployment (copies everything)
rsync -avz ~/my-website/ valente@myserver.com:/var/www/html/

# Later updates (only copies changes!)
# Much faster the second time
rsync -avz ~/my-website/ valente@myserver.com:/var/www/html/

Downloading from Remote Server

Download files from remote server to local machine:

# Download remote directory
rsync -avz valente@myserver.com:/var/www/html/ ~/website-backup/

# Download specific files
rsync -avz valente@myserver.com:/var/log/nginx/*.log ~/logs/

# Backup remote directory
rsync -avz valente@myserver.com:/home/valente/projects/ ~/backups/

The --delete Flag (Mirror Sync)

Make destination exactly match source by deleting extra files:

# Make remote directory exactly match local
rsync -avz --delete my-website/ valente@myserver.com:/var/www/html/

# If you deleted a file locally, it gets deleted remotely too

Use --delete Carefully! This deletes files in the destination that don't exist in source. Always test with --dry-run first (see below)!

Dry Run: Test Before You Transfer

The --dry-run flag shows what would happen without actually doing it:

# See what would be transferred
rsync -avz --dry-run my-website/ valente@myserver.com:/var/www/html/

# Test a --delete operation safely
rsync -avz --delete --dry-run my-website/ valente@myserver.com:/var/www/html/

# Combine with -v to see detailed output
rsync -avzn my-website/ valente@myserver.com:/var/www/html/
# Note: -n is shorthand for --dry-run

Best Practice: Always run with --dry-run first, especially when using --delete. Check what will happen, then run again without --dry-run to actually transfer.

Useful Rsync Options

Show Progress (--progress)

# See transfer progress for each file
rsync -avz --progress my-website/ valente@myserver.com:/var/www/html/

sending incremental file list index.html 2,345 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=3/5) style.css 5,678 100% 5.41MB/s 0:00:00 (xfr#2, to-chk=2/5)

Exclude Files (--exclude)

# Exclude specific files or patterns
rsync -avz --exclude='*.log' my-website/ valente@myserver.com:/var/www/html/

# Exclude multiple patterns
rsync -avz --exclude='*.log' --exclude='*.tmp' \
    my-website/ valente@myserver.com:/var/www/html/

# Exclude directories
rsync -avz --exclude='node_modules' --exclude='.git' \
    my-project/ valente@myserver.com:/home/valente/projects/

Include Only Specific Files (--include)

# Only sync HTML files
rsync -avz --include='*.html' --exclude='*' \
    my-website/ valente@myserver.com:/var/www/html/

# Include pattern, exclude everything else
rsync -avz --include='*.jpg' --include='*.png' --exclude='*' \
    images/ valente@myserver.com:/var/www/html/images/

Partial Transfers (--partial)

# Keep partially transferred files if interrupted
rsync -avz --partial large-files/ valente@myserver.com:/backup/

# Resume interrupted transfer later
rsync -avz --partial large-files/ valente@myserver.com:/backup/

# Combine partial and progress
rsync -avz --partial --progress large-files/ valente@myserver.com:/backup/

# Or use -P (shorthand for --partial --progress)
rsync -avzP large-files/ valente@myserver.com:/backup/

Bandwidth Limit (--bwlimit)

# Limit bandwidth to 1000 KB/s
rsync -avz --bwlimit=1000 large-files/ valente@myserver.com:/backup/

# Useful on shared connections to avoid hogging bandwidth

Custom SSH Port (-e)

# Use non-standard SSH port
rsync -avz -e "ssh -p 2222" my-website/ valente@myserver.com:/var/www/html/

# Use specific SSH key
rsync -avz -e "ssh -i ~/.ssh/work_key" \
    my-website/ valente@myserver.com:/var/www/html/

# Combine port and key
rsync -avz -e "ssh -p 2222 -i ~/.ssh/work_key" \
    my-website/ valente@myserver.com:/var/www/html/

Real-World Use Cases

Website Deployment Script

#!/bin/bash
# deploy-website.sh

echo "Deploying website to production..."

# Test first
rsync -avz --dry-run --delete \
    ~/my-website/ valente@myserver.com:/var/www/html/

read -p "Proceed with deployment? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    # Actually deploy
    rsync -avz --delete --progress \
        ~/my-website/ valente@myserver.com:/var/www/html/
    echo "✓ Deployment complete!"
else
    echo "✗ Deployment cancelled"
fi

Automated Backup

#!/bin/bash
# backup-website.sh

BACKUP_DIR=~/backups/website-$(date +%Y-%m-%d)

echo "Backing up website to $BACKUP_DIR..."

rsync -avz --progress \
    valente@myserver.com:/var/www/html/ \
    "$BACKUP_DIR/"

echo "✓ Backup complete: $BACKUP_DIR"

Mirror Two Directories

# Keep two directories perfectly synchronized
rsync -avz --delete \
    ~/my-website/ \
    valente@myserver.com:/var/www/html/

# Any files deleted locally are deleted remotely
# Any new files locally are copied remotely
# Changed files are updated

Sync Only Newer Files

# Only upload files that are newer locally
rsync -avzu my-website/ valente@myserver.com:/var/www/html/

# -u (update) skips files that are newer on destination

Common Rsync Patterns

Exclude Common Files

# Exclude development files
rsync -avz \
    --exclude='.git' \
    --exclude='node_modules' \
    --exclude='*.log' \
    --exclude='.DS_Store' \
    --exclude='*.tmp' \
    my-project/ valente@myserver.com:/var/www/project/

Exclude Using File

# Create exclude file
cat > rsync-exclude.txt << EOF
.git
node_modules
*.log
.DS_Store
*.tmp
.env
EOF

# Use exclude file
rsync -avz --exclude-from=rsync-exclude.txt \
    my-project/ valente@myserver.com:/var/www/project/

Rsync Between Two Remote Servers

Copy directly from one remote server to another:

# This routes through your computer (slower)
rsync -avz \
    valente@server1.com:/var/www/html/ \
    valente@server2.com:/backup/

# For faster server-to-server transfer, SSH to one server first
ssh valente@server1.com
rsync -avz /var/www/html/ valente@server2.com:/backup/

Troubleshooting Rsync

Why Is Nothing Transferring?

Rsync skips unchanged files. Use -v to see what it's checking:

# Verbose output shows what's being checked
rsync -avz my-website/ valente@myserver.com:/var/www/html/

# If files haven't changed, you'll see:
# sending incremental file list
# sent 85 bytes  received 12 bytes  64.67 bytes/sec
# total size is 42,345  speedup is 436.44

Permission Denied

# You don't have write permission in destination
# Solution: Rsync to a directory you own, then sudo mv

rsync -avz my-website/ valente@myserver.com:/home/valente/temp/
ssh valente@myserver.com
sudo mv /home/valente/temp/* /var/www/html/

Verify Transfer with --checksum

# Force verification of every file (slower but thorough)
rsync -avz --checksum my-website/ valente@myserver.com:/var/www/html/

Quick Reference

# Basic sync
rsync -avz source/ user@host:/destination/

# Sync with progress
rsync -avzP source/ user@host:/destination/

# Mirror (delete extra files)
rsync -avz --delete source/ user@host:/destination/

# Dry run first
rsync -avzn source/ user@host:/destination/

# Exclude patterns
rsync -avz --exclude='*.log' source/ user@host:/destination/

# Custom SSH port
rsync -avz -e "ssh -p 2222" source/ user@host:/destination/

# Download from remote
rsync -avz user@host:/source/ ./local-destination/

# Only update newer files
rsync -avzu source/ user@host:/destination/

# Show details and progress
rsync -avz --progress --stats source/ user@host:/destination/

# Common flags explained:
# -a: archive mode (preserve everything)
# -v: verbose (show what's happening)
# -z: compress during transfer
# -P: partial transfer + progress (-P = --partial --progress)
# -n: dry run (--dry-run)
# -u: update only (skip newer files on destination)
# -e: specify remote shell options

Practice Exercise

Master rsync with these tasks:

Sync your nginx website to a remote server using rsync -avz
Make a change locally and sync again - notice it's much faster!
Use --dry-run to preview a sync operation
Practice excluding files with --exclude
Create a backup script using rsync
Test mirror sync with --delete (carefully!)
Set up automated daily backups using rsync

Pro Tip: Create a deploy.sh script with your rsync command. Then deploying is just ./deploy.sh instead of typing a long command every time!

Key Takeaways

Rsync only transfers differences, making it faster than SCP for updates
Start with -avz for most operations
Trailing slashes matter: dir/ vs dir
Always use --dry-run first, especially with --delete
--exclude patterns keep unwanted files from syncing
-P shows progress and enables partial transfers
Perfect for deployments, backups, and keeping directories in sync

Next Up: Put everything together! The final lesson provides practical exercises combining SSH, SCP, and rsync to deploy and manage real websites.