Linux sync folders between computers

10 Practical Examples of Rsync Command in Linux

Rsync (Remote Sync) is the most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems.

With the help of the rsync command, you can copy and synchronize your data remotely and locally across directories, disks, and networks, perform data backups, and mirror between two Linux machines.

Rsync Local and Remote File Synchronization

This article explains 10 basic and advanced usage of the rsync command to transfer your files remotely and locally in Linux-based machines. You don’t need to be a root user to run the rsync command.

Some Advantages and Features of Rsync Command

• It efficiently copies and sync files to or from a remote system.
• Supports copying links, devices, owners, groups, and permissions.
• It’s faster than scp (Secure Copy) because rsync uses a remote-update protocol which allows transferring just the differences between two sets of files. The first time, it copies the whole content of a file or a directory from source to destination but from next time, it copies only the changed blocks and bytes to the destination.
• Rsync consumes less bandwidth utilization as it uses compression and decompression method while sending and receiving data on both ends.

The basic syntax of the rsync command

Some common options used with rsync commands

• -v : verbose
• -r : copies data recursively (but don’t preserve timestamps and permission while transferring data.
• -a : archive mode, which allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships, and timestamps.
• -z : compress file data.
• -h : human-readable, output numbers in a human-readable format.

Install Rsync in Linux System

We can install the rsync package with the help of the following command in your Linux distribution.

1. Copy/Sync Files and Directory Locally

Copy/Sync a File on a Local Computer

The following command will sync a single file on a local machine from one location to another location. Here in this example, a file name backup.tar needs to be copied or synced to /tmp/backups/ folder.

In the above example, you can see that if the destination is not already existed rsync will create a directory automatically for the destination.

Rsync Local Files

Copy/Sync a Directory on Local Computer

The following command will transfer or sync all the files from one directory to a different directory in the same machine. Here in this example, /root/rpmpkgs contains some rpm package files and you want that directory to be copied inside /tmp/backups/ folder.

Rsync Local Directory

2. Copy/Sync Files and Directory to or From a Server

Copy a Directory from Local Server to a Remote Server

This command will sync a directory from a local machine to a remote machine. For example, there is a folder in your local computer “rpmpkgs” that contains some RPM packages and you want that local directory’s content sends to a remote server, you can use the following command.

Rsync Directory Remote System

Copy/Sync a Remote Directory to a Local Machine

This command will help you sync a remote directory to a local directory. Here in this example, a directory /root/rpmpkgs which is on a remote server is being copied in your local computer in /tmp/myrpms.

Rsync Remote Directory to Local

3. Rsync Over SSH

With rsync, we can use SSH (Secure Shell) for data transfer, using SSH protocol while transferring our data you can be ensured that your data is being transferred in a secured connection with encryption so that nobody can read your data while it is being transferred over the wire on the internet.

Also when we use rsync we need to provide the user/root password to accomplish that particular task, so using the SSH option will send your logins in an encrypted manner so that your password will be safe.

Copy a File from a Remote Server to a Local Server with SSH

To specify a protocol with rsync you need to give the “-e” option with the protocol name you want to use. Here in this example, We will be using the “ssh” with the “-e” option and perform data transfer.

Rsync Copy Remote File to Local

Copy a File from a Local Server to a Remote Server with SSH

4. Show Progress While Transferring Data with rsync

To show the progress while transferring the data from one machine to a different machine, we can use the ‘–progress’ option. It displays the files and the time remaining to complete the transfer.

Rsync Progress While Copying Files

5. Use of –include and –exclude Options

These two options allow us to include and exclude files by specifying parameters with these option helps us to specify those files or directories which you want to include in your sync and exclude files and folders with you don’t want to be transferred.

Here in this example, the rsync command will include those files and directory only which starts with ‘R’ and exclude all other files and directory.

Rsync Include and Exclude Files

6. Use of –delete Option

If a file or directory does not exist at the source, but already exists at the destination, you might want to delete that existing file/directory at the target while syncing.

We can use the ‘–delete‘ option to delete files that are not there in the source directory.

Source and target are in sync. Now create a new file test.txt at the target.

Target has the new file called test.txt, when synchronizing with the source with the ‘–delete‘ option, it removed the file test.txt.

Rsync Delete Option

7. Set the Max Size of Files to be Transferred

You can specify the Max file size to be transferred or sync. You can do it with the “–max-size” option. Here in this example, the Max file size is 200k, so this command will transfer only those files which are equal to or smaller than 200k.

Rsync Set Max File Transfer Size

8. Automatically Delete source Files After Successful Transfer

Now, suppose you have the main web server and a data backup server, you created a daily backup and synced it with your backup server, now you don’t want to keep that local copy of backup in your web server.

So, will you wait for the transfer to complete and then delete that local backup file manually? Of Course NO. This automatic deletion can be done using the ‘–remove-source-files‘ option.

Rsync Delete Source File After Transfer

9. Do a Dry Run with rsync

If you are a newbie using rsync and don’t know what exactly your command going to do. Rsync could really mess up the things in your destination folder and then doing an undo can be a tedious job.

Use of this option will not make any changes to the files and shows the output of the command, if the output shows exactly the same you want to do then you can remove the ‘–dry-run‘ option from your command and run on the terminal.

Rsync Dry Run

10. Rsync Set Bandwidth Limit and Transfer File

You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ option. This option helps us to limit I/O bandwidth.

Also, by default rsync syncs changed blocks and bytes only, if you want explicitly want to sync the whole file then you use the ‘-W‘ option with it.

That’s all with rsync now, you can see man pages for more options. Stay connected with Tecmint for more exciting and interesting tutorials in the future. Do leave your comments and suggestions.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник

How to Synchronize Directories Using Lsyncd in Linux

In this article, I’ll explain how to install Lsyncd(Live Syncing Mirror Daemon) on Linux systems and how to synchronize remote and local directories in Linux. Lsyncd is a light-weight live mirror solution that is comparatively easy to install without hampering existing local filesystem performance.

It is very useful to keep track of any data modification and sync those data between the directories which are frequently updated with new contents. By default, it is rsync only.

All custom configuration files are written in Lua language, this way powerful, flexible and simple configuration can be obtained. Lsyncd 2.2.1 requires rsync 3.1 on all source and target machines.

Install Lsyncd on RHEL/CentOS 7

In order to enable lsyncd on a CentOS 7.5 system, we will need to enable EPEL repository. You can simply run this command to install it.

You can confirm the installed version by running this command:

Lsyncd Configuration

Lsyncd configuration file is created automatically at /etc/lsyncd.conf on a RHEL/CentOS 7.5 system. By default, its contents look as below:

We need to modify this configuration file as per our purpose. As mentioned in this configuration file, all example scripts are available in the following location: /usr/share/doc/lsyncd-2.2.2/examples/

All these files are lsyncd example configuration files. Out of these files, we are explaining more on the usage of these files namely lrsync.lua and lrsyncssh.lua here. Let’s see those example configuration files below:

Sample configuration for local sync:

Sample configuration for remote sync:

To retain this sync process instead of running a cron job every X minutes, lsyncd uses linux kernel hooks to get notifications when any file within a directory has been changed. By default, it queues up any sync commands in 20 seconds. we can even modify this time interval as required with the —delay option along with your sync command.

During the use of default.rsyncssh sync command it’s always recommended to use —delete option to prevent missing of files from the destination folders. After creating or modifying the lsyncd config file, you must restart the lsyncd process.

Since Lsyncd uses rsync tool to copy, move and delete files from the source to the destination. We can make use of the rsync switches to prevent making unnecessary duplicates on the destination and smoothen this process. Some of the important rsync options are explained below:

—delete: This option ensures that any files in the remote directory that aren’t in the source directory are deleted.
—times: This option is fairly important if you are going to be running this script over and over again, as it will keep the times between the two files in sync.
—force: This option allows the deletion of a non-empty directory in order to be replaced by an empty directory.
—links: This option is used to copy symlinks as symlinks.
—progress2: It results in showing the overall progress of the entire transfer, not just of the single file that is being copied.
—dry-run: This option do a trial run without actually performing any deletions or transfers but telling you what it will do. I highly recommend using this option the first time you run any rsync command after writing it.
—owner: ensures that owner user of the file is kept (not the permission level for the owner).
—group: ensures that group user of the file is kept (not the permission level for the group).
—perms: preserve permissions.
—sparse: ensures that sparse image files are transferred efficiently.

Synchronize local directories

Now let’s see how to synchronize two local folders using lsyncd. We can create a source folder namely SRC_DIR and a target folder namely DEST_DIR to explain this process more vividly. Let’s create the folders and add some files to the source directory to perform the synchronization.

Next, you can create the lsyncd log files and status files to keep track of the process. These steps are optional. But I would recommend maintaining log files for all our tasks.

Now we need to modify our lsyncd configuration file to perform this local rsync. As we discussed before, the default lsyncd configuration file for a CentOS system is /etc/lsyncd.conf . We need to update these configuration files with our source and target directories and our log files.

You can replace the source and target directories path with your own values. Save and close the configuration file. Once it’s done, restart and enable lsyncd service.

Now compare both source and target directories contents to confirm its working.

Hurray! The contents of source directory SRC_DIR have been successfully synchronized to target directory.

Furthermore, you can review the log and status files to verify the status of the replication to confirm whether it’s completed or not.

Synchronize multiple local folders

In order to sync multiple folders to one or more target directories, we will need to update the configuration file with more sync command statements with our required source and target directories.

For example, please see my Lsyncd configuration file /etc/lsyncd.conf to sync two of my folders namely /root/SRC and /etc/nginx to a target directory located at /backup below:

After updating the configuration file with the required changes, you can restart the lsyncd process to initiate the rsync process. Once done, you can verify the log files to confirm its status.

Similarly, you can use this «default-rsync» sync command statements multiple times when you want to sync same source directories to multiple targets or multiple source directories to the same target directory.

Synchronize to remote directories

In order to initiate remote directory synchronization, we will need to set up passwordless SSH login. This will help Lsyncd to automatically replicate the contents of the local directory to a remote directory without user intervention. Since we are doing synchronization across two servers. we can take a source server and a target server to explain this process more clearly. Please see the SRC and DEST server IPs below:

Step 1) Creating SSH keys on Source server for Passwordless login

We need to create an SSH key for the source server and copy the public key over to the target server to enhance the SSH connection between the servers during the account sync. You can generate the RSA keys using the command below:

Step 2) Copying the public keys to the target server

Now copy the public keys over to the target server to enable password-less logins.

Here, 45.33.121.82 is the destination server IP. Now we will be able to connect to our remote destination server from the source without any passwords.

Step 3) Creating a target directory for remote synchronization

Let’s create a target directory namely Remote_Dir on the destination server. Once it’s done you can logout the target server.

Step 4) Modify the Lsyncd configuration file to enable Remote sync

You can back up the current lsyncd configuration file at /etc/lsyncd.conf and copy the sample lsyncd configuration file for remote sync located at /usr/share/doc/lsyncd-2.2.2/examples/lrsyncssh.lua to the main lsyncd configuration file /etc/lsyncd.conf . Once it’s done you can edit the source directory, host and target directory accordingly in the configuration file. Please see my lsyncd configuration file for remote sync as per my setup below:

Copy the sample configuration to retain its proper syntax.

Edit the configuration file accordingly.

Step 5) Restart the lsyncd service on the source

Once the configuration file is proper edited and saved, you can restart the lsyncd service to make these changes effective. Now login to the remote server (DEST_SERVER) and confirm the contents on the remote Rsync directory namely /root/Remote_Dir .

Howdy! you can see all the files from the source directory on the local system to be replicated on the target directory in the destination server. You can even verify the successful completion of the rsync process viewing the lsyncd log files on the source.

# tail /var/log/lsyncd/lsyncd.log
Sun Aug 5 07:04:42 2018 Normal: — Startup —
Sun Aug 5 07:04:42 2018 Normal: recursive startup rsync: /root/SRC_DIR/ -> 45.33.121.82:/root/Remote_Dir/
Sun Aug 5 07:04:43 2018 Normal: Startup of «/root/SRC_DIR/» finished: 0
Sun Aug 5 07:13:48 2018 Normal: Rsyncing list
/
Sun Aug 5 07:13:49 2018 Normal: Finished (list): 0

Synchronize across multiple remote servers

Previously, we described how to sync a folder across a remote server. Similarly, you can use the default.rsyncssh command statements multiple times to sync the required source folders over multiple remote destination folders. But we need to make sure to enable password-less SSH login to replicate the contents of the local directories over to multiple remote directories without user intervention.

I’ll explain this process with a simple example of transferring the /etc/nginx folder from my SRC server to the remote servers DEST 1 and DEST 2.

The first step to this process will be generating an RSA key on my source server and copying its public keys over to my remote servers DEST 1 and DEST 2 as explained above. We need to repeat the same procedure from Step 1 to Step 3 for the two remote servers DEST 1 and DEST 2 to ensure password-less SSH login.

Secondly, we need to modify the lsyncd configuration file with multiple default.rsyncssh command statements with the required source, host and target folders. Please see my lsyncd configuration file for this sync process below:

After making these configuration changes, you can restart the lsyncd service to start this sync process. Once, it’s done you can verify the log files to confirm its status.

Install Lsyncd on Debian/Ubuntu 18.04

On Debian and its derivatives like Ubuntu, Linux Mint etc, you can install lsyncd using the simple apt command as below:

You can confirm the installed lsyncd version using this command:

Configuration on Debian/Ubuntu

On Ubuntu based systems, it won’t provide any default lsyncd configuration file. It is recommended to manually create those configuration files as per our purpose. We can get sample configuration files at the location. /usr/share/doc/lsyncd/examples/ These sample example configuration files provide us with a basic idea of what/how it does the synchronization.

All these configuration files are written in Lua programming language. Please see the lsyncd sample configuration for a simple local rsync below:

This procedure is exactly the same comparing to CentOS systems described above. Firstly, we will need to set up passwordless SSH login generating the RSA keys on the source server. This will ensure Lsyncd to automatically replicate the contents to a remote directory without user intervention. The key points to remember during remote sync comparing local sync is that we will need to change default.rsync to default.rsyncssh to enable rsync over ssh, and we should replace the «target» variable with the «host» and «targeted» variables. In addition, we will need to maintain the configuration file location as /etc/lsyncd/lsyncd.conf.lua on Ubuntu/Debian systems. Rest of the lsyncd configuration procedures are the same in all Linux Sytems.

Read Also:

I hope this article is useful and informative to you! Please post your valuable comments and suggestions on this.

Источник