rsync real-time remote synchronization

Posted by varun_146100 on Wed, 05 Jan 2022 11:58:28 +0100

What is rsync, its principle and preliminary construction have been introduced earlier. Here I will add the rest, real-time remote synchronization of rsync.

1, What is real-time synchronization

Real time synchronization uses the inotify notification interface to monitor various changes of the file system. The changes of the system include the following: file access, deletion, movement, modification, etc.

Using this mechanism, it is very convenient to realize the alarm and incremental backup of file changes, and respond to directories or files in time.

Combining inotify mechanism with rsync can realize triggered backup, that is, real-time synchronization. That is, as long as the document in the original location changes, the incremental backup operation is started immediately, otherwise it is in a silent waiting state. In this way, the delay and cycle density of backup according to fixed cycle are avoided

2, Construction process

1. Server

  1. Update rsync first

    yum install rsync -y
  2. Enter the configuration file for modification

    vim /etc/rsyncd.conf
  3. Meaning of each command line

    uid = nobody			#The default is nobody, which indicates the permissions of the shared directory that can be used and the user identity used
    gid = nobody			#The default is nobody, which shows the permissions of the shared directory that can be used and the user identity used
    use chroot = yes		#The user's permissions are limited according to the environment. The original permissions of the user will be confined in this directory, and the permissions of other directories will be degraded
    address =		#The listening address. The default is to fill in your own local IP address
    port 873			#The listening port is 873 by default
    log file = /var/log/rsyncd.log		#log file location
    pid file = /var/run/		#pid log file location
    hosts allow = 		#Allowed client addresses
    [wwwroot]				#Called synchronization module
    path = /var/www/html	#The path to the synchronization file
    comment = Document Root of	#You can modify it or not
    read only = yes			#Whether to read only. The default value is yes. You can directly default it
    dont compress = *.gz *.tgz *.zip *.z *.rar *.rpm *.deb *.bz2			#A format that no longer requires compression
    auth users = backuper		#Users logged in to this module
    secrets file = /etc/rsyncd_users.db		#Configure the password for the user accessing the module
  4. Enter the user password file to configure the password

    vim /etc/rsyncd_users.db		#Enter the configuration file and insert the following statement
    backuper:123123				#User name: password
  5. Give the user permission to execute the password file

    chmod 600 /etc/rsyncd_users.db
  6. Create shared directory

    mkdir -p /var/www/html		#create a file
    chmod +r /var/www/html/		#Give directory read permission
  7. Open service

    rsync --daemon			#Open service
    netstat -natp | grep rsync		#View port details
  8. Temporarily shut down the service first

    kill 3062		#Followed by my own process number

Create a file under / var/www/html /

2. Enter client

  1. Synchronize
rsync -avz backuper@ /lhk		

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-cX5G7b5T-1640832049263)(C:\Users \ Li Hongkun \ appdata \ roaming \ typora user images \ image-20211230084801205. PNG)]

As shown in the figure, the file synchronized to

  1. Face interaction synchronization mode
echo '123123' > /etc/rsyncd_users.db		#Write the password into the file with the same name as the synchronization source
chmod 600 /etc/rsyncd_users.db				#change permission
  1. Face interactive synchronization file
rsync -avz --password-file=/etc/rsyncd_users.db backuper@ /lhk

3. Real time synchronization construction

  1. Enter the synchronization source to configure parameters

    vim /etc/rsyncd.conf		#Enter the configuration file and close it read-only
    read only = no			#After closing, save and exit
    kill `cat /var/run/`		#Kill the process first
     Restart again
    rsync --daemon			#start-up
  2. Go to the initiator and set the configuration parameters

    cat /proc/sys/fs/inotify/max_user_instances 
    128					#The default value is 128
    cat /proc/sys/fs/inotify/max_user_watches
    8192					#The default value is 8192
    cat /proc/sys/fs/inotify/max_queued_events
    16384					#The default value is 16384
    #Enter the above three commands to view the inotify kernel parameters
    #Write kernel parameters to a file
    vim /etc/sysctl.conf
    fs.inotify.max_queued_events = 16384		
    #Monitors the size of the transaction queue
    fs.inotify.max_user_instances = 128
    #Maximum number of monitored instances
    fs.inotify.max_user_watches = 8192
    #How many files can each instance monitor at most
    #After insertion, save and exit to view the addition
    sysctl -p			#View added information
    fs.inotify.max_queued_events = 16384
    fs.inotify.max_user_instances = 128
    fs.inotify.max_user_watches = 8192
     The above is the output result. Compare it with the value you entered. Is there any error
  3. Initiator installation inotify

    yum install -y gcc gcc-c++			#Install dependent environment first
    tar zxvf inotify-tools-3.14.tar.gz	#Unzip the installation package
    cd inotify-tools-3.14/				#Enter file
    ./configure							#Compile first
    make && make install				#make installation file
  4. Open a new initiator (this one is OK, just open a new connection)

    inotifywait -mrq -e modify,create,move,delete /lhk
    #Initiate monitoring -e, real-time monitoring, followed by the type of monitoring. Listening directory
  5. Test

    touch 2.2			#To create a file
    [root@server ~]# inotifywait -mrq -e modify,create,move,delete /lhk
    /lhk/ CREATE 2.2
    #After you go to the listening machine, you will find that the operation steps of creating a file have appeared
    /lhk/ CREATE .2.2.swp
    /lhk/ CREATE .2.2.swx
    /lhk/ DELETE .2.2.swx
    /lhk/ DELETE .2.2.swp
    /lhk/ CREATE .2.2.swp
    /lhk/ MODIFY .2.2.swp
    /lhk/ MODIFY .2.2.swp
    #These log information will be generated when accessing the file
    /lhk/ CREATE 4913
    /lhk/ DELETE 4913
    /lhk/ MOVED_FROM 2.2
    /lhk/ MOVED_TO 2.2~
    /lhk/ CREATE 2.2
    /lhk/ MODIFY 2.2
    /lhk/ MODIFY .2.2.swp
    /lhk/ DELETE 2.2~
    /lhk/ DELETE .2.2.swp
     These are the log information generated after modifying the file
  6. Create synchronization script

vim /opt/

INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /lhk/"
#Functions for listening
RSYNC_CMD="rsync -apzH --delete --password-file=/etc/rsyncd_users.db /lhk/ backuper@"
#Functions for synchronization

#Transfer the value monitored by the parameter to the later, and traverse the following types of logs
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
   #After the change, it will detect whether synchronization is in progress. If synchronization is in progress, no operation will be performed. If no operation is performed, the following parameters will be executed.
#       echo "${FILE} was rsynced" >>/opt/inotify_rsync.log

  1. Modify file permissions

    chmod +x /opt/		#Add execution permissions to script files
    chmod +x /etc/rc.d/rc.local		#Boot file, add execution permission
    echo '/opt/' >> /etc/rc.d/rc.local	#Add to boot file
    sh -x /opt/			#Execute script
  2. Test after opening

    As above, open another connection to the monitoring directory to create a new file
    touch 1.1
     After the file is created, the synchronization operation is displayed.
    + rsync -apzH --delete --password-file=/etc/rsyncd_users.db /lhk/ backuper@
    ++ wc -l
    ++ pgrep rsync
    + '[' 0 -le 0 ']'
    + rsync -apzH --delete --password-file=/etc/rsyncd_users.db /lhk/ backuper@
    To the script, fill in the automatic synchronization server to view
    [root@rsync html]# ls
    [root@rsync html]#
    #As you can see, it is now an empty directory
     Then go back to the server that started the script to create the service
    [root@server lhk]# touch 1.1
    [root@server lhk]# ls
    #After the creation is completed, wait a moment and return to the server specified in the script
    [root@rsync html]# ls
    #This server has not performed any artificial creation operation, which means that the synchronization has been successfully performed


rsync is a very practical module with a wide range of application scenarios. It can be used to remotely synchronize files, delete their own or target files based on synchronization, so as to achieve fast and safe deletion, as well as data migration.

Topics: rsync