rsync synchronization tool

Posted by lilleman on Mon, 03 Jan 2022 23:11:04 +0100

rsync synchronization tool

  • Rsync (remote synchronization) is an open source, fast, multi-functional and excellent tool for full and incremental local or remote data backup

Introduction to rsync

  • rsync can be seen from the name of the software. rsync has the functions of fast data replication, synchronous mirroring and remote backup between local and remote hosts. This function is similar to the scp command with ssh, but better than the scp command. scp is a full copy every time, while rsync can be copied incrementally. Of course, rsync can also replicate data between different partitions or directories of the local host in full and incremental amounts, which is similar to the cp command. But it is also better than the cp command. cp is a full copy every time, while rsync can copy incrementally.
rsync Official address:
rsync Listening port: 873
rsync Operation mode: C/S   client/server

rsync It is called remote synchronization for short. It can realize data synchronization between different hosts, and also supports full and incremental synchronization

rsync feature

  • 1) Support copying special files, such as linked files, devices, etc.
  • 2) It can exclude the synchronization of specified files or directories, which is equivalent to the exclusion function of the packaging command tar.
  • 3) You can keep all attributes of the original file or directory, such as permission, time, soft and hard links, owner and group, unchanged – p.
  • 4) Incremental synchronization can be realized, that is, only the changed data can be synchronized, so the data transmission efficiency is very high (tar-N).
  • 5) You can use rcp, rsh, ssh and other methods to transfer files (rsync itself does not encrypt data).
  • 6) You can transfer files and data (server and client) through socket (process mode).
  • 7) Support anonymous live authentication (without system users) process mode transmission, which can realize convenient and safe data backup and mirroring.

rsync application scenario

Full backup
 Incremental backup

Data synchronization between two servers.
Synchronize all client server data to the backup server, and the production scenario cluster architecture server backup scheme.
rsync combination inotify Function to do real-time data synchronization.

Transmission mode of rsync

push PUSH:
  The client pushes data from the local to the server(rsync The server actively pushes data to other hosts. The server overhead is large, which is suitable for the situation with few back-end servers)
pull PULL:
  The client pulls data from the server to the local server(Client initiative rsync Server pull data)

rsync transport mode

1.Local mode (similar to cp,Push and pull are not supported, but simply copy)
2.Remote mode (similar to scp,And different from scp),scp Only full backup is supported, rsync Support incremental backup and differential backup
3.Daemon( socket)Mode (client and server)

rsync usage

-a   #Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L
-v   #Detailed mode output, print rate, number of files, etc
      [root@m01 ~]# rsync -v ./2.txt  root@
-z   #Compression during transmission to improve efficiency
      [root@m01 ~]# rsync -vz ./2.txt  root@
-r   #Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same.
      [root@m01 ~]# rsync -vzr ./a  root@
-t   #Keep file time information
      [root@m01 ~]# rsync -vzrt ./a/b/c/2.txt  root@
-o   #Keep file master information
-g   #Keep file group information
      [root@m01 ~]# rsync -vzrtgo  ./a/b/c/2.txt  root@
-p   #Keep file permissions
      [root@m01 ~]# rsync -vzrtgop  ./a/b/c/2.txt  root@
-l   #Keep soft connection
      [root@m01 ~]# rsync -vzrtgopl  ./*  root@
-P   #Display information such as synchronization process and transmission progress
      [root@m01 ~]# rsync -vzrtgoplP  ./*  root@
-D   #Keep device file information
      [root@m01 dev]# rsync -vzrtgDopl /dev/tty1   root@
-L   #Keep the target file pointed to by the soft connection
-e   #Using the channel protocol, specify the shell program that replaces rsh

--append          # The specified file continues to be transferred where the last transfer was interrupted
--append-verify   # Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different)

--exclude=PATTERN   # Specifies to exclude files that do not need to be transferred
       [root@m01 ~]# rsync -avzP --append-verify --exclude=2.txt  ./* root@

--exclude-from=file # Exclude as specified in the file
       [root@m01 ~]# rsync -avzP --append-verify --exclude-from=/tmp/exclude.txt  ./* root@

--bwlimit=100       # Speed limit transmission (unit: MB)
       [root@m01 ~]# rsync -avzP --append-verify --bwlimit=10  ./* root@

--delete            # Keep the target directory and source directory data consistent

--password-file=xxx # Use password file

--port              # Specify port transport

rsync daemon mode

1. Server

 [root@backup ~]# yum install -y rsync

2,Modify profile
 [root@m01 ~]# vim /etc/rsyncd.conf 
  uid = rsync
  gid = rsync
  port = 873
  fake super = yes
  use chroot = no
  max connections = 200
  timeout = 600
  ignore errors
  read only = false
  list = false
  auth users = rsync_backup
  secrets file = /etc/rsync.passwd
  log file = /var/log/rsyncd.log
  comment = welcome to backup!
  path = /backup
  comment = welcome to linux!

3,Create system user
 [root@backup opt]# groupadd rsync -g 666
 [root@backup opt]# useradd rsync -u 666 -g 666 -M -s /sbin/nologin -r

4,Create password file
 [root@backup opt]# echo "rsync_backup:123456" > /etc/rsync.passwd

5,Authorization (must be 600)
 [root@backup opt]# chmod 600 /etc/rsync.passwd

6,Create backup directory
 [root@backup opt]# mkdir /backup
 [root@backup opt]# mkdir /tmp/linux

7,Directory authorization
 [root@backup opt]# chown rsync.rsync /backup/
 [root@backup opt]# chown rsync.rsync /tmp/linux/

8,Turn off firewalls and selinux
 [root@backup opt]# systemctl disabel --now firewalld
 [root@backup opt]# setenforce 0

9,start-up rsyncd service
 [root@backup opt]# systemctl start rsyncd

2. Client

Method 1: enter your own password
  [root@m01 ~]# rsync -avzP ./* rsync_backup@
   rsync_backup :  Virtual user, used only during data transmission  :  backup Server side IP
   backup       :  Module name
Method 2: set the password file and read it at run time

  1,Write password file
   [root@backup opt]# echo "123456" > /etc/rsync.passwd

  2,to grant authorization
   [root@m01 ~]# chmod 600 /etc/rsync.passwd

   [root@m01 ~]# rsync -avzP --password-file=/etc/rsync.passwd  ./* rsync_backup@

Method 3: add environment variables
  1,Define environment variables
   export RSYNC_PASSWORD=123456
   [root@m01 ~]# rsync -avzP  ./* rsync_backup@

rsync real-time synchronization

rsync Real time synchronization is not supported. Usually, we use inotify This software monitors file changes in real time once inotify If the file changes, it is called immediately rsync Synchronize.

1,install inotify(Installed on the client)
[root@web01 ~]# yum -y install inotify-tools

2,inotify Parameter introduction
-m Continuous monitoring
-r recursion
-q Silent, printing only time information
--timefmt Specifies the output time format
--format Specifies the event output format
    %Xe event
    %w catalogue
    %f file
-e Specify the events to monitor
    access visit
    modify Content modification
    attrib Attribute modification
    close_write Modify real file content
    open open
    create establish
    delete delete
    umount uninstall
3,Start monitoring
[root@m01 ~]# /usr/bin/inotifywait  -mrq  --format '%Xe  %w  %f' -e create,modify,delete,attrib,close_write  /root

4,Real time monitoring and synchronization
[root@m01 ~]# /usr/bin/inotifywait  -mrq  --format '%Xe  %w  %f' -e create,modify,delete,attrib,close_write  /root | while read line;do
  cd  /root
  rsync -avzP --delete --password-file=/etc/rsyncd.passwd ./* rsync_backup@