rsync synchronization tool
- Rsync (remote synchronization) is an open source, fast, multi-functional and excellent tool for full and incremental local or remote data backup
Introduction to rsync
- rsync can be seen from the name of the software. rsync has the functions of fast data replication, synchronous mirroring and remote backup between local and remote hosts. This function is similar to the scp command with ssh, but better than the scp command. scp is a full copy every time, while rsync can be copied incrementally. Of course, rsync can also replicate data between different partitions or directories of the local host in full and incremental amounts, which is similar to the cp command. But it is also better than the cp command. cp is a full copy every time, while rsync can copy incrementally.
rsync Official address: https://rsync.samba.org/
rsync Listening port: 873
rsync Operation mode: C/S client/server
rsync It is called remote synchronization for short. It can realize data synchronization between different hosts, and also supports full and incremental synchronization
rsync feature
- 1) Support copying special files, such as linked files, devices, etc.
- 2) It can exclude the synchronization of specified files or directories, which is equivalent to the exclusion function of the packaging command tar.
- 3) You can keep all attributes of the original file or directory, such as permission, time, soft and hard links, owner and group, unchanged – p.
- 4) Incremental synchronization can be realized, that is, only the changed data can be synchronized, so the data transmission efficiency is very high (tar-N).
- 5) You can use rcp, rsh, ssh and other methods to transfer files (rsync itself does not encrypt data).
- 6) You can transfer files and data (server and client) through socket (process mode).
- 7) Support anonymous live authentication (without system users) process mode transmission, which can realize convenient and safe data backup and mirroring.
rsync application scenario
Full backup
Incremental backup
Data synchronization between two servers.
Synchronize all client server data to the backup server, and the production scenario cluster architecture server backup scheme.
rsync combination inotify Function to do real-time data synchronization.
Transmission mode of rsync
push PUSH:
The client pushes data from the local to the server(rsync The server actively pushes data to other hosts. The server overhead is large, which is suitable for the situation with few back-end servers)
pull PULL:
The client pulls data from the server to the local server(Client initiative rsync Server pull data)
rsync transport mode
1.Local mode (similar to cp，Push and pull are not supported, but simply copy)
2.Remote mode (similar to scp，And different from scp)，scp Only full backup is supported, rsync Support incremental backup and differential backup
3.Daemon( socket)Mode (client and server)
rsync usage
-a #Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L
-v #Detailed mode output, print rate, number of files, etc
[root@m01 ~]# rsync -v ./2.txt root@172.16.1.41:/opt/
-z #Compression during transmission to improve efficiency
[root@m01 ~]# rsync -vz ./2.txt root@172.16.1.41:/opt/
-r #Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same.
[root@m01 ~]# rsync -vzr ./a root@172.16.1.41:/opt/
-t #Keep file time information
[root@m01 ~]# rsync -vzrt ./a/b/c/2.txt root@172.16.1.41:/opt/
-o #Keep file master information
-g #Keep file group information
[root@m01 ~]# rsync -vzrtgo ./a/b/c/2.txt root@172.16.1.41:/opt/
-p #Keep file permissions
[root@m01 ~]# rsync -vzrtgop ./a/b/c/2.txt root@172.16.1.41:/opt/
-l #Keep soft connection
[root@m01 ~]# rsync -vzrtgopl ./* root@172.16.1.41:/opt/
-P #Display information such as synchronization process and transmission progress
[root@m01 ~]# rsync -vzrtgoplP ./* root@172.16.1.41:/opt/
-D #Keep device file information
[root@m01 dev]# rsync -vzrtgDopl /dev/tty1 root@172.16.1.41:/opt/
-L #Keep the target file pointed to by the soft connection
-e #Using the channel protocol, specify the shell program that replaces rsh
--append # The specified file continues to be transferred where the last transfer was interrupted
--append-verify # Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different)
--exclude=PATTERN # Specifies to exclude files that do not need to be transferred
[root@m01 ~]# rsync -avzP --append-verify --exclude=2.txt ./* root@172.16.1.41:/opt/
--exclude-from=file # Exclude as specified in the file
[root@m01 ~]# rsync -avzP --append-verify --exclude-from=/tmp/exclude.txt ./* root@172.16.1.41:/opt/
--bwlimit=100 # Speed limit transmission (unit: MB)
[root@m01 ~]# rsync -avzP --append-verify --bwlimit=10 ./* root@172.16.1.41:/opt/
--delete # Keep the target directory and source directory data consistent
--password-file=xxx # Use password file
--port # Specify port transport
rsync daemon mode
1. Server
1,install
[root@backup ~]# yum install -y rsync
2,Modify profile
[root@m01 ~]# vim /etc/rsyncd.conf
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[backup]
comment = welcome to backup!
path = /backup
[linux]
comment = welcome to linux!
path=/tmp/linux
3,Create system user
[root@backup opt]# groupadd rsync -g 666
[root@backup opt]# useradd rsync -u 666 -g 666 -M -s /sbin/nologin -r
4,Create password file
[root@backup opt]# echo "rsync_backup:123456" > /etc/rsync.passwd
5,Authorization (must be 600)
[root@backup opt]# chmod 600 /etc/rsync.passwd
6,Create backup directory
[root@backup opt]# mkdir /backup
[root@backup opt]# mkdir /tmp/linux
7,Directory authorization
[root@backup opt]# chown rsync.rsync /backup/
[root@backup opt]# chown rsync.rsync /tmp/linux/
8,Turn off firewalls and selinux
[root@backup opt]# systemctl disabel --now firewalld
[root@backup opt]# setenforce 0
9,start-up rsyncd service
[root@backup opt]# systemctl start rsyncd
2. Client
Method 1: enter your own password
[root@m01 ~]# rsync -avzP ./* rsync_backup@172.16.1.41::backup
rsync_backup : Virtual user, used only during data transmission
172.16.1.41 : backup Server side IP
backup : Module name
Method 2: set the password file and read it at run time
1,Write password file
[root@backup opt]# echo "123456" > /etc/rsync.passwd
2,to grant authorization
[root@m01 ~]# chmod 600 /etc/rsync.passwd
3,connect
[root@m01 ~]# rsync -avzP --password-file=/etc/rsync.passwd ./* rsync_backup@172.16.1.41::linux
Method 3: add environment variables
1,Define environment variables
export RSYNC_PASSWORD=123456
2,synchronization
[root@m01 ~]# rsync -avzP ./* rsync_backup@172.16.1.41::linux
rsync real-time synchronization
rsync Real time synchronization is not supported. Usually, we use inotify This software monitors file changes in real time once inotify If the file changes, it is called immediately rsync Synchronize.
1,install inotify(Installed on the client)
[root@web01 ~]# yum -y install inotify-tools
2,inotify Parameter introduction
-m Continuous monitoring
-r recursion
-q Silent, printing only time information
--timefmt Specifies the output time format
--format Specifies the event output format
%Xe event
%w catalogue
%f file
-e Specify the events to monitor
access visit
modify Content modification
attrib Attribute modification
close_write Modify real file content
open open
create establish
delete delete
umount uninstall
3,Start monitoring
[root@m01 ~]# /usr/bin/inotifywait -mrq --format '%Xe %w %f' -e create,modify,delete,attrib,close_write /root
4,Real time monitoring and synchronization
[root@m01 ~]# /usr/bin/inotifywait -mrq --format '%Xe %w %f' -e create,modify,delete,attrib,close_write /root | while read line;do
cd /root
rsync -avzP --delete --password-file=/etc/rsyncd.passwd ./* rsync_backup@172.16.1.41::backup
done