2, Backup service (rsync)
(1) rsync service introduction
Rsync is an open source, fast, multi-functional, full and incremental local or remote data synchronization backup tool. And it can realize the backup and migration of data without changing the attribute information of the original data. Rsync software is suitable for unix/linux/windows and other operating system platforms.
Rsync is a fast and very general file replication tool. It can be replicated locally, remotely, or by remote daemons. It provides a large number of parameters to control all aspects of its behavior, and allows a very flexible way to transfer and copy files. It is known for its delta transfer algorithm. Reduce the number of data sent through the network, and use only the difference information between the source file and the target file to achieve incremental synchronous replication of data.
(2) Simple application of rsync service command
1. Principle of Rsync replication synchronous data
When synchronizing backup data, by default, Rsync uses its unique "quick check algorithm, which synchronizes only the files or directories whose size or last modification time has changed. Of course, it can also synchronize according to the changes of permissions, attributes and other attributes. However, it needs to specify the corresponding parameters, and it can even synchronize only the changed parts of a file. Therefore, it can quickly synchronize the backup data, that is, it uses incremental replication method to advance the data information Row synchronization is different from the full copy replication of traditional CP and SCP copy tools. Incremental synchronous replication of data is much more efficient than full copy.
2. Summary of 7 features of Rsync backup software:
Support copying common files and special files such as linked files, devices, etc. It supports the function of excluding the specified file or directory synchronization, similar to the tar command exclusion function. It is supported to keep all attribute information of the original file or directory unchanged. It supports incremental synchronization, which only synchronizes change data and improves data transmission efficiency. Support the use of rcp,rsh,ssh and other ways to cooperate with the tunnel encryption transmission file. Support the use of socket (daemons) to transfer file or directory data information. Support user authentication to transfer data and improve data synchronization security.
3. Rsync service command is a very powerful command when it is applied. You can replace the following four commands with one Rsync command:
① . realize local data synchronous replication (equivalent command cp)
[root@backup ~]# # rsync == cp effect
[root@backup ~]# cp -a /etc/hosts /tmp/
[root@backup ~]# ll /tmp/
total 4
-rw-r--r--. 1 root root 352 Jan 27 01:15 hosts
[root@backup ~]# rsync -a /etc/sysconfig/network /tmp/
[root@backup ~]# ll /tmp/
total 8
-rw-r--r--. 1 root root 352 Jan 27 01:15 hosts
-rw-r--r-- 1 root root 31 Jan 26 18:16 network
② . realize remote data synchronous replication (equivalent command scp)
rsync == scp
[root@backup ~]# scp -rp /tmp/ 172.16.1.31:/tmp/
The authenticity of host '172.16.1.31 (172.16.1.31)' can't be established.
RSA key fingerprint is 5b:9b:e6:79:a9:95:4f:be:06:41:e3:bb:7a:12:ee:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.1.31' (RSA) to the list of known hosts.
root@172.16.1.31's password:
network 100% 31 0.0KB/s 00:00
hosts 100% 352 0.3KB/s 00:00
[root@backup ~]# ll /tmp/
total 8
-rw-r--r--. 1 root root 352 Jan 27 01:15 hosts
-rw-r--r-- 1 root root 31 Jan 26 18:16 network
[root@backup ~]# rsync -rp /tmp/ 172.16.1.31:/tmp/
root@172.16.1.31's password:
[root@backup ~]#
//Note: when synchronizing data, / tmp / directory is followed by / information, which means that the data content under the directory is backed up and synchronized
//When synchronizing data, there is no / information behind the / tmp directory, which means that the directory and the data content under the directory are backed up and synchronized
③ . implement data information deletion function (equivalent command rm)
explain: rsync To realize the process of deleting the data content in the directory, it is necessary to synchronize an empty directory with a directory with data
//Finally, the files in the directory with data will be emptied
[root@backup ~]# mkdir /null
[root@backup ~]# rsync --delete /null/ /tmp/
rsync: --delete does not work without -r or -d.
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
[root@backup ~]#
[root@backup ~]# rsync -r --delete /null/ /tmp/
[root@backup ~]# ll /tmp/
total 0
④ . realize data information viewing function (equivalent command ls)
[root@backup ~]# ls /etc/hosts
/etc/hosts
[root@backup ~]# ls -l /etc/hosts
-rw-r--r--. 2 root root 352 Jan 27 01:15 /etc/hosts
[root@backup ~]# rsync /etc/hosts
-rw-r--r-- 352 2018/01/27 01:15:59 hosts
(3) How rsync software works
- Local data backup mode
Local: rsync [OPTION...] SRC... [DEST]
rsync -- data backup transmission command
option - enter the parameters related to rsync data transmission
src -- data (file / directory) to be backed up
dest -- where to back up the data information (in the corresponding path)
Practice:
[root@backup ~]# rsync -a /etc/hosts /tmp/ok.txt
[root@backup ~]# ll /tmp/ok.txt
-rw-r--r-- 1 root root 352 Jan 27 01:15 /tmp/ok.txt
- Remote data backup mode
Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
Syntax description of pull mode:
rsync -- data backup transmission command
option - enter the parameters related to rsync data transmission
[USER@]HOST: --- you need to specify the user to log in to the remote host,
If USER information is omitted, login as current USER
Login host address or domain name information
SRC -- specifies the data information to be transmitted from the remote host to the local
dest -- what path to save data locally
Syntax description of push mode:
rsync -- data backup transmission command
option - enter the parameters related to rsync data transmission
[USER@]HOST: --- you need to specify the user to log in to the remote host,
If USER information is omitted, login as current USER
Login host address or domain name information
SRC -- specifies the data to be transferred from the local host to the remote host
dest -- which path to save the local data to the remote
- Daemons transfer mode
③. Daemons transfer mode
Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
[USER@]HOST:: --- Specify the authenticated user for the remote connection
SRC --- Specify the corresponding module information
[DEST] --- Save remote data to local path information
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
[USER@]HOST:: --- Specify the authenticated user for the remote connection
SRC --- Specify the data information to be pushed locally
[DEST] --- Remote module information for saving data
(4) rsync daemons deployment process
1. Server deployment process
First mileage: check whether the software is installed
[root@backup ~]# rpm -qa rsync rsync-3.0.6-12.el6.x86_64
Second mileage: write configuration file
vim /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start## uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24 hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by oldboy" path = /backup read only = true [nfs] comment = "backup dir by oldboy" path = /nfs
Third mileage: create backup directory management user
useradd rsync -M -s /sbin/nologin
Fourth mileage: create backup directory
mkdir /backup chown -R rsync.rsync /backup
The fifth mileage: creating certification documents
echo "rsync_backup:oldboy123" >>/etc/rsync.password chmod 600 /etc/rsync.password
The sixth mileage: start rsync service
rsync --daemon
2. Client Deployment Process
First mileage: confirm whether the software is installed
[root@backup ~]# rpm -qa rsync rsync-3.0.6-12.el6.x86_64
Second mileage: create authentication password file
echo "oldboy123" >>/etc/rsync.password chmod 600 /etc/rsync.password
Third mileage: data backup test
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list hosts sent 189 bytes received 27 bytes 432.00 bytes/sec total size is 352 speedup is 1.63
(5) Troubleshooting methods for rsync service errors
1) Check error log
2) Can be in a simulated environment. Some mistakes in simulation exercises
Extension description: rsync start service extension parameter
--Port specifies the rsync service port number information. The default value is 873
--config=xxx specifies the recognized rsync service configuration file information
(6) rsync service extended application
① Configuration of multi module function of Daemons
Step 1: modify the configuration file
vim /etc/rsyncd.conf [backup01] comment = "backup dir by oldboy" path = /backup [backup02] comment = "backup dir by oldboy" path = /backup02
Step 2: create a multi module directory
mkdir /backup02 chown -R rsync.rsync /backup02
Step 3: restart the service program
killall rsync rsync --daemon
Step 4: Test and check
② The practice of exclusion function of Daemons
The first way to exclude data backup:--exclude
rsync -avz /test_dir/ --exclude=b --exclude=d rsync_backup@172.16.1.41::backup01 --password-file=/etc/rsync.password
rsync -avz /test_dir/ --exclude={b，d} rsync_backup@172.16.1.41::backup01 --password-file=/etc/rsync.password
rsync -avz /test_dir/ --exclude={b..d} rsync_backup@172.16.1.41::backup01 --password-file=/etc/rsync.password
The second way to exclude data backup: -- exclude from = file rsync -avz /test_dir/ --exclude-from=./exclude_file.txt rsync_backup@172.16.1.41::backup01 --password-file=/etc/rsync.password
③ Daemons to create backup directories
rsync -avz /etc/hosts --exclude-from=./exclude_file.txt rsync_backup@172.16.1.41::backup01/sa/ --password-file=/etc/rsync.password
rsync -avz /etc/hosts --exclude-from=./exclude_file.txt rsync_backup@172.16.1.41::backup01/dev/ --password-file=/etc/rsync.password
rsync -avz /etc/hosts --exclude-from=./exclude_file.txt rsync_backup@172.16.1.41::backup01/dba/ --password-file=/etc/rsync.password
④ Access control configuration of Daemons
Three situations: 1. Only whitelist, whitelist segment or host information is allowed. Other block 2. Only blacklist, blacklist segment or host information is blocked, others are allowed 3. If there is a blacklist, it should also be a whitelist. The information of the whitelist segment or host is allowed, the information of the blacklist segment or host is blocked, and the rest is allowed It is recommended to select only the first two configurations hosts allow = 172.16.1.0/24 hosts deny = 0.0.0./32
⑤ Daemons no difference synchronization configuration (- delete)
I have, you have; I don't have, you can't have rsync -avz /test_dir/ --delete rsync_backup@172.16.1.41::backup01 --password-file=/etc/rsync.password Note: be sure to use it carefully, or you may empty the backup directory; If you want to clear the catalog data quickly, you can also use no difference synchronization to clear
⑥ List function configuration of Daemons
list = false Note: indicates whether to display all module information of rsync server in list [root@nfs01 test_dir]# rsync rsync_backup@172.16.1.41:: backup01 "backup dir by oldboy" backup02 "backup dir by oldboy"
(7) Rsync service FAQ summary
-
iptables firewall opened by rsync server
[Client error] No route to host [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: No route to host (113) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] [Abnormal problem solving] //Turn off the firewall service (iptables) of rsync server [root@backup mnt]# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@backup mnt]# /etc/init.d/iptables status iptables: Firewall is not running.
-
rsync client executing rsync command error
[Client error] The remote path must start with a module name not a / [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::/backup ERROR: The remote path must start with a module name not a / rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [Abnormal problem solving] rsync Command syntax understanding error,::/backup It's the wrong syntax. It should be::backup(rsync Modular)
-
rsync service failed to authenticate user*****
[Client error] auth failed on module oldboy [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [Abnormal problem solving] [Client error] auth failed on module oldboy [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [Abnormal problem solving] 1. Wrong password, wrong user name 2. secrets file = /etc/rsync.password The specified password file and the actual password file name are inconsistent 3. /etc/rsync.password File permission is not 600 4. rsync_backup:123456 Note that there should be no space after the password configuration file 5. rsync Only input the password information in the client password file, do not enter the virtual authentication user name -
rsync service location module error
1. [Client error] Unknown module 'backup' [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup @ERROR: Unknown module 'backup' rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [Abnormal problem solving] 2. /etc/rsyncd.conf Error writing profile module name -
rsync service permission blocking problem
1. [Client error] Permission denied [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts rsync: mkstemp ".hosts.5z3AOA" (in backup) failed: Permission denied (13) sent 196 bytes received 27 bytes 63.71 bytes/sec total size is 349 speedup is 1.57 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] [Abnormal problem solving] 2. The owner and group of the backup directory are incorrect, not rsync 3. Incorrect permissions for backup directory, not 755 -
rsync service backup directory exception
1. [Client error] chdir failed [error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [abnormal problem solving] 2. Backup storage directory is not established 3. The established backup storage directory and configuration file definitions are inconsistent Note: if the storage directory is not backed up -
rsync service invalid user information
[Client error] invalid uid rsync [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: invalid uid rsync rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] [Abnormal problem solving] rsync Service correspondence rsync Virtual user no longer exists
-
The client has configured the password file, but the password is still required for the keyless login mode
[Client error] password file must not be other-accessible [Error demonstration process] [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password password file must not be other-accessible continuing without password file Password: sending incremental file list sent 26 bytes received 8 bytes 5.23 bytes/sec total size is 349 speedup is 10.26 [Abnormal problem solving] rsync The client's secret key file must also have 600 permissions
-
Slow connection of rsync client
IP = = = domain name reverse DNS resolution [error log information] Error log output 2017/03/08 20:14:43 [3422] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:14:43 [3422] name lookup failed for 172.16.1.31: Name or service not known 2017/03/08 20:14:43 [3422] connect from UNKNOWN (172.16.1.31) 2017/03/08 20:14:43 [3422] rsync to backup/ from rsync_backup@unknown (172.16.1.31) 2017/03/08 20:14:43 [3422] receiving file list 2017/03/08 20:14:43 [3422] sent 76 bytes received 83 bytes total size 349 Correct log output 2017/03/08 20:16:45 [3443] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:16:45 [3443] connect from nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] rsync to backup/ from rsync_backup@nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] receiving file list 2017/03/08 20:16:45 [3443] sent 76 bytes received 83 bytes total size 349 [abnormal problem solving] Check the log for analysis, and write the rsync server hosts analysis file
10 rsync service did not start correctly
[Error log information]
Connection refused (111)
[Error demonstration process]
[root@oldboy-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup
rsync: failed to connect to 172.16.1.41: Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6]
[Abnormal problem solving]
[root@oldboy-muban ~]# rsync --daemon
[root@oldboy-muban ~]# ss -lntup |grep rsync
tcp LISTEN 0 5 :::873 :::* users:(("rsync",1434,5))
tcp LISTEN 0 5 *:873 *:* users:(("rsync",1434,4))
[root@oldboy-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup
Password:
sending incremental file list
hosts
sent 196 bytes received 27 bytes 49.56 bytes/sec
total size is 349 speedup is 1.57