Background:
One OA system of the company is a stand-alone server. Due to disk failure, some OA accessories are lost. In order to avoid this problem, it is planned to realize the file spare parts function through rsync+inotify software and back up the accessories to other servers;
1, Environment introduction
Source side (server that needs to synchronize files): 172.12.6.123, and the file directory is / u01/weaverfile/file
Target side (server for backing up files): 172.12.7.51. The file directory is / file/oafile/weaverfile/file
The operating system is centos7 four
You can view the version of rsync software through rsync --version:
Both systems are centos7 4. Therefore, the rsync version is the same as:
rsync version 3.0.9 protocol version 30
Copyright © 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
2, First, let's introduce inotify
inotify is Linux kernel 2.6 13 (June 18, 2005) adds a new subsystem (API), which provides a monitoring file system The event mechanism (based on inode) can monitor the changes of the file system, such as file modification, addition, deletion, etc., and can notify the application of the corresponding events. This mechanism is introduced by the famous desktop search engine project beagle to replace dnotify, which has similar functions but has many defects.
Inotify can monitor both files and directories. When monitoring a directory, it can simultaneously monitor the directory and all subdirectories and files in the directory. In addition, inotify uses file descriptors as interfaces, so you can use the usual file I/O operations select, poll, and epoll to monitor changes in the file system.
Inotify tools provides two command-line tools:
inotifywait: wait for the corresponding events on the monitored file through the inotify API and return the monitoring results. By default, the normal results are returned to the standard output, and the diagnostic class information is returned to the standard error output. It can exit after monitoring the specified event on the corresponding monitoring object, or it can conduct continuous monitoring.
inotifywatch: collect events related to monitored files or directories through inotify API and output statistics.
The inotifywait tool is used here. The following are the corresponding parameters and events:
inotify Common file system events that can be monitored include: Event name Event description access Read file or directory contents modify Modify file or directory contents attrib The properties of a file or directory have changed close_write Modify real file content close_nowrite close open The file or directory is open moved_to Move files or directories to moved_from Move files or directories from move Move file or directory to monitor directory create Create a file or directory under the monitoring directory delete Delete the file or directory under the monitoring directory delete_self unmount Unmount file system
inotify parameter:
Parameter name Parameter description -m,–monitor Always keep event listening -r,–recursive Recursive query directory -q,–quiet Print only information for monitoring events –excludei Case insensitive when excluding files or directories -t,–timeout Timeout –timefmt Specifies the time output format –format Specifies the time output format -e,–event Events such as deletion, addition and modification are specified later
Check if inotify is supported
[root@oadb inotify]# ll /proc/sys/fs/inotify Total consumption 0 -rw-r--r-- 1 root root 0 12 23 / 13:12 max_queued_events -rw-r--r-- 1 root root 0 12 23 / 13:12 max_user_instances -rw-r--r-- 1 root root 0 12 23 / 13:12 max_user_watches
Set the memory size that inotify can use through the following parameters in the / proc interface:
1,/proc/sys/fs/inotify/max_queue_events
When an application calls inotify, it needs to initialize the inotify instance and set an event queue for it. The value in this file is used to set the upper limit of the queue length; Events exceeding this limit will be discarded;
2,/proc/sys/fs/inotify/max_user_instances
The value in this file is used to set the maximum number of inotify instances that can be created by each user ID (user identified by ID);
3,/proc/sys/fs/inotify/max_user_watches
The value in this file is used to set the maximum number of files or directories that can be monitored by each user ID;
3, Install rsync and inotify
3.1 install rsync at the target end (install 172.12.7.51 at the target end first)
Before installation, you can check whether to install through the command:
[root@oadb inotify]# rpm -qc rsync
/etc/rsyncd.conf /etc/sysconfig/rsyncd
If there is no installation, execute the following commands. If the system disk needs to be mounted and configured, Yum
Centos 7.4yum configuration:
First, add / etc / yum.com repos. d/CentOS-Base. Rename the repo file to CentOS base repo. bak
#mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak Then modify CentOS-Media.repo The documents are as follows: vi /etc/yum.repos.d/CentOS-Media.repo [c7-media] name=CentOS-$releasever - Media baseurl=file:///mnt/ gpgcheck=1 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Hang on CD to / mnt Directory:
#mount /dev/cdrom /mnt
Install rsync on the target side:
#yum install rsync -y
Add startup after installation
#echo 'rsync --daemon' >> /etc/rc.d/rc.local
Set the rsync password (not the system user password) and modify the permissions
#echo 'admin:1234567' > /etc/rsyncd.scrt #chmod 600 /etc/rsyncd.scrt
Configure Rsync Conf file
Configuration file (/ etc/rsyncd.conf)
#vi /etc/rsyncd.conf
# /etc/rsyncd: configuration file for rsync daemon mode # See rsyncd.conf man page for more options. # configuration example: uid = root gid = root use chroot = no max connections = 10 port = 873 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsync.log log format = %t %a %m %f %b syslog facility = local3 timeout = 300 [OA] path = /file/oafile/weaverfile/file comment = rsync info ignore errors read only = no list = no auth users = admin secrets file = /etc/rsyncd.scrt #exclude = * #Directories that do not need to be backed up #exclude from = /etc/rsync_exclude.txt #Directories not backed up hosts allow = 172.12.6.123/255.255.255.0 hosts deny = *
Start rsync
#rsync --daemon
3.2 source side installation rsync and inotify
Install rsync on the source side (172.12.6.123)
Before installation, you can check whether to install through the command:
[root@oadb inotify]# rpm -qc rsync
/etc/rsyncd.conf /etc/sysconfig/rsyncd
Install rsync:
#yum install rsync -y
Add startup after installation
#echo 'rsync --daemon' >> /etc/rc.d/rc.local
Set the rsync password (not the system user password) and modify the permissions
#echo '1234567' > /etc/rsyncd.scrt #chmod 600 /etc/rsyncd.scrt
Configure Rsync Conf file
Configuration file (/ etc/rsyncd.conf)
#vi /etc/rsyncd.conf
# /etc/rsyncd: configuration file for rsync daemon mode # See rsyncd.conf man page for more options. # configuration example: # uid = nobody # gid = nobody # use chroot = yes # max connections = 4 # pid file = /var/run/rsyncd.pid # exclude = lost+found/ # transfer logging = yes # timeout = 900 # ignore nonreadable = yes # dont compress = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2 # [ftp] # path = /home/ftp # comment = ftp export area # /etc/rsyncd: configuration file for rsync daemon mode # See rsyncd.conf man page for more options. # configuration example:
Start rsync
#rsync --daemon
Install notify on the source side:
The installation package is divided into source code and rpm package. The download address is as follows:
Source code download address: http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
RPM package download page: http://rpm.pbone.net/index.php3/stat/4/idpl/15265939/dir/redhat_el_5/com/inotify-tools-3.14-1.el5.i386.rpm.html
This case is installed through the source code:
[root@inotify-master]# wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz [root@inotify-master]# tar zxf inotify-tools-3.14.tar.gz [root@inotify-master]# cd inotify-tools-3.14 [root@inotify-master inotify-tools-3.14]# ./configure --prefix=/usr/local/include/ [root@inotify-master inotify-tools-3.14]# make && make install
After installing inotify, test and analyze inotifywait:
[root@oadb bin]# /usr/local/include/bin/inotifywait -mrq --format '%Xe %w%f' -e modify,create,delete,attrib,close_write,move /u01/weaverfile/file CREATE /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip CLOSE_WRITEXCLOSE /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip MODIFY /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip MODIFY /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip MODIFY /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip MODIFY /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip MODIFY /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip CLOSE_WRITEXCLOSE /u01/weaverfile/file/202112/S/067e5762-565c-428c-b310-6dda1b52c9ed.zip
The results show that the first column is the event and the second column is the path of the event
To achieve real-time, it is necessary to reduce the recursive scanning judgment of rsync on the directory, and only synchronize inotify to monitor the changed files as much as possible. Combined with the characteristics of rsync, it is necessary to judge separately to realize the corresponding operations of adding, deleting, changing and querying a directory.
Set the script on the source side. The script is as follows:
#vi /root/rsync_ino.sh
#!/bin/bash
src=/u01/weaverfile/file # Source path to synchronize
des=OA # The name published by rsync --daemon on the target server. rsync --daemon will not be introduced here. It is relatively simple to search online.
rsync_passwd_file=/etc/rsyncd.scrt # Password file for rsync authentication
ip1=172.12.7.51 # Target server 1
user=admin # Rsync -- authentication user name defined by daemon
cd ${src}
/usr/local/include/bin/inotifywait -mrq --format '%Xe %w%f' -e modify,create,delete,attrib,close_write,move ./ | while read file
do
INO_EVENT=$(echo $file | awk '{print $1}') # Cut the inotify output and assign the event type part to INO_EVENT
INO_FILE=$(echo $file | awk '{print $2}') # Cut the inotify output and assign the file path part to INO_FILE
echo "-------------------------------$(date)------------------------------------"
echo $file
if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]] # Judge event type
then
echo 'CREATE or MODIFY or CLOSE_WRITE or MOVED_TO'
rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
fi
if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]]
then
echo 'DELETE or MOVED_FROM'
rsync -avzR --delete --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
fi
if [[ $INO_EVENT =~ 'ATTRIB' ]]
then
echo 'ATTRIB'
if [ ! -d "$INO_FILE" ]
then
rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
fi
fi
done
Add execute permissions to scripts
#chmod +x /root/rsync_ino.sh
4, Execute the script to verify the results
[root@oadb ~]# sh rsync_ino.sh > rsync_ino.log &
[1] 24080
[root@oadb ~]# tail -f rsync_ino.log
-------------------------------2021 Friday, December 24, 2009 09:02:28 CST------------------------------------ CREATE ./202112/U/536a0421-d6cb-4cfd-b5f0-01e7d00d1a32.zip CREATE or MODIFY or CLOSE_WRITE or MOVED_TO sending incremental file list ./ 202112/U/ 202112/U/05975f07-d2fe-4e92-8f17-827c143444c8.zip 202112/U/1040417372.zip 202112/U/1145499273.zip 202112/U/1150692820.zip 202112/U/1217963347.zip 202112/U/1225719111.zip 202112/U/1303346186.zip 202112/U/1579297567.zip 202112/U/1707659339.zip 202112/U/171e9f82-b076-49fc-a879-0854b7e7ed09.zip 202112/U/1759253984.zip 202112/U/1884103102.zip 202112/U/1907833910.zip 202112/U/1969725107.zip 202112/U/1973033628.zip 202112/U/1dcd0ae8-40e5-44bb-8610-436d0d0533f5.zip 202112/U/21bbfdd3-0f63-4ecf-b4c6-a356acc9bbad.zip 202112/U/22825127-7802-47d2-94e1-feb3839d6c45.zip 202112/U/2d453e90-9f11-48db-952a-e35905502bf3.zip 202112/U/3902b136-4195-47b9-8862-b96417d1e1e8.zip 202112/U/3d1e574c-3e25-4253-95bb-e5759a795cbd.zip 202112/U/4208f55e-a59c-4998-97a6-aae08d2e8dd1.zip 202112/U/48c89853-4e89-4407-9661-f6f6f659df6a.zip 202112/U/4eae8196-2d0e-400c-a8fb-fb6a8c51c1f7.zip 202112/U/536a0421-d6cb-4cfd-b5f0-01e7d00d1a32.zip 202112/U/6aced335-42e0-4469-827f-bdae787ddd7a.zip 202112/U/7164bf77-059a-41c5-ad56-90dd83398415.zip 202112/U/74e4d3fc-87a1-4928-ba1d-03a184615c0e.zip 202112/U/7aab9548-0b09-491c-a117-7ecb0ca397fb.zip 202112/U/864e86f1-1759-4b18-8976-2d1e7de31494.zip 202112/U/88dae8e1-67da-49b8-b34a-b5991d9ddf3d.zip 202112/U/8be0c03e-9822-4abf-9eb2-f55ac208e14b.zip 202112/U/8d22b984-3f97-4623-b45c-76925373d074.zip 202112/U/8d43faed-a08a-41a5-80b1-8f1bbdda1ac8.zip 202112/U/9ff6b97e-62ec-411a-888b-370e4b8aa69c.zip 202112/U/a92797c7-a98f-4ec6-a4eb-008722f2bc81.zip 202112/U/b060e9b2-aabc-4de7-837a-02c4fc6560af.zip 202112/U/b10db7ab-8014-4824-97a6-cd674dcb6620.zip 202112/U/c10fcb28-6fa3-493a-b135-24e9e7eea92c.zip 202112/U/c1a68aca-aefa-4bb0-a32e-841ee7a0972d.zip 202112/U/c32f02b8-111c-4806-ac52-708ef3fcbb53.zip 202112/U/ccc7e85c-3821-4d1a-ab41-c2b0cc1556ab.zip 202112/U/cfbc5398-d8a6-4a5c-9c58-2b41f26fa384.zip 202112/U/daf7ebea-f5b9-4e0d-8955-ab2361e34165.zip 202112/U/f6ba5447-2dfb-47ec-8660-53b4870db507.zip 202112/U/fcb73094-120f-4fbc-9b1c-06cc0baa9ecd.zip 202112/U/fcebd7f7-a3cf-4dd7-8ad8-b503ba2040d8.zip 202112/U/fe9ea313-07cb-4b3f-998a-596f9004db91.zip sent 29292071 bytes received 929 bytes 1362465.12 bytes/sec total size is 915383785 speedup is 31.25 -------------------------------2021 Friday, December 24, 2009 09:02:49 CST------------------------------------ CLOSE_WRITEXCLOSE ./202112/U/536a0421-d6cb-4cfd-b5f0-01e7d00d1a32.zip CREATE or MODIFY or CLOSE_WRITE or MOVED_TO sending incremental file list
At this point, the installation of synchronization settings is completed!!