Configure inotify+rsync real-time synchronization

Posted by skissiks on Thu, 15 Aug 2019 14:11:25 +0200

If you are not familiar with rsync services and commands, you can refer to the blog post. Remote synchronization through rsync
There's not much to say here!

Linux kernel has provided inotify notification interface since version 2.6.13 to monitor various changes of file system, such as file access, deletion, movement, modification, etc. Using this mechanism, it is very convenient to realize file alert and incremental backup, and to respond to changes in directories or files in a timely manner.

Combining inotify mechanism with rsync tool, backup at departure (real-time synchronization) can be achieved. As long as the original location of the document changes, incremental backup will be started immediately, otherwise it will be in a silent waiting state, as shown in the figure:

In this way, it avoids the problem of delay and overdense period when backing up according to fixed period.

It is precisely because the inotify notification mechanism is provided by the Linux kernel, so it is mainly for local monitoring, which is more suitable for upstream synchronization in triggered backup applications.

The software package inotify-tools-3.14.tar.gz is needed for the experiment. https://pan.baidu.com/s/1Ov006-6MV76VMVWL1LFyAA
Extraction code: np88

Configuring inotify+rsync real-time synchronization can be roughly divided into four steps:
(1) adjusting inotify kernel parameters;
(2) Install inotify-tools software package;
(3) Writing trigger synchronization script;
(4) Testing the experimental results.

This is based on the experimental environment of the previous blog: Remote synchronization through rsync

Following is the introduction and configuration (the following operations are performed in Server B!) :

(1) Adjusting inotify kernel parameters

In the Linux kernel, the default inotify mechanism provides three control parameters:
(1) max_queue_events: monitor queue size (operation events);
(2) max_user_instances: the maximum number of monitoring instances (monitoring directory);
(3) max_user_watches: The maximum number of monitored files per instance.

[root@localhost ~]# cat /proc/sys/fs/inotify/max_queued_events 
16384
[root@localhost ~]# cat /proc/sys/fs/inotify/max_user_instances 
128
[root@localhost ~]# cat /proc/sys/fs/inotify/max_user_watches 
8192
//The default value of the system can be modified if necessary.
[root@localhost ~]# vim /etc/sysctl.conf
                     ............                //Omit part of the commentary
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@localhost ~]# sysctl -p
//Effective immediately

Normally, the recommended value for monitoring is greater than the total number of files for monitoring objectives!

(2) Installing inotify-tools package

The inotify-tools package can be downloaded either from the inotify-tools official website or from the disks links at the beginning of the article.
Install on the client side:

[root@localhost ~]# tar zxf inotify-tools-3.14.tar.gz -C /usr/src
[root@localhost ~]# cd /usr/src/inotify-tools-3.14/
[root@localhost inotify-tools-3.14]# ./configure && make && make install

When the inotify-tools tool is installed, two commands are generated:
inotifywait: for continuous monitoring and real-time output of results;
inotifywatch: Used for short-term monitoring, after the completion of the task to produce results.

The parameters used by the command are:
- m. Continuous monitoring
 - r, recursively monitor all subobjects
 - q, simplify output information
 - e, specifying which event types to monitor

For detailed usage of commands, refer to its man manual!

Take the monitoring website root directory as an example:

[root@localhost ~]# inotifywait -mrq -e modify,move,create,delete,attrib /var/www/html
//To recursively and continuously monitor the changes, movement, creation, deletion, property changes and other events of the entire directory

(3) Writing trigger synchronization script

Write scripts on the client side:

[root@localhost ~]# vim 123.sh
#!/bin/bash
A="inotifywait -mrq -e modify,move,create,delete /var/www/html"
B="rsync -azH --password-file=/root/123.pass /var/www/html/* backuper@192.168.1.1::wwwroot"
$A | while read DIRECTORY EVENT FILE
do
        if
                [ $(pgrep rsync | wc -l) -le 0 ]
        then
                $B
        fi
done
[root@localhost ~]# chmod 777 123.sh
[root@localhost ~]# vim /etc/rc.d/rc.local
             ............         //Eliminate some of the content and write the following
/root/123.sh &
[root@localhost ~]# chmod 777 /etc/rc.d/rc.local
//        / etc/rc.d/rc.local This file mainly stores some boot-up scripts.

(4) Testing experimental results

Restart the client for testing to see if you can synchronize the content to the server!

[root@localhost ~]# pgrep 123.sh // / Check if the script has been automatically started
1033
1039
//Represents the process number of the script

Create files on the client for testing:

[root@localhost ~]# touch /var/www/html/666

Verify on the server side!

[root@localhost ~]# ls /var/www/html
666

Server has been automatically synchronized! The experiment is finished!

Topics: Linux inotify rsync vim