Introduction to cluster & introduction to keep alived & configuring high availability cluster with keep alived

Posted by samusk on Thu, 16 May 2019 14:12:21 +0200

18.1 Cluster Introduction

Linux clusters are divided into two main categories based on their functions: high availability and load balancing

High Availability Clusters usually consist of two servers, one working and the other serving as redundancy, which will continue to serve when the serving machine goes down.

* Open source software for high availability: heartbeat, keepalived

In a load balancing cluster, a server is needed as a distributor to distribute users'requests to the back-end server for processing. In this cluster, besides distributors, servers are provided to users, and the number of servers is at least 2.

* Open source software for load balancing includes LVS, keepalived, haproxy, nginx, and commercial F5, Netscaler.

18.2 Presentation of keepalived

Use keepalived to achieve high availability cluster, because heartbeat has some problems on centos6, which affect the experimental results

keepalived achieves high availability through VRRP (Virtual Router Redundancy Protocl).

In this protocol, multiple routers with the same function will be organized into a group, in which there will be a master role and N (N >= 1) backup role.

Master sends VRRP packets to various backups in the form of multicast. When the backup does not receive VRRP packets from the master, it will think that the master is down. At this point, we need to decide who will become the new mater according to the priority of each backup.

keepalived has three modules: core, check and vrrp. The core module is the core of keeping alived. It is responsible for the initiation, maintenance of the main process and loading and parsing of the global configuration file. The check module is responsible for the health check. The VRRP module is used to implement the VRRP protocol.

18.3/18.4/18.5 Configuring High Availability Clusters with Keeping Alive

Prepare two servers, one as master and the other as backup.

Install the keepalived package

Keeping alived, which actually contains a service, can also be said to be used to achieve high availability

Both machines execute Yum install-y keepalived

Install nginx

Using nginx as a highly available object - > the reason why nginx is used as a demo object, because nginx is used as a load balancer in work and production environments by many enterprises. Assuming that once nginx hangs up, all the back-end web, even if it is normal, can not be accessed.

yum installs nginx

yum install -y nginx

Source Pack Installation Nginx

Modify the keepalived configuration file on the primary server

########################  Clear the original configuration  #######################
[root@linux-5 ~]# vim /etc/keepalived/keepalived.conf 
[root@linux-5 ~]#  > !$
 > /etc/keepalived/keepalived.conf
[root@linux-5 ~]# vim /etc/keepalived/keepalived.conf 
########################  Global configuration  #######################
global_defs {                            //global_defs global configuration identifier
   notification_email {                  //notification_email is used to set the alarm mail address
   lem@qq.com                            //You can set up more than one, one for each line.
   }
   notification_email_from root@lem.com    //Setting the mail delivery address  
   smtp_server 127.0.0.1                   //Setting smtp server address for mail
   smtp_connect_timeout 30                 //Setting connection smtp sever timeout
   router_id LVS_DEVEL
}

#######################  check Module configuration  ######################
vrrp_script chk_nginx {               
    script "/usr/local/sbin/check_ng.sh"   //Check whether the service is normal or not, by writing scripts to achieve, scripts to check the health of the service
    interval 3                             //The time interval for inspection is 3 seconds.
}
#######################  vrrp Module configuration  ######################
vrrp_instance VI_1 {                      //VRRP configuration identifier VI_1 is the instance name 
    state MASTER                          //Defining master correlation
    interface ens33                       //To communicate and broadcast through vrrp protocol. When configuring, pay attention to the name of your network card
    virtual_router_id 51                  //Define the router ID and configure it in accordance with the slave machine
    priority 100                          //Weight, the main role and the subordinate role have different weights
    advert_int 1                          //Set the time interval of synchronous inspection between MASTER and BACKUP host in seconds
    authentication {                      //Authentication-related information
        auth_type PASS                    //The type of authentication here is PASS.
        auth_pass 123456                  //The form of a password is a string
    }
    virtual_ipaddress {                   //Setting Virtual IP Address (VIP), also known as Drift IP Address
        192.168.88.100                    //Change to 192.168.88.100
    }
    track_script {                        //Loading script 
        chk_nginx            
    }
}

The drift IP address is a common address. When the host is down, the standby Nginx starts. If the standby Nginx parsing address is still the source host IP, it still can not be accessed normally (the source host is down, the IP address is invalid). In order to solve the above problems, both the host and the standby are resolved to a common IP address, and the normal accessible IP address can be resolved when the standby Nginx service starts.

Configuration monitoring script

Host configuration monitoring script
vim /usr/local/sbin/check_ng.sh
#!/bin/bash
#Time variable for logging
d=`date --date today +%Y%m%d_%H:%M:%S`
#Calculate the number of nginx processes
n=`ps -C nginx --no-heading|wc -l`
#If the process is 0, start nginx and detect the number of nginx processes again.
#If it's still 0, it means that nginx can't start, and you need to turn off keepalived
if [ $n -eq "0" ]; then
        /etc/init.d/nginx start
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

In high availability (HA) systems, when the heartbeat of two nodes is disconnected, the HA system, which is a whole and coordinated action system, is split into two independent individuals. Because they lost contact with each other, they all thought that the other party had broken down. HA software on two nodes, like "split-brain person", competes for "shared resources" and "application services", and serious consequences will occur - or shared resources are divided, two sides of "services" are not up; or both sides of "services" are up, but at the same time read and write "shared storage", resulting in data damage.

How to judge cleft brain?

If both servers have virtual IP, it will show that there is a brain fissure. This proves that there is a problem in the current communication between two servers. The original reason for this problem is that neither server can detect the status of other servers in the group (heartbeat request can not respond properly). If the other server is suspended, it will seize the virtual IP if it decides privately that the other server is suspended. Brain fissure is not allowed. The solution to this problem is to check the firewall settings (close the firewall) or use serial communication.

Script permission change

chmod 755 /usr/local/sbin/check_ng.sh

Start the keepalived service

Check selinux and whether the firewall is closed before the mainframe and standby start the keep alived service

[root@linux-5 ~]# systemctl start keepalived.service
[root@linux-5 ~]# ps aux |grep keepalived
root       4572  0.0  0.0 118608  1384 ?        Ss   23:00   0:00 /usr/sbin/keepalived -D
root       4573  0.0  0.1 122804  2364 ?        S    23:00   0:00 /usr/sbin/keepalived -D
root       4574  0.0  0.1 122804  2408 ?        S    23:00   0:00 /usr/sbin/keepalived -D
root       4576  0.0  0.0 112676   984 pts/0    S+   23:00   0:00 grep --color=auto keepalived

Note: Before starting the keepalived service, you need to start the Nginx service first. Otherwise, it will be judged that the Nginx service is down and the keepalived service cannot be started.

Topics: Nginx Linux yum vim