1, Understand the working principle of LVS-DR
1.1 LVS-DR packet flow analysis
(1) The client sends a request to the Director Server (load balancer), and the requested data message (the source IP is CIR and the target IP is VIP) reaches the kernel space.
(2)DirectorServer and RealServer are in the same network, and the data is transmitted through the two-layer data link layer.
(3) The kernel space judges that the target IP of the packet is the local VIP. At this time, IPVS(IP virtual server) compares whether the service requested by the packet is a cluster service. If it is a cluster service, the packet will be re encapsulated. Modify the source MAC address to the MAC address of Director Server and the target MAC address to the MAC address of Real Server. The source IP address and the target IP address have not changed, and then send the packet to Real Server.
(4) If the MAC address of the request message arriving at the Real Server is its own MAC address, this message will be received. The data packet re encapsulates the message (the source IP address is VIP and the target IP is CIP), transmits the response message to the physical network card through the lo interface, and then sends it out.
(5) The real server directly transmits the response message to the client.
Features of DR mode:
(1) Director Server and Real Server must be in the same physical network.
(2)RealServer can use private address or public address. If the public network address is used, RIP can be accessed directly through the Internet.
(3) The director server serves as an access portal to the cluster, but not as a gateway.
(4) All request messages pass through the Director Server, but the reply response message cannot pass through the Director Server
(5) The gateway of the Real Server is not allowed to point to the Director Server IP, that is, the packets sent by the Real Server are not allowed to pass through the Director Server
(6) The lo interface on the real server configures the IP address of the VIP.
1.2 ARP problem in LVS-DR
1) Question 1:
- In LVS-DR load balancing cluster, both load balancing and node server should be configured with the same VIP address.
- Having the same IP address in the LAN is bound to cause the disorder of ARP communication among servers.
Solution:
- When ARP broadcast is sent to LVS-DR cluster, both load balancer and node server will receive ARP broadcast because they are connected to the same network.
- Only the front-end load balancer responds, and other node servers should not respond to ARP broadcast.
- Process the node server so that it does not respond to the ARP request for VIP.
resolvent:
- Use virtual interface lo:0 to host VIP address
- Set kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is local IP
2) Question 2:
- The message returned by RealServer (the source IP is VIP) is forwarded by the router. When repackaging the message, you need to obtain the MAC address of the router first.
- When sending an ARP request, Linux uses the source IP address of the IP packet (VIP s) as the source IP address in the ARP request packet by default instead of the IP address of the sending interface
For example: ens33
- After receiving the ARP request, the router will update the ARP table entry
- The MAC address of the Director corresponding to the original VIP will be updated to the MAC address of the RealServer corresponding to the VIP
- According to the ARP table entry, the router will forward the new request message to RealServer, resulting in the failure of the Director's VIP
resolvent:
Process the node server and set the kernel parameter ARP_ Announcement = 2: the system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface.
Setting methods to solve two problems of ARP
Modify / etc / sysctl Conf file
net.ipv4.conf.lo.arp_ignore=1 net.ipv4.conf.lo.arp_announce=2 net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.all.arp_announce=2
2, Master LVS-DR deployment process
| host | operating system | IP address | Required services |
|---|---|---|---|
| DR server (load scheduler) | CentOS7 | ens33:192.168.100.10 ens33:0(VIP):192.168.100.100 | ipvsadm |
| Web node server 1 | CentOS7 | ens33: 192.168.100.12 lo:0 (VIP): 192.168.100.100 | nfs-utils,rpcbind,httpd |
| Web node server 2 | CentOS7 | ens33: 192.168.100.13 lo:0 (VIP): 192.168.100.100 | nfs-utils,rpcbind,httpd |
| NFS server | CentOS7 | 192.168.100.40 | rpcbind,nfs-utils |
| client | Windows7 | 192.168.100.254 |
Note: this time it is built in the same LAN. When setting up the network, there is no need to use gateway and DNS. Just note. If you are not in the same network segment, you need to configure the gateway.
2.1. Deploy shared storage
NFS server: 192.168.100.40
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum -y install nfs-utils rpcbind systemctl start rpcbind.service systemctl start nfs.service systemctl enable nfs.service systemctl enable rpcbind.service mkdir /opt/test1 mkdir /opt/test2 chmod 777 /opt/test1 chmod 777 /opt/test2 vim /etc/exports /opt/test1 192.168.100.0/24(rw,sync) /opt/test2 192.168.100.0/24(rw,sync) exportfs -rv
2.2. Configure node server
Web node server 1: ens33:192.168.100.12 lo:0 (VIP): 192.168.100.100
Web node server 2: ens33:192.168.100.13 lo:0 (VIP): 192.168.100.100
Note: the configuration of the two servers is the same
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0
1) . configure virtual IP address (VIP:192.168.100.100)
#This address is only used as the source address for sending Web response packets, and does not need to listen to the client's access requests (instead, it is monitored and distributed by the scheduler). #Therefore, the virtual interface lo:0 is used to host the VIP address, and a path is added for the local machine with records, so as to limit the data accessing the VIP locally to avoid communication disorder. vim /etc/sysconfig/network-scripts/ifcfg-lo:0 DEVICE=lo:0 IPADDR=192.168.163.100 NETMASK=255.255.255.255 ONBOOT=yes ifup lo:0 ifconfig lo:0 #Set temporary route and restart failure; Confinement routing route add -host 192.168.163.100 dev lo:0 #View route route -n #Add routing automatically after startup. This should be used in production environment vim /etc/rc.local /sbin/route add -host 192.168.163.100 dev lo:0 chmod +x /etc/rc.d/rc.local
2) Adjust the ARP response parameters of the kernel to prevent updating the MAC address of VIP and avoid conflict
vim /etc/sysctl.conf ...... net.ipv4.conf.lo.arp_ignore = 1 #The system only responds to ARP requests whose destination IP is local IP net.ipv4.conf.lo.arp_announce = 2 #The system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 sysctl -p yum install -y nfs-utils rpcbind httpd systemctl start rpcbind systemctl enable rpcbind systemctl start httpd.service systemctl enable httpd.service
Note: the following two servers have different configurations
Web node server 1: ens33:192.168.100.12 lo:0 (VIP): 192.168.100.100
showmount -e 192.168.100.40 mount.nfs 192.168.100.40:/opt/test1 /var/www/html echo 'this is test1 web!' > /var/www/html/index.html
Web node server 2: ens33:192.168.100.13 lo:0 (VIP): 192.168.100.100
showmount -e 192.168.100.40 mount.nfs 192.168.100.40:/opt/test2 /var/www/html echo 'this is test2 web!' > /var/www/html/index.html
2.3. Configure load scheduler
Load scheduler: 192.168.100.10 lo:0 (VIP): 192.168.100.100
1) Turn off the firewall and load the IP address_ Vs module
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 #IP loading_ Vs module and install ipvsadm tool modprobe ip_vs cat /proc/net/ip_vs yum install -y ipvsadm
2) . configure virtual IP address (VIP: 192.168.100.100)
vim /etc/sysconfig/network-scripts/ifcfg-ens33:0 DEVICE=ens33:0 ONBOOT=yes IPADDR=192.168.100.100 NETMASK=255.255.255.255 ifup ens33:0 ifconfig ens33:0
3) Adjust proc response parameters
#Since the LVS load scheduler and all nodes need to share VIP addresses, the redirection parameter response of the Linux kernel should be turned off instead of acting as a router, vim /etc/sysctl.conf net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0 sysctl -p
4) . configure load distribution strategy
ipvsadm-save > /etc/sysconfig/ipvsadm perhaps ipvsadm --save > /etc/sysconfig/ipvsadm systemctl start ipvsadm.service #Clear original policy ipvsadm -C ipvsadm -A -t 192.168.001.100:80 -s rr ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.12:80 -g #If this is tunnel mode, just replace - g with - i ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.13:80 -g #Check the node status. Route represents DR mode ipvsadm -ln
2.4 test and verification
On client access http://192.168.100.100/ , refresh to test whether load balancing is successful
