Memcached overview
An open source high performance distributed memory object caching system
All data is stored in memory
Support data of any storage type
Speed up Web site access
Memcached caching mechanism
When the program writes the cache data request, the API interface of Memcached routes the KEY input routing algorithm module to a service in the cluster, and then the API interface communicates with the server to complete a distributed cache write The Key index is built in the API, and the value data exists in memcached
Memcached distributed
It depends on the Memcached client to implement
Multiple Memcached servers are independent
How to store distributed data is decided by routing algorithm
Memcached routing algorithm
hash algorithm for remainder First use key to do hash operation to an integer, then do hash algorithm, and route according to the remainder. Not suitable for dynamic environment Consistency hash algorithm According to the hash algorithm, the corresponding key is mapped through a certain hash algorithm to form a head tail closed loop, and then the machine is mapped into the ring by using the same hash algorithm as the object storage, and all objects are stored in the nearest machine in the clockwise direction. Suitable for dynamic change
Memcached is a distributed memory object caching system developed by danga.com (the technical team running LiveJournal), which is used to reduce database load and improve performance in dynamic system. I believe that many people have used it. This paper aims to get a deeper understanding of this excellent open source software through the implementation of memcached and code analysis, and further optimize it according to our needs. Finally, we will deepen our understanding of the use of memcached through the analysis of Bsm Memcache extension.
What is Memcached
Before we elaborate on this issue, we should first make it clear that it is "not what". Many people use it as a storage carrier in the form of shared memory. Although memcached uses the same "key = > value" method to organize data, it is very different from shared memory, APC and other local caches. Memcached is distributed, which means it is not local. It is based on the network connection (of course, it can also use localhost) mode to complete the service, and it is an application independent program or Daemon mode. Memcached uses the libevent library to implement the network connection service. Theoretically, it can handle unlimited connections. But unlike Apache, memcached is more oriented to stable and continuous connections, so its actual concurrency is limited. In the conservative case, the maximum number of simultaneous connections of memcached is 200, which is related to Linux thread capability, and this value can be adjusted. For libevent, please refer to the relevant documents. Memcached memory is also used in a different way than APC. APC is based on shared memory and MMAP. Memcache has its own memory allocation algorithm and management mode. It has nothing to do with shared memory and no limitation of shared memory. Generally, each memcached process can manage 2GB of memory space. If more space is needed, the number of processes can be increased.
What occasion is Memcached suitable for
In many cases, memcached has been abused, which of course can not be without complaining about it. I often see someone post on the forum, similar to "how to improve efficiency". The reply is "use memcached". As for how to use it, where to use it, and what to do, there is nothing. Memcached is not universal, nor is it applicable to all occasions.
Memcached is a "distributed" memory object cache system. That is to say, those applications that do not need to be "distributed", do not need to be shared, or are simply small enough to have only one server, memcached will not bring any benefits, but will slow down the system efficiency, because network connection also needs resources, even UNIX local connection. In my previous test data, memcached's local read and write speed is dozens of times slower than the direct PHP memory array, while APC and shared memory mode are almost the same as the direct array. It can be seen that if it is only local level cache, it is not cost-effective to use memcached.
Memcached is often used as the front-end cache of the database. Because it has less SQL parsing, disk operation and other costs than database, and it uses memory to manage data, so it can provide better performance than directly reading database. In large-scale system, access to the same data is very frequent, memcached can greatly reduce database pressure, and improve system execution efficiency. In addition, memcached is often used as a storage medium for data sharing between servers. For example, data stored in SSO system can be saved in memcached and shared by multiple applications.
It should be noted that memcached uses memory to manage data, so it is volatile. When the server is restarted or the memcached process is stopped, the data will be lost. Therefore, memcached cannot be used to persist data. Many people's misconception is that the performance of memcached is very good, which is good enough to compare memory with hard disk. In fact, the memory used by memcached will not get hundreds of read-write speed improvement. Its actual bottleneck lies in the network connection. Compared with the database system using disk, the advantage lies in that it is very "light" because there is not too much overhead and direct read-write side Type, it can easily cope with very large data exchange volume, so it often occurs that two gigabit network bandwidth are full, and the memcached process itself does not occupy much CPU resources.
Let's start with memcached cluster
Experimental environment
4 192.168.136.238 main server
5 192.168.136.239 slave server
6 192.168.136.185 client
Drift address of client access 192.168.136.188**
4. Install memcached, libevent event library, and mamgent agent package on the main server
[root@localhost ~]# mkdir /abc [root@localhost ~]# mount.cifs //192.168.100.25/memcached/abc ා mount Password for root@//192.168.100.25/memcached: [root@localhost ~]# cd /abc/ [root@localhost abc]# ls LAMP-php5.6 magent-0.5.tar.gz memcached-1.5.6.tar.gz libevent-2.1.8-stable.tar.gz memcache-2.2.7.tgz [root@localhost abc]# tar zxvf libevent-2.1.8-stable.tar.gz -C /opt #Event library, memcached depends on event library [root@localhost abc]# tar zxvf memcached-1.5.6.tar.gz -C /opt/ #memcached on the server [root@localhost abc]# yum install gcc gcc-c++ make -y #Install environment package [root@localhost abc]# mkdir /opt/magent [root@localhost abc]# tar zxvf magent-0.5.tar.gz -C /opt/magent/ [root@localhost opt]# cd libevent-2.1.8-stable/ [root@localhost libevent-2.1.8-stable]# ./configure --prefix=/usr [root@localhost libevent-2.1.8-stable]# make && make install #Build install [root@localhost libevent-2.1.8-stable]# cd ../memcached-1.5.6/ [root@localhost memcached-1.5.6]# ./configure --with-libevent=/usr make make install
5. Install memcached and libevent event library from the server
[root@localhost ~]# mkdir /abc [root@localhost ~]# mount.cifs //192.168.100.25/memcached /abc Password for root@//192.168.100.25/memcached: [root@localhost ~]# cd /abc/ [root@localhost abc]# ls LAMP-php5.6 magent-0.5.tar.gz memcached-1.5.6.tar.gz libevent-2.1.8-stable.tar.gz memcache-2.2.7.tgz [root@localhost abc]# tar zxvf libevent-2.1.8-stable.tar.gz -C /opt #Event library, memcached depends on event library [root@localhost abc]# tar zxvf memcached-1.5.6.tar.gz -C /opt/ #memcached on the server [root@localhost abc]# yum install gcc gcc-c++ make -y [root@localhost opt]# cd libevent-2.1.8-stable/ [root@localhost libevent-2.1.8-stable]# ./configure --prefix=/usr [root@localhost libevent-2.1.8-stable]# make && make install [root@localhost libevent-2.1.8-stable]# cd ../memcached-1.5.6/ [root@localhost memcached-1.5.6]# ./configure --with-libevent=/usr make make install
4 main server configuration magent
[root@localhost memcached-1.5.6]# cd /opt/ [root@localhost opt]# ls libevent-2.1.8-stable magent memcached-1.5.6 rh [root@localhost opt]# cd magent/ [root@localhost magent]# vim ketama.h #Change two lines #ifndef SSIZE_MAX #define SSIZE_MAX 32767 [root@localhost magent]# vim Makefile #Specify makefile file, change one line LIBS = -levent -lm make #compile [root@localhost magent]# ls ketama.c ketama.h ketama.o magent magent.c magent.o Makefile [root@localhost magent]# yum install openssh-clients -y #Install scp remote synchronization package [root@localhost magent]# cp magent /usr/bin/ #Put the magent script in / usr/local so that the system can recognize [root@localhost magent]# scp magent root@192.168.136.239:/usr/bin/ #Copy the mangent file to the slave server
Both master and slave turn off the firewall
[root@localhost bin]# systemctl stop firewalld.service [root@localhost bin]# setenforce 0
Master and slave
[root@localhost bin]# yum install keepalived -y
4 master server configuration master slave synchronization
[root@localhost magent]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
vrrp_script magent { #Write a function script
script "/opt/shell/magent.sh" ##Specify script location
interval 2 ##Test script interval
}
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id MAGENT_HA #Primary server id, two cannot be the same
}
vrrp_instance VI_1 {
state MASTER
interface ens33 #Master server network card
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111 #Default validation
}
track_script { #Call function name magent
magent
}
virtual_ipaddress {
192.168.136.188 #Drift address of client access
}
} 5 install openssh client from server
[root@localhost bin]# cd /etc/keepalived/ [root@localhost keepalived]# mv keepalived.conf keepalived.conf.bak #Renamed [root@localhost keepalived]# yum install openssh-clients -y
4. The master server uses scp to transfer the maintained file to the slave server
[root@localhost magent]# cd /etc/keepalived/ [root@localhost keepalived]# scp keepalived.conf root@192.168.136.239:/etc/keepalived/
5. Configure master-slave synchronization for slave server
[root@localhost keepalived]# ls
keepalived.conf keepalived.conf.bak
[root@localhost keepalived]# vim keepalived.conf
! Configuration File for keepalived
vrrp_script magent {
script "/opt/shell/magent.sh"
interval 2
}
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id MAGENT_HB #Routed? ID cannot be the same
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 52 #The slave virtual id cannot be the same as the master server
priority 90 #Priority is lower than the primary server
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
magent
}
virtual_ipaddress {
192.168.136.188
}
} 4 main server configuration magent script
[root@localhost keepalived]# mkdir /opt/shell
[root@localhost keepalived]# cd /opt/shell/
[root@localhost shell]# vim magent.sh
#!/bin/bash
k=`ps -ef | grep keepalived | grep -v grep | wc -l` #Check the keepalived process, if it is enabled
if [ $k -gt 0 ]; then # -n number of connections - l specifies the drift address, - p specifies the address of the port mapped to the master-slave server
magent -u root -n 51200 -l 192.168.136.188 -p 12000 -s 192.168.136.238:11211 -b 192.168.136.239:11211
else
pkill -9 magent
fi
[root@localhost shell]# chmod +x magent.sh
[root@localhost shell]# systemctl start keepalived.service
[root@localhost shell]# netstat -ntap | grep 12000
tcp 0 0 192.168.136.188:12000 0.0.0.0:* LISTEN 124720/magent 5 same operation from server
[root@localhost keepalived]# mkdir /opt/shell
[root@localhost keepalived]# cd /opt/shell/
[root@localhost shell]# vim magent.sh
#!/bin/bash
k=`ps -ef | grep keepalived | grep -v grep | wc -l`
if [ $k -gt 0 ]; then
magent -u root -n 51200 -l 192.168.136.188 -p 12000 -s 192.168.136.238:11211 -b 192.168.136.239:11211
else
pkill -9 magent
fi
[root@localhost shell]# chmod +x magent.sh
[root@localhost shell]# systemctl start keepalived.service
[root@localhost shell]# netstat -ntap | grep 12000 #View magent port
tcp 0 0 192.168.136.188:12000 0.0.0.0:* LISTEN 11660/magent Turn on 4 main servers memcached
[root@localhost shell]# memcached -m 512k -u root -d -l 192.168.136.238 -p 11211 #Start master, - m specifies the space size [root@localhost shell]# netstat -ntap | grep 11211 tcp 0 0 192.168.136.238:11211 0.0.0.0:* LISTEN 44647/memcached
Turn on 5 slave server memecached
[root@localhost shell]# memcached -m 512k -u root -d -l 192.168.136.239 -p 11211 #Starting from [root@localhost shell]# netstat -ntap | grep 11211 tcp 0 0 192.168.136.239:11211 0.0.0.0:* LISTEN 42654/memcached
telnet is used to connect memcached memory database for client verification
[root@localhost ~]# telnet 192.168.136.188 12000 Trying 192.168.136.188... Connected to 192.168.136.188. Escape character is '^]'. add username 0 0 7 #Let's write a key value pair 1234567 STORED
Go to 4 main servers to check whether there is any data
[root@localhost shell]# telnet 192.168.136.238 11211 Trying 192.168.136.238... Connected to 192.168.136.238. Escape character is '^]'. geer^H^H ERROR get username VALUE username 0 7 1234567 END
Verify that 5 slave data is synchronized
[root@localhost shell]# telnet 192.168.136.239 11211 Trying 192.168.136.239... Connected to 192.168.136.239. Escape character is '^]'. get username VALUE username 0 7 1234567 END