Step by step to complete Kubernetes binary deployment -- etcd cluster building (single node)

Posted by RestlessThoughts on Tue, 05 May 2020 17:14:40 +0200

Completing Kubernetes binary deployment step by step (I) -- etcd cluster building (single node)


The basic theory and core components of Kubernetes are briefly introduced. The first step of deploying Kubernetes cluster with single node and binary system is to build etcd cluster experiment flow.

List of cluster planning (the next article will follow the planning step by step configuration)

1. Deploy etcd cluster on three nodes
2. Deploy docker environment and flannel on two node nodes (communication within the container depends on vxlan Technology)
3. Deploy Kube apiserver, Kube controller manager and Kube scheduler on the master node
4. Deployment of kubelet and Kube proxy on node

Server ip address planning

master01 address:

node01 address:

node02 address:

This paper will give the etcd cluster building configuration process.

Construction process

1. Environmental preparation

It is recommended to set the host name, bind the static ip and turn off the network management service for the three servers. Next, it is necessary to turn off the firewall and core protection and clear the iptables

Take master01 as an example. Other nodes can follow similar settings of this node

[root@localhost ~]# hostnamectl set-hostname master01
[root@localhost ~]# su
[root@master01 ~]# systemctl stop firewalld
[root@master01 ~]# setenforce 0
[root@master01 ~]# iptables -F

Set static ip address

[root@master01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 
[root@master01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 
[root@master01 ~]# systemctl restart network
[root@master01 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:06:97:04 brd ff:ff:ff:ff:ff:ff
    inet brd scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5bd9:44c7:cf2c:ef20/64 scope link 
       valid_lft forever preferred_lft forever
[root@master01 ~]# ping #Test whether the Internet can be connected here
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=128 time=13.4 ms
64 bytes from ( icmp_seq=2 ttl=128 time=11.2 ms
64 bytes from ( icmp_seq=3 ttl=128 time=11.1 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2008ms
rtt min/avg/max/mdev = 11.135/11.967/13.476/1.068 ms
#Turn off network management
[root@master01 ~]# systemctl stop NetworkManager
[root@master01 ~]# systemctl disable NetworkManager
Removed symlink /etc/systemd/system/
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.

2. Create, generate and view ca certificate by downloading related command tools

First of all, on the master node, install the tools related to making and viewing certificates, such as cfssl, cfssljson, cfssl certinfo

//Create and enter the working directory by yourself
[root@master01 ~]# mkdir k8s
[root@master01 ~]# cd k8s/
#Write a script to download the above three command tools to the specified directory
[root@master01 k8s]# vim #The script is shown separately in the next code block
[root@master01 k8s]# bash 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.8M  100  9.8M    0     0   131k      0  0:01:17  0:01:17 --:--:-- 47667
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2224k  100 2224k    0     0   823k      0  0:00:02  0:00:02 --:--:--  823k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6440k  100 6440k    0     0   426k      0  0:00:15  0:00:15 --:--:--  523k

The shell script of the download command is as follows:

[root@master01 k8s]# cat 
curl -L -o /usr/local/bin/cfssl
curl -L -o /usr/local/bin/cfssljson 
curl -L -o /usr/local/bin/cfssl-certinfo 
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
#Check whether the command is successfully installed to the specified directory
[root@master01 k8s]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson

There are already tools for creating ca certificates. Now you need materials (various files) to create certificates

Create a certificate generation directory to generate various required certificates

[root@master01 k8s]# mkdir etcd-cert
[root@master01 k8s]# cd etcd-cert/
[root@master01 etcd-cert]# ls
#Write the materials required for Certificate creation - various files, which can be executed by one click of shell script. The next code block will explain the main contents and relevant parameters of the script
[root@master01 etcd-cert]# vim
#Execute script
[root@master01 etcd-cert]# bash 
2020/05/03 20:05:57 [INFO] generating a new CA key and certificate from CSR
2020/05/03 20:05:57 [INFO] generate received request
2020/05/03 20:05:57 [INFO] received CSR
2020/05/03 20:05:57 [INFO] generating key: rsa-2048
2020/05/03 20:05:57 [INFO] encoded CSR
2020/05/03 20:05:57 [INFO] signed certificate with serial number 594719677485784071979843988457153533430072455164
2020/05/03 20:05:57 [INFO] generate received request
2020/05/03 20:05:57 [INFO] received CSR
2020/05/03 20:05:57 [INFO] generating key: rsa-2048
2020/05/03 20:05:58 [INFO] encoded CSR
2020/05/03 20:05:58 [INFO] signed certificate with serial number 603393813825663113730743133623713339445920555574
2020/05/03 20:05:58 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (;
specifically, section 10.2.3 ("Information Requirements").

#View the files generated by script execution, and the following will analyze the origin and related content explanation of these files
[root@master01 etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem  server-csr.json  server.pem

shell script file and related explanations (cancel my comments after copying the script, otherwise an error will be reported)

#Write ca certificate configuration file, the first json file
cat > ca-config.json <<EOF
  "signing": {
    "default": {
      "expiry": "87600h" #Expires 10 years
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "key encipherment",
            "server auth", #The authorization verification of service and client indicates that the client can use the CA to verify the certificate provided by the server;
            "client auth" #Indicates that the server can use the CA to verify the certificate provided by the client;
#Write ca certificate signature file, the second json file
cat > ca-csr.json <<EOF
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",#Asymmetric key format
        "size": 2048 #Byte length
    "names": [
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
#The download command relies on the json file to generate the CA certificate, thus generating two PEM certificate files: ca.pem and Ca key.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#To specify the communication verification between three nodes in etcd cluster, the third json file of the verification signature file on the server side needs to be written
cat > server-csr.json <<EOF
    "CN": "etcd",
    "hosts": [
    "key": {
        "algo": "rsa",
        "size": 2048
    "names": [
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
#Use cfssl command to depend on ca certificate and configuration file, - profiles is to specify specific usage scenario, which will generate:
# server-key.pem and server.pem Certificate (private key)
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

The script is a. sh file. Three json files are written in the script. Four certificate (private key) files (one for Ca, the other for server) and corresponding signature files (ca.csr and server.csr files) are generated by commands

The above certificates we need are created. Now we start to build the etcd cluster

3. Set up etcd cluster according to certificate, software package and script

First of all, we need to download etcd software package. Here, I use the third-party tool to download it from the official. You can use it for you. The package link

Extraction code: wwz4

The ETCD package's website link is also posted: ETCD binary package address

Now that the resources are out, let's start working! C

First extract the package

[root@master01 k8s]# ls  etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz
[root@master01 k8s]# tar zxf etcd-v3.3.10-linux-amd64.tar.gz 
[root@master01 k8s]# cd etcd-v3.3.10-linux-amd64/
#Take a look at the files in the package
[root@master01 etcd-v3.3.10-linux-amd64]# ls
Documentation  etcd  etcdctl

#What we need are two command tools: etcd and etcdctl. Later, we need to move them to the cluster building directory we created. Now we create various commands, and then save or write the required files to the corresponding directory

First, create various directories for cluster building

[root@master01 ~]# mkdir /opt/etcd/{cfg,bin,ssl} -p 
#They are profile directory, command directory and certificate directory respectively
[root@master01 ~]# ls -R /opt/etcd/
bin  cfg  ssl



#There are no files in the current directory

Move related files

[root@master01 etcd-v3.3.10-linux-amd64]# mv etcd etcdctl /opt/etcd/bin/
[root@master01 etcd-v3.3.10-linux-amd64]# cd /root/k8s/etcd-cert/
[root@master01 etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem  server-csr.json  server.pem
[root@master01 etcd-cert]# cp *.pem /opt/etcd/ssl
[root@master01 etcd-cert]# ls -R /opt/etcd/
bin  cfg  ssl

etcd  etcdctl


ca-key.pem  ca.pem  server-key.pem  server.pem

Now the configuration files (including configuration files and service startup files) are poor. You can still use shell scripts (you can really appreciate the charm of shell scripts) to write them. Be sure to remove the comments before executing the script

[root@master01 etcd-cert]# cd /opt/etcd/cfg/
[root@master01 cfg]# ls
[root@master01 cfg]# vim
# example: ./ etcd01 etcd02=,etcd03=

#Position variable
#Work path
#Redirect write file
cat <<EOF >$WORK_DIR/cfg/etcd
ETCD_LISTEN_PEER_URLS="https://${etcd  IP}: 2380 " communication port between etcd cluster servers
ETCD_LISTEN_CLIENT_URLS="https://${etcd  IP}: 2379 " port of external access node

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #Token name cluster node server needs to be consistent

cat <<EOF >/usr/lib/systemd/system/etcd.service
Description=Etcd Server

ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS}, \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \


systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

At this time, you can execute the script first. The above script gives an example of executing the script, which will enter the blocking waiting node joining state. If there is no node joining in a certain period of time, the blocking state will be terminated due to timeout

[root@master01 cfg]# ./ etcd01 etcd02=,etcd03=

You can restart a terminal to check the status of etcd service

[root@master01 ~]# ps -ef | grep etcd
root      50829  15209  0 20:46 pts/1    00:00:00 /bin/bash ./ etcd01 etcd02=,etcd03=
root      50874  50829  0 20:46 pts/1    00:00:00 systemctl restart etcd
root      50880      1  4 20:46 ?        00:00:01 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls= --listen-client-urls=, --advertise-client-urls= --initial-advertise-peer-urls= --initial-clusteretcd01=,etcd02=,etcd03= --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root      50937  50896  0 20:46 pts/2    00:00:00 grep --color=auto etcd
#In a certain period of time, if no nodes join the cluster, the blocking status will be pushed out. Because other nodes have not been set up, they will exit
[root@master01 cfg]# ./ etcd01 etcd02=,etcd03=
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.

Next, we will remotely copy the files and service startup files in the etcd directory of the master node to the corresponding directories of the two node servers. Here, take the node1 node as an example

[root@master01 cfg]# scp -r /opt/etcd root@
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:bkzGRcdP2iJrSTerWtyuqSDENF2mKLWUZHMRkzJZBFI.
ECDSA key fingerprint is MD5:b0:9b:9f:31:de:da:51:8a:d3:ff:87:86:fa:19:63:2c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
root@'s password:                                                                                100% 1812     1.5MB/s   00:00    
etcd                                                                                   100%  509   583.4KB/s   00:00    
etcd                                                                                   100%   18MB 109.8MB/s   00:00    
etcdctl                                                                                100%   15MB 104.3MB/s   00:00    
ca-key.pem                                                                             100% 1679   689.5KB/s   00:00    
ca.pem                                                                                 100% 1265     1.1MB/s   00:00    
server-key.pem                                                                         100% 1675     1.1MB/s   00:00    
server.pem                                                                             100% 1338     2.2MB/s   00:00

[root@master01 cfg]# scp /usr/lib/systemd/system/etcd.service root@
root@'s password: 
etcd.service                                                                           100%  923   430.4KB/s   00:00   

Next to each node is configuration modification (mainly etcd cluster node name and ip address in the configuration file)

[root@node01 ~]# cd /opt/etcd/cfg/
[root@node01 cfg]# ls
#Mainly modify configuration file etcd
[root@node01 cfg]# vim etcd
#Modify ip address and node name


At this time, we execute the cluster building command on the master, and then quickly start the etcd service on the two node servers. After the service is started, it will automatically exit the blocking state

[root@master01 cfg]# ./ etcd01 etcd02=,etcd03=
#Two nodes start etcd service
[root@node01 cfg]# systemctl start etcd
[root@node02 cfg]# systemctl start etcd

Finally, let's check the status of etcd cluster. If the status is health, it means that the etcd cluster is built successfully

[root@master01 cfg]# cd ../ssl/ #Certificate is required to check the cluster status
[root@master01 ssl]# ls
ca-key.pem  ca.pem  server-key.pem  server.pem
[root@master01 ssl]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints=",," cluster-health 
member a25c294d3a391c7c is healthy: got healthy result from
member b2db359ffad36ee5 is healthy: got healthy result from
member eddae83baed564ba is healthy: got healthy result from
cluster is healthy

OK, etcd has been built successfully. Later, we will continue to deploy other components. Therefore, it is recommended to suspend the virtual machine as the best choice in the experimental environment. When more and more services come, restarting after shutdown may cause various problems.

Topics: SSL JSON network Linux