Installing etcd cluster

Posted by aximbigfan on Tue, 14 May 2019 22:41:01 +0200

kuberntes system uses etcd to store all data, which is one of the most important components. Note that the etcd cluster can only have an odd number of nodes (1,3,5...). This document uses three nodes to cluster.

I. Basic Environment

software package

etcd download address: https://github.com/coreos/etcd/releases

The server

Architecture diagram

Generating etcd certificates and private keys

Create etcd configuration file

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.11",
    "192.168.1.12",
    "192.168.1.13",
    "etcd1",
    "etcd2",
    "etcd3"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "Ctyun",
      "OU": "ops"
    }
  ]
}

hosts: The etcd node IP (which should contain the IP and hostname of all nodes in the cluster) that is authorized to use the certificate
C: state
ST: provinces
L: City
O: Company
OU: Department

Generate private key, certificate request file, certificate

CA certificates are created in previous chapters Kubernetes Certificate Relevance (CFSSL)

cfssl gencert -ca=/opt/ssl/k8sca/ca.pem \
-ca-key=/opt/ssl/k8sca/ca-key.pem \
-config=/opt/ssl/k8sca/ca-config.json \
-profile=kubernetes /opt/ssl/etcd/etcd-csr.json | cfssljson -bare etcd

- ca: Specify CA certificate path
- ca-key: Specify the path of CAKey
- config: Specify CA Certificate Signing Policy Profile

Start and configure etcd

Create etcd configuration file etcd.conf, which defines some variables to facilitate direct reference in etcd.service file and later maintenance.
All the meaning of the configuration file is explained in detail after the etcd.service configuration file is created.

mkdir -p /etc/etcd
vim /etc/etcd/etcd.conf
# [member]
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.11:2380" 
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.11:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.11:2379"

Create etcd.service configuration file

Variables of etcd.service are references to etcd configuration files

mkdir /var/lib/etcd
vim /usr/lib/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR} 
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
  --name=etcd1 \
  --cert-file=/opt/ssl/etcd/etcd.pem \
  --key-file=/opt/ssl/etcd/etcd-key.pem \
  --peer-cert-file=/opt/ssl/etcd/etcd.pem \
  --peer-key-file=/opt/ssl/etcd/etcd-key.pem \
  --trusted-ca-file=/opt/ssl/k8sca/ca.pem \
  --peer-trusted-ca-file=/opt/ssl/k8sca/ca.pem \
  --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster etcd1=https://192.168.1.11:2380,etcd2=https://192.168.1.12:2380,etcd3=https://192.168.1.13:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

--initial-cluster: Other nodes in the cluster
- cert-file: etcd certificate path
key-file: etcd private key path
peer-cert-file: peer-to-peer certificate (two-way certificate) path
peer-key-file: peer-to-peer certificate (two-way certificate) private key path
trusted-ca-file: CA certificate path as client
peer-trusted-ca-file: CA certificate path for peer certificates
- initial-advertise-peer-urls: Lists the URL s for communication among cluster members to notify other cluster members
listen-peer-urls: A list of URL s used to listen for other members of the cluster
listen-client-urls: List of URL s used to listen for client communications
- advertise-client-urls: Notify the client's URL for listing all clients
- initial-cluster-token: the initial cluster token of the etcd cluster, the server must pass the token to join the etcd cluster

Start the etcd cluster

All nodes in the cluster are configured with configuration files and started at the same time.

systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

Configuration without variables

Previously, we created two configuration files, / etc/etcd/etcd.conf and / var/lib/system/system/etcd.service respectively, and etcd.service refers to the variables redefined by etcd.conf.

If you don't want to refer to variables, you don't write etcd.conf. Just create etcd.service as follows

[root@etcd1 k8sca]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name etcd1 \
  --cert-file=/opt/ssl/etcd/etcd.pem \
  --key-file=/opt/ssl/etcd/etcd-key.pem \
  --peer-cert-file=/opt/ssl/etcd/etcd.pem \
  --peer-key-file=/opt/ssl/etcd/etcd-key.pem \
  --trusted-ca-file=/opt/ssl/k8sca/ca.pem \
  --peer-trusted-ca-file=/opt/ssl/k8sca/ca.pem \
  --initial-advertise-peer-urls=https://192.168.1.11:2380 \
  --listen-peer-urls=https://192.168.1.11:2380 \
  --listen-client-urls=https://192.168.1.11:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.1.11:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster etcd1=https://192.168.1.11:2380,etcd2=https://192.168.1.12:2380,etcd3=https://192.168.1.13:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

III. Testing Cluster Status

All hosts in the cluster have the same configuration, only IP addresses are different.

$ etcdctl --ca-file=/opt/ssl/k8sca/ca.pem --cert-file=/opt/ssl/etcd/etcd.pem --key-file=/opt/ssl/etcd/etcd-key.pem cluster-health
member aa869cb0f2e7ed31 is healthy: got healthy result from https://192.168.1.11:2379
member b08a644fd7247c5e is healthy: got healthy result from https://192.168.1.13:2379
member bb9bd2baaebf7d95 is healthy: got healthy result from https://192.168.1.12:2379

Common problem

publish error: etcdserver: request timed out
When I deployed it, I used one to test it. No matter how I started etcd, it couldn't start. It prompted publish error: etcdserver: request timed out. Later, it was found that other etcd hosts were established in etcd.service, so when it was not possible to connect other hosts to start one, etcd failed to start. So we need to configure all hosts in the etcd cluster and start OK at the same time.

Reference resources

http://blog.51cto.com/sgk2011/2108542
https://github.com/gjmzj/kubeasz/blob/master/docs/setup/02-install_etcd.md
https://blog.csdn.net/qq_33199919/article/details/80623055
https://skyao.gitbooks.io/learning-etcd3/content/

Topics: Linux SSL network github Kubernetes

Programmer Think