Virtual machine creation
Use Parallels Desktop to create three virtual machines in your Mac system. The details are as follows:
CentOS7-Node1: 10.211.55.7 parallels/centos-testCentOS7-Node2: 10.211.55.8 parallels/centos-testCentOS7-Node3: 10.211.55.9 parallels/centos-test
Master installation
Select the CentOS7-Node1 machine as the Master node.
Configure yum
Update yum source:
[parallels@CentOS7-Node1 yum.repos.d]$ cd /etc/yum.repos.d[parallels@CentOS7-Node1 yum.repos.d]$ sudo touch kubernetes.repo[kubernetes]name=Kubernetesbaseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=0repo_gpgcheck=0
Installing the Kubernetes environment
After evaluation, it is recommended to use kubedm to build a cluster, and so is the company's cluster. So don't worry about using kubedm.
[parallels@CentOS7-Node1 yum.repos.d]$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetesLoaded plugins: fastestmirror, langpacksYou need to be root to perform this command.[parallels@CentOS7-Node1 yum.repos.d]$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetesLoaded plugins: fastestmirror, langpacksLoading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.comkubernetes | 1.4 kB 00:00:00 kubernetes/primary | 58 kB 00:00:00 kubernetes 421/421Resolving Dependencies--> Running transaction check...... # Omit a bunch of meaningless logs dependency installed: conntrack-tools.x86_ 64 0:1.4.4-5.el7_ 7.2 cri-tools.x86_ 64 0:1.13.0-0 kubernetes-cni.x86_ 64 0:0.7.5-0 libnetfilter_ cthelper.x86_ 64 0:1.0.0-10.el7_ 7.1 libnetfilter_ cttimeout.x86_ 64 0:1.0.0-6.el7_ 7.1 libnetfilter_ queue.x86_ 64 0:1.0.2-2.el7_ 2 socat.x86_ 64 0:1.7.3.2-2.el7 Complete!
About configuring and upgrading yum
yum install -y yum-utils device-mapper-persistent-data lvm2yum update
Start docker
Start Docker and add the start maneuver item:
[parallels@CentOS7-Node1 ~]$ sudo systemctl enable docker && systemctl start docker[sudo] password for parallels: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
Launch kubelet
Start kubelet and add startup items:
sudo systemctl enable kubelet && systemctl start kubelet[parallels@CentOS7-Node1 ~]$ sudo systemctl enable kubelet && systemctl start kubeletCreated symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===Authentication is required to manage system services or units.Authenticating as: Parallels (parallels)Password: ==== AUTHENTICATION COMPLETE ===
kubeadm config
[parallels@CentOS7-Node1 Workspace]$ kubeadm config print init-defaultsapiVersion: kubeadm.k8s.io/v1beta2bootstrapTokens:- groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authenticationkind: InitConfigurationlocalAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443nodeRegistration: criSocket: /var/run/dockershim.sock name: centos7-node1 taints: - effect: NoSchedule key: node-role.kubernetes.io/master---apiServer: timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta2certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: type: CoreDNSetcd: local: dataDir: /var/lib/etcdimageRepository: k8s.gcr.iokind: ClusterConfigurationkubernetesVersion: v1.16.0networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12scheduler: {}kubeadm config print init-defaults > /home/parallels/Workspace/init.default.yaml
Configure Docker
First, install the Docker environment. Please refer to the previous http://www.cyblogs.com/centos7shang-an-zhuang-docker/
Some related commands of Docker
yum install docker-ce-18.09.9-3.el7 # The specified version is 18.09.9-3.el7systemctl status dockersystemctl restart dockersystemctl daemon reload
Download the relevant images of kubernetes
Configure the mirror address, but it's useless. Domestic images are needed later:
echo '{"registry-mirrors":["https://Docker.mirrors.ustc.edu.cn "]} '> / etc / docker / daemon.json# if you are prompted that you do not have permission, add vim manually. Then restart the docker serviceTake a look at the image name and version that kubernetes depends on
[parallels@CentOS7-Node1 Workspace]$ kubeadm config images listW1022 13:51:12.550171 19704 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)W1022 13:51:12.550458 19704 version.go:102] falling back to the local client version: v1.16.2k8s.gcr.io/kube-apiserver:v1.16.2k8s.gcr.io/kube-controller-manager:v1.16.2k8s.gcr.io/kube-scheduler:v1.16.2k8s.gcr.io/kube-proxy:v1.16.2k8s.gcr.io/pause:3.1k8s.gcr.io/etcd:3.3.15-0k8s.gcr.io/coredns:1.6.2
If the network is OK, you should directly execute this command, but an error will be reported.
[parallels@CentOS7-Node1 Workspace]$ sudo kubeadm config images pull --config=/home/parallels/Workspace/init.default.yaml# Due to the problem of pulling the image from the network, it is basically impossible to operate. You can only go to aliyun to get it and then modify the tag. The error is as follows[ parallels@CentOS7 -Node1 Workspace]$ sudo kubeadm config images pull --config=/home/parallels/Workspace/init.default.yaml[sudo] password for parallels: failed to pull image "k8s.gcr.io/kube-apiserver:v1.16.0 ": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), error: exit status 1To see the stack trace of this error execute with --v=5 or higher
Get image
Obtain the image in another way:
touch kubeadm.sh#!/bin/bashKUBE_VERSION=v1.16.0KUBE_PAUSE_VERSION=3.1ETCD_VERSION=3.3.15-0CORE_DNS_VERSION=1.6.2GCR_URL=k8s.gcr.ioALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containersimages=( kube-apiserver:${KUBE_VERSION} kube-controller-manager:${KUBE_VERSION} kube-scheduler:${KUBE_VERSION} kube-proxy:${KUBE_VERSION} pause:${KUBE_PAUSE_VERSION} etcd:${ETCD_VERSION} coredns:${CORE_DNS_VERSION})for imageName in ${images[@]} ; do docker pull $ALIYUN_URL/$imageName docker tag $ALIYUN_URL/$imageName $GCR_URL/$imageName docker rmi $ALIYUN_URL/$imageNamedonePull image
chmod u+x kubeadm.sh # Add permission sudo. / kubedm.sh
The rest is to wait patiently
View the final local image
[root@CentOS7-Node1 Workspace]# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEk8s.gcr.io/kube-apiserver v1.16.0 b305571ca60a 4 weeks ago 217MBk8s.gcr.io/kube-proxy v1.16.0 c21b0c7400f9 4 weeks ago 86.1MBk8s.gcr.io/kube-controller-manager v1.16.0 06a629a7e51c 4 weeks ago 163MBk8s.gcr.io/kube-scheduler v1.16.0 301ddc62b80b 4 weeks ago 87.3MBk8s.gcr.io/etcd 3.3.15-0 b2756210eeab 6 weeks ago 247MBk8s.gcr.io/coredns 1.6.2 bf261d157914 2 months ago 44.1MBk8s.gcr.io/pause 3.1 da86e6ba6ca1 22 months ago 742kB
[parallels@CentOS7-Node1 Workspace]$ sudo kubeadm init --config=init.default.yaml [init] Using Kubernetes version: v1.16.0[preflight] Running pre-flight checks [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`To see the stack trace of this error execute with --v=5 or higher
Turn off firewall
To solve the firewall problem, see: http://www.cyblogs.com/centos7cha-kan-he-guan-bi-fang-huo-qiang/
cgroupfs error
detected "cgroupfs" as the Docker cgroup driver
New: / etc/docker/daemon.json
{ "registry-mirrors": [ "https://Registry. Docker CN. Com "]," live restore ": true," exec opts ": [" native. Cgroupdriver = SYSTEMd "# modify user]}# restart dockersystemctl status dockerDisable swap
Or do you find it necessary to disable swap
Oct 22 16:35:36 CentOS7-Node1 kubelet[1395]: F1022 16:35:36.065168 1395 server.go:271] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename Type Size Used Priority /dev/dm-1 partition 2097148 29952 -1]
swapoff -a#To permanently disable the swap partition, open the following file and comment out the swap line sudo vi /etc/fstab
Start kubedm init again
kubeadm init --config=init.default.yaml[init] Using Kubernetes version: v1.16.2...[preflight] Pulling images required for setting up a Kubernetes cluster...[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"...[certs] Using certificateDir folder "/etc/kubernetes/pki"...[kubeconfig] Using kubeconfig folder "/etc/kubernetes"...[kubelet-check] Initial timeout of 40s passed.[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
An error occurs. If you continue to execute after changing the Docker version, an error will still be reported.
[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
Reset kubedm
You need to reset kubedm here. The specific operations are as follows:
kubeadm resetecho '1' > /proc/sys/net/bridge/bridge-nf-call-iptables echo '1' > /proc/sys/net/ipv4/ip_forward
journalctl viewing logs
journalctl -xefu kubelet
There will still be errors here, because the previous
apiVersion: kubeadm.k8s.io/v1beta2kind: ClusterConfigurationimageRepository: k8s.gcr.iokubernetesVersion: v1.16.0networking: dnsDomain: cluster.local serviceSubnet: "10.96.0.0/16"
Continue the init process. Kubedm init -- config = / home / parallels / workspace / init.default.yaml
[addons] Applied essential addon: CoreDNS[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 10.211.55.7:6443 --token imwj34.ksfiwzj5ga80du0r \ --discovery-token-ca-cert-hash sha256:7ffef85880ed43dd539afa045715f9ad5bef15e904cede96213d6cfd4adb0795
It's really not easy. It's always executed over and over again. As long as it's the version of images and the init process, it's easy to make mistakes.
Verify configmap
[root@CentOS7-Node1 ~]# kubectl get -n kube-system configmap NAME DATA AGEcoredns 1 5m49sextension-apiserver-authentication 6 5m53skube-proxy 2 5m49skubeadm-config 2 5m50skubelet-config-1.16 1 5m50s
Install the Node and join the cluster
Install the basic environment that has been with the Master, including docker, kubelet, kubedm, etc. repeat the above actions.
scp root@10.211.55.7:/home/parallels/Workspace/init.default.yaml .scp root@10.211.55.7:/home/parallels/Workspace/kubeadm.sh .yum install docker-ce-18.06.3.ce-3.el7
Generate a configuration file for the kubedm command and create join-config.yaml as follows:
apiVersion: kubeadm.k8s.io/v1beta2kind: JoinConfigurationdiscovery: bootstrapToken: apiServerEndpoint: 10.211.55.7:6443 token: imwj34.ksfiwzj5ga80du0r unsafeSkipCAVerification: true tlsBootstrapToken: imwj34.ksfiwzj5ga80du0r
Among them, the value of apiServerEndpoint comes from the server address of the Master, which is 10.211.55.7. The values of token and tlsBootstrapToken come from the last line of prompt message when kubedm init installs the Master. Be sure to pay attention to the format of yaml file here, otherwise an error will be reported during execution.
[root@CentOS7-Node2 Workspace]# kubeadm join --config=join-config.yaml[preflight] Running pre-flight checkserror execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`To see the stack trace of this error execute with --v=5 or higher[root@CentOS7-Node2 Workspace]# swapoff -a[root@CentOS7-Node2 Workspace]# kubeadm join --config=join-config.yaml[preflight] Running pre-flight checks[preflight] Reading configuration from the cluster...[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Activating the kubelet service[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Install network plug-in
Go to the Master machine and execute:
[root@CentOS7-Node1 Workspace]# kubectl get nodesNAME STATUS ROLES AGE VERSIONcentos7-node1 NotReady master 154m v1.16.2centos7-node2 NotReady <none> 2m49s v1.16.2
The NotReady status is shown here because the CNI network plug-in has not been installed. We choose the weave plug-in to install it.
[root@CentOS7-Node1 Workspace]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"serviceaccount/weave-net createdclusterrole.rbac.authorization.k8s.io/weave-net createdclusterrolebinding.rbac.authorization.k8s.io/weave-net createdrole.rbac.authorization.k8s.io/weave-net createdrolebinding.rbac.authorization.k8s.io/weave-net createddaemonset.apps/weave-net created
Verify that the cluster installation is complete
[root@CentOS7-Node1 Workspace]# kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-5644d7b6d9-9fr9p 0/1 ContainerCreating 0 172mcoredns-5644d7b6d9-pmpkq 0/1 ContainerCreating 0 172metcd-centos7-node1 1/1 Running 0 171mkube-apiserver-centos7-node1 1/1 Running 0 171mkube-controller-manager-centos7-node1 1/1 Running 0 171mkube-proxy-ccnht 1/1 Running 0 21mkube-proxy-rdq9l 1/1 Running 0 172mkube-scheduler-centos7-node1 1/1 Running 0 171mweave-net-6hw26 2/2 Running 0 8m7sweave-net-qv8vz 2/2 Running 0 8m7s
It is found that coredns is always in ContainerCreating state. Please see the error message for details.
[root@CentOS7-Node1 Workspace]# kubectl describe pod coredns-5644d7b6d9-9fr9p -n kube-systemName: coredns-5644d7b6d9-9fr9pNamespace: kube-systemPriority: 2000000000Priority Class Name: system-cluster-criticalNode: centos7-node2/10.211.55.8Start Time: Tue, 22 Oct 2019 20:49:47 +0800Labels: k8s-app=kube-dns pod-template-hash=5644d7b6d9.... # Some events are omitted here: type reason age from message ----------------------- warning failedscheduling < unknown > Default scheduler 0 / 1 nodes are available: 1 node (s) had taints that the pod didn't tolerate. Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate. Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/coredns-5644d7b6d9-9fr9p to centos7-node2 Warning FailedCreatePodSandBox 2m kubelet, centos7-node2 Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceede d Normal SandboxChanged 119s kubelet, centos7-node2 Pod sandbox changed, it will be killed and re-created.
Some errors can be seen here:
Oct 22 10:50:15 CentOS7-Node1 kubelet[7649]: F1022 10:50:15.170550 7649 server.go:196] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", Oct 22 10:50:15 CentOS7-Node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
You can delete a pod and restart it:
[root@CentOS7-Node1 ~]# kubectl delete pod coredns-5644d7b6d9-9fr9p -n kube-systempod "coredns-5644d7b6d9-9fr9p" deleted
After reading too many articles and blogs, I found that few of them were written completely. They were all successful experiences. In fact, I didn't know there were all kinds of strange problems. To tell the truth, k8s is very convenient, but the threshold is very high, and there are too many things to rely on. Especially the problems caused by version problems are difficult to solve.
Finally, take a look at the successful pictures
Summary of common commands
systemctl daemon-reloadsystemctl restart kubeletkubectl get pods -n kube-systemkubectl describe pod coredns-5644d7b6d9-lqtks -n kube-systemkubectl delete pod coredns-5644d7b6d9-qh4bc -n kube-system# allow master Node deployment podkubectl taint nodes --all node-role.kubernetes.io/master-# Prohibit master from deploying podkubectl taint nodes k8s node role. Kubernetes. IO / Master = true: noschedulekubeadm resetsystemctl enable docker & & systemctl start dockersystemctl enable kubelet & & systemctl start kubeletjournalctl - xefu kubelet
Reference address
- https://www.cnblogs.com/yufeng218/p/8370670.html
- https://stackoverflow.com/questions/55531834/kubeadm-fails-to-initialize-when-kubeadm-init-is-called
- https://zhuanlan.zhihu.com/p/31398416
- https://blog.csdn.net/M82_A1/article/details/97626309
- https://blog.csdn.net/liumiaocn/article/details/99608323
- https://www.hi-linux.com/posts/54191.html
- https://blog.csdn.net/BigData_Mining/article/details/88683459