TL;DR

The knowledge of network is many and miscellaneous, and many of them are part of the system kernel. Originally, I didn't do network, and my knowledge of system kernel is also weak. But it is precisely these strange contents full of temptation, coupled with the current work is more related to the network, seize the opportunity to learn.

This article takes Kubernetes LoadBalancer as the starting point, uses MetalLB to realize the load balancer of the cluster, and understands some network knowledge while exploring its working principle.

Since MetalLB has a lot of content, we will only introduce the simple and easy to understand part of it step by step. There will be a next article (too complex, wait until I understand: D).

LoadBalancer type Service

Since the IP address of Pod in kubernets is not fixed, the IP will change after restart and cannot be used as the communication address. Kubernets provides a Service to solve this problem and expose it to the public.

Kubernetes provides the same DNS name and virtual IP for a group of pods, as well as the ability of load balancing. Here, the grouping of pods is completed by labeling pods. When defining a Service, a Label selector will be declared to associate the Service with this group of pods.

According to different usage scenarios, services are divided into four types: ClusterIP, NodePort, LoadBalancer and ExternalName. The default is ClusterIP. We won't introduce them in detail here. Please check them if you are interested Service official document.

Except for today's protagonist LoadBalancer, the other three types are commonly used. The official explanation of LoadBalancer is:

Expose services externally using the load balancer of the cloud provider. The external load balancer can route traffic to the automatically created NodePort service and ClusterIP service.

When you see the words "provided by the cloud provider", you often flinch. Sometimes you need the LoadBalancer to do some verification work on the external exposure Service (although you can also use the NodePort type Service in addition to the 7-layer progress), but Kubernetes officially does not provide the implementation. For example, the following will be introduced MetalLB It's a good choice.

MetalLB introduction

MetalLB is the load balancer implementation of bare metal Kubernetes cluster, using standard routing protocol.

Note: MetalLB is still in beta.

As mentioned earlier, Kubernetes officials do not provide the implementation of LoadBalancer. Various cloud vendors provide implementations, but if they are not running on these cloud environments, the created LoadBalancer Service will always be in Pending state (see the Demo below).

MetalLB provides two functions:

• Address assignment: when creating LoadBalancer Service, MetalLB will assign IP address to it. This IP address is obtained from the < U > pre configured IP address library < / u >. Similarly, after the Service is deleted, the assigned IP address will return to the address library.
• External broadcasting: after the IP address is allocated, the network outside the cluster needs to know the existence of this address. MetalLB uses standard routing protocols: ARP, NDP or BGP.

There are two ways of broadcasting. The first is Layer 2 mode, which uses ARP (ipv4) / NDP (ipv6) protocol; The second is BPG.

<u> Today, we mainly introduce the simple Layer 2 mode < / u >, which is the implementation of OSI Layer 2 as the name suggests.

The specific implementation principle will be analyzed after reading the Demo. If you can't wait, please jump to the end directly.

Runtime

MetalLB runs with two workloads:

• Controller: Deployment, which is used to listen for Service changes and allocate / reclaim IP addresses.
• Speaker: DaemonSet, the IP address of the external broadcast Service.

Demo

Before installation, introduce the network environment. Kubernetes uses K8s to install in Virtual machine for Proxmox Come on.

Install K3s

To install k3s, you need to disable the default servicelb of k3s through -- disable servicelb.

reference resources K3s document , K3s is used by default Traefik Inress controller and Klipper Service load balancer to expose services.

curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable servicelb --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config

Create workload

Using nginx mirroring, create two workloads:

kubectl create deploy nginx --image nginx:latest --port 80 -n default
kubectl create deploy nginx2 --image nginx:latest --port 80 -n default

Create services for two deployments at the same time. Here, select LoadBalancer as the type:

kubectl expose deployment nginx --name nginx-lb --port 8080 --target-port 80 --type LoadBalancer -n default
kubectl expose deployment nginx2 --name nginx2-lb --port 8080 --target-port 80 --type LoadBalancer -n default

Check the Service and find that the status is Pending. This is because we disabled the implementation of LoadBalancer when installing K3s:

kubectl get svc -n default
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP      10.43.0.1       <none>        443/TCP          14m
nginx-lb     LoadBalancer   10.43.108.233   <pending>     8080:31655/TCP   35s
nginx2-lb    LoadBalancer   10.43.26.30     <pending>     8080:31274/TCP   16s

At this time, MetalLB is needed.

Install MetalLB

Use the officially provided manifest to install. At present, the latest version is 0.12.1. In addition, other installation methods can be selected, such as Helm,Kustomize perhaps MetalLB Operator.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

kubectl get po -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
speaker-98t5t                 1/1     Running   0          22s
controller-66445f859d-gt9tn   1/1     Running   0          22s

At this time, check that the status of LoadBalancer Service is still Pending, huh? Because MetalLB needs to assign IP addresses to the Service, but the IP addresses do not come out of thin air, but need to provide an address Library in advance.

Here, we use the Layer 2 mode to provide an IP segment through Configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.30-192.168.1.49

At this time, check the status of the Service. You can see that MetalLB has assigned IP addresses 192.168.1.30 and 192.168.1.31 to the two services:

kubectl get svc -n default
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)          AGE
kubernetes   ClusterIP      10.43.0.1       <none>         443/TCP          28m
nginx-lb     LoadBalancer   10.43.201.249   192.168.1.30   8080:30089/TCP   14m
nginx2-lb    LoadBalancer   10.43.152.236   192.168.1.31   8080:31878/TCP   14m

You can request the following tests:

curl -I 192.168.1.30:8080
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Wed, 02 Mar 2022 15:31:15 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
Connection: keep-alive
ETag: "61f01158-267"
Accept-Ranges: bytes

curl -I 192.168.1.31:8080
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Wed, 02 Mar 2022 15:31:18 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
Connection: keep-alive
ETag: "61f01158-267"
Accept-Ranges: bytes

mac OS uses arp -a locally. You can find the two IP addresses and mac addresses by looking at the ARP table. You can see that both IP addresses are bound to the same network card and the IP address of the virtual machine. That is, three IPS are bound to en0 of the virtual machine:

Go to the virtual machine (node) to check the network card (here you can only see the IP bound by the system):

How Layer 2 works

The Speaker workload in Layer 2 is of DeamonSet type, and a Pod is scheduled on each node. First, several pods will be elected first to elect leaders. The Leader obtains all LoadBalancer type services and binds the assigned IP address to the current host network card. In other words, the IP addresses of all LoadBalancer type services are bound to the network card of the same node at the same time.

When an external host has a request to send to a Service in the cluster, it is necessary to first determine the mac address of the network card of the target host (for why, refer to Wikipedia ). This is by sending an ARP request, and the Leader node will respond with its mac address. The external host will cache it in the local ARP table and get it directly from the ARP table next time.

After the request reaches the node, the node will request the load balancing target Pod through Kube proxy. Therefore, if the Service is multi Pod, it may jump to another host.

Advantages and disadvantages

The advantages are obvious and easy to implement (compared with another BGP mode, the router should support BPG). Just like the author's environment, as long as the IP address library and the cluster are the same network segment.

Of course, the disadvantages are more obvious. The bandwidth of the Leader node will become a bottleneck; At the same time, availability is poor and failover takes 10 seconds( Each speaker process has a 10s cycle).

reference resources

• Address resolution protocol

• MetalLB concept

  The article is unified in the official account of the cloud.