When we view and modify Kubernetes resources through kubectl, have you ever thought about the interface behind? Is there any way to probe these interactive data?

The interface between kuberentes client and server is based on http protocol. Therefore, we only need to be able to capture and parse https traffic, and we can see kubernetes API traffic.

However, because kubenetes uses the client private key to authenticate the client, the configuration of packet capturing is a little more complex. The specific structure is as follows:

If you want to know more about Kubernetes certificates, you can take a look This article on Kubernetes certificate parsing

Extract the client certificate and private key from kubeconfig

kubeconfig contains the certificate and private key of the client. We must extract them first:

# Extract client certificate
grep client-certificate-data ~/.kube/config | \
  awk '{ print $2 }' | \
  base64 --decode > client-cert.pem
# Extract client private key
grep client-key-data ~/.kube/config | \
  awk '{ print $2 }' | \
  base64 --decode > client-key.pem
# Extract the server CA certificate
grep certificate-authority-data ~/.kube/config | \
  awk '{ print $2 }' | \
  base64 --decode > cluster-ca-cert.pem

Reference from Reddit

Configure Charles agent software

As can be seen from the first figure, the proxy software has two functions: one is to receive https traffic and forward it; the other is to use the specified client private key when forwarding to kubernetes apiserver.

First, configure Charles to intercept all https traffic:

Then configure the client private key, that is, for requests sent to apiserver, uniformly use the specified client private key for authentication:

Configure kubectl

To capture kubectl traffic, two conditions are required: 1. kubectl uses Charles as the proxy, and 2. kubectl needs to trust Charles' certificate.

# Charles's proxy port is 8888. Set https_proxy environment variable, let kubectl use Charles proxy
$ export https_proxy=http://127.0.0.1:8888/
# Secure skip TLS verify indicates that the server certificate is not verified
$ kubectl --insecure-skip-tls-verify get pod
NAME                    READY   STATUS    RESTARTS   AGE
sc-b-7f5dfb694b-xtfrz   2/2     Running   0          2d20h

We can see the network request of get pod:

You can see that the endpoint of get pod is get / API / V1 / namespaces / < namespace > / pods.

Let's try the following request to create a pod:

$ cat <<EOF >pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-robberphex
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
EOF
$ kubectl --insecure-skip-tls-verify apply -f pod.yaml
pod/nginx-robberphex created

You can also catch the bag:

The endpoint for creating a pod is post / API / V1 / namespaces / < namespace > / pods

Configure kubenetes client

Let's start by writing an example of obtaining a pod with kubernetes go client (note that all certificates have been trusted in the code, so you can catch the package):

package main

/*
require (
	k8s.io/api v0.18.19
	k8s.io/apimachinery v0.18.19
	k8s.io/client-go v0.18.19
)
*/
import (
	"context"
	"flag"
	"fmt"
	"path/filepath"

	apiv1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
)

func main() {
	ctx := context.Background()
	var kubeconfig *string
	if home := homedir.HomeDir(); home != "" {
		kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
	} else {
		kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
	}
	flag.Parse()

	config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
	if err != nil {
		panic(err)
	}
	// Let clientset trust all certificates
	config.TLSClientConfig.CAData = nil
	config.TLSClientConfig.Insecure = true
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		panic(err)
	}
	podClient := clientset.CoreV1().Pods(apiv1.NamespaceDefault)
	podList, err := podClient.List(ctx, metav1.ListOptions{})
	if err != nil {
		panic(err)
	}

	for _, pod := range podList.Items {
		fmt.Printf("podName: %s\n", pod.Name)
	}

	fmt.Println("done!")
}

Then compile and execute:

$ go build -o kube-client
$ export https_proxy=http://127.0.0.1:8888/
$ ./kube-client
podName: nginx-robberphex
podName: sc-b-7f5dfb694b-xtfrz
done!

The same result can be caught at this time:

Based on this, we can analyze what Kubernetes does, and it is also the entrance for us to analyze the implementation of Kubernetes.