About the fact that Kubernetes' Secret is not secure

Posted by bertfour on Sat, 18 Dec 2021 04:46:44 +0100

K8s provides Secret resources for us to save and set some sensitive information, such as API endpoint address, various user passwords or token s. When k8s is not used, this information may be set during deployment through configuration files or environment variables.

However, Secret is not secure. Anyone who has checked Secret with kubectl knows that we can easily see the original text of Secret as long as we have relevant permissions. Although its content is base64 encoded, it is basically equivalent to plaintext.

Therefore, the K8s native Secret is very simple and not particularly suitable for direct use in large companies. It also poses a great challenge to RBAC. Many people who should not see clear text information may be able to see it.

Especially now many companies adopt the so-called GitOps concept, and many things need to be put into VCS, such as git. This problem is becoming more and more prominent, because VCS also needs to set necessary permissions.

problem

In short, there are several places where people who should not see Secret content can get Secret content:

  • etcd storage
  • Via API server
  • View files directly on node

Here we take this example to see the use of Secret in K8s.

Secret definition, user name and password are admin and hello secret respectively:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: aGVsbG8tc2VjcmV0Cg==

Pod definition. Here, we add Secret as volume mount to the container.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: docker.io/containerstack/alpine-stress
    command:
      - top
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

After the Pod is started, we can go to the container to view the contents of the file after the Secret is added to the container as volume mount.

$ kubectl exec -it mypod sh
/ # cd /etc/foo/
/etc/foo # ls -tal
total 4
drwxr-xr-x    1 root     root          4096 Apr 14 08:55 ..
drwxrwxrwt    3 root     root           120 Apr 14 08:55 .
drwxr-xr-x    2 root     root            80 Apr 14 08:55 ..2021_04_14_08_55_54.401661151
lrwxrwxrwx    1 root     root            31 Apr 14 08:55 ..data -> ..2021_04_14_08_55_54.401661151
lrwxrwxrwx    1 root     root            15 Apr 14 08:55 password -> ..data/password
lrwxrwxrwx    1 root     root            15 Apr 14 08:55 username -> ..data/username
/etc/foo # ls -tal ..2021_04_14_08_55_54.401661151
total 8
drwxr-xr-x    2 root     root            80 Apr 14 08:55 .
drwxrwxrwt    3 root     root           120 Apr 14 08:55 ..
-rw-r--r--    1 root     root            13 Apr 14 08:55 password
-rw-r--r--    1 root     root             5 Apr 14 08:55 username
/etc/foo # cat password
hello-secret
/etc/foo #

etcd storage

The resources in the API server are saved in etcd. We can directly see the relevant contents from the file:

# hexdump -C /var/lib/etcd/member/snap/db | grep -A 5 -B 5 hello
00043640  12 00 1a 07 64 65 66 61  75 6c 74 22 00 2a 24 32  |....default".*$2|
00043650  35 66 37 35 38 30 38 2d  37 33 31 33 2d 34 38 64  |5f75808-7313-48d|
00043660  39 2d 39 61 38 65 2d 38  61 35 66 66 32 32 63 64  |9-9a8e-8a5ff22cd|
00043670  64 35 39 32 00 38 00 42  08 08 98 dc da 83 06 10  |d592.8.B........|
00043680  00 7a 00 12 19 0a 08 70  61 73 73 77 6f 72 64 12  |.z.....password.|
00043690  0d 68 65 6c 6c 6f 2d 73  65 63 72 65 74 0a 12 11  |.hello-secret...|
000436a0  0a 08 75 73 65 72 6e 61  6d 65 12 05 61 64 6d 69  |..username..admi|
000436b0  6e 1a 06 4f 70 61 71 75  65 1a 00 22 00 00 00 00  |n..Opaque.."....|
000436c0  00 00 00 08 95 5f 00 00  00 00 00 00 00 00 0a 37  |....._.........7|
000436d0  2f 72 65 67 69 73 74 72  79 2f 73 65 72 76 69 63  |/registry/servic|
000436e0  65 73 2f 65 6e 64 70 6f  69 6e 74 73 2f 6b 75 62  |es/endpoints/kub|

It can be seen that the content in the basic yaml is stored in plaintext and is the content after base64 decoding.

Similar results can be obtained using the following command.

$ ETCDCTL_API=3 etcdctl get --prefix /registry/secrets/default/mysecret | hexdump -C

Etcd originally stores plaintext data. It seems that encrypted storage has been supported since 1.7, and direct access to etcd is not so easy physically.

API server

It is much simpler to access API server. As long as you have permission, you can access API server from any node to get the clear text content of secret.

$ kubectl get secret mysecret -o yaml

apiVersion: v1
data:
  password: aGVsbG8tc2VjcmV0Cg==
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: "2021-04-14T08:55:52Z"
  name: mysecret
  namespace: default
  resourceVersion: "2196"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: 25f75808-7313-48d9-9a8e-8a5ff22cdd59
type: Opaque

On node

You can also see the contents of the Secret file on the node.

Find the mount point for foo volume:

# mount | grep foo 
tmpfs on /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo type tmpfs (rw,relatime)

View the contents of the file below this volume:

# ls -tal /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo
total 4
drwxrwxrwt 3 root root  120 4 June 14-16:55 .
drwxr-xr-x 2 root root   80 4 June 14-16:55 ..2021_04_14_08_55_54.401661151
lrwxrwxrwx 1 root root   31 4 June 14-16:55 ..data -> ..2021_04_14_08_55_54.401661151
lrwxrwxrwx 1 root root   15 4 June 14-16:55 password -> ..data/password
lrwxrwxrwx 1 root root   15 4 June 14-16:55 username -> ..data/username
drwxr-xr-x 4 root root 4096 4 June 14-16:55 ..

# cat /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo/password
hello-secret

Third party solutions

For the above-mentioned points that may leak Secret, it is not difficult to think of the following solutions:

  • etcd encryption
  • API server strictly carries out permission design
  • Strengthen node user authority management and system security

However, to ensure the absolute security of Secret, the above schemes are necessary. The lack of which is equivalent to leaving a hole in the wall.

Both community and public cloud providers have some products and solutions, which we can refer to.

  • shyiko/kubesec: Secure Secret management for Kubernetes (with gpg, Google Cloud KMS and AWS KMS backends)
  • bitnami-labs/sealed-secrets: A Kubernetes controller and tool for one-way encrypted Secrets
  • Vault by HashiCorp
  • mozilla/sops
  • Kubernetes External Secrets
  • Kamus

shyiko/kubesec

kubesec only encrypts / decrypts the data in Secret and supports the following key management services or software:

  • AWS Key Management Service
  • Google Cloud KMS
  • GnuPG

bitnami-labs/sealed-secrets

Bitnami is also a well-known company in the field of K8s and outputs a lot of technologies and best practices.

This picture is from Sealed Secrets: Protecting your passwords before they reach Kubernetes

SealeSecret saves the whole encryption of the secret resource as SealedSecret resource, and decryption can only be performed by the controller in the cluster.

SealeSecret provides a kubeseal tool to encrypt secret resources. This process requires a public key (public key), which is obtained from SealeSecret controller.

However, only from the description documents, the key s that SealeSecret controller relies on for encryption and decryption are also saved through an ordinary Secret. Isn't this a problem? At the same time, it also increases the operation and maintenance cost of SealeSecret controller.

mozilla/sops

Strictly speaking, sops is not necessarily related to K8s. It is just an encrypted file editor that supports YAML/JSON/ENV/INI and other file formats. It supports AWS KMS, GCP KMS, Azure Key Vault, age, PGP and other services and applications.

If you are interested, you can see its home page.

Kubernetes External Secrets

Kubernetes External Secrets is an open source software developed by godaddy, a well-known domain name service provider. It can directly transfer the confidential information stored in the external KMS to K8s. Currently supported KSM S include:

  • AWS Secrets Manager
  • AWS System Manager
  • Hashicorp Vault
  • Azure Key Vault
  • GCP Secret Manager
  • Alibaba Cloud KMS Secret Manager

It is implemented by customizing controller and CRD. The specific architecture is as follows:

Specifically, the user needs to create an ExternalSecret resource to map the external KMS data to the K8s Secret.

However, there are probably only two benefits of this approach:

  • Unified key management, or use existing key assets
  • key information doesn't want to be put in VCS, etc

It has little effect on preventing Sercet information disclosure, because its plaintext resources can still be seen on API server/etcd.

In other words, what External Secrets really does is to map the key in the external KMS into the Secret resource in K8s, which is not useful to ensure the security of data in K8s cluster.

Kamus

Kamus also provides a method for encrypting a key (a command-line tool). At the same time, this key can be decrypted only through the controller in K8s. However, the Secret saved in K8s is encrypted. Users cannot directly obtain the description of the Secret like External Secrets.

Kamus consists of three components:

  • Encrypt API
  • Decrypt API
  • Key Management System (KMS)

KMS is a package of external encryption services, and currently supports the following services: - AES - AWS KMS - Azure KeyVault - Google Cloud KMS

Kamus encrypts the secret in the unit of service account, and then Pod will request kamus's decryption service to decrypt the secret through service account.

For K8s, the decryption secret can be realized through init container: define a memory based emptyDir. The business container and init container use the same volume. After the init container decrypts, the data is stored in the volume, and then the business container can use the decrypted secret data.

Vault by HashiCorp

HashiCorp is one of the best companies in the field of cloud computing / DevOps.

Vault itself is a KMS like service for managing confidential data. The native secret of K8s is supported in the following two ways:

  • Agent Sidecar Injector/vault-k8s
  • Vault CSI Provider

Agent Sidecar Injector

Similar to the above Kamus, this method also requires two components:

  • Mutation webhook: Modify the definition of pod and inject init/sidecar
  • agent-sidecar: It is responsible for obtaining and decrypting data and saving the data to the specified volume / path

The Vault agent sidecar injector not only provides init container to initialize the secret, but also updates the secret regularly through sidecar, which is very close to the implementation of the native secret.

The application only needs to read the specified file on the file system, and does not need to be related to how to obtain encrypted information from the outside.

This is an example from the official blog:

Pod information:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
        vault.hashicorp.com/role: "myapp"

In this definition, vault-k8s will inject vault agent into the Pod and initialize it with secrets/helloworld. After Pod runs, you can find a file named helloworld under / vault/secrets.

$ kubectl exec -ti app-XXXXXXXXX -c app -- cat /vault/secrets/helloworld
data: map[password:foobarbazpass username:foobaruser]
metadata: map[created_time:2019-12-16T01:01:58.869828167Z deletion_time: destroyed:false version:1]

Of course, this data is raw data and has not been formatted. If you want to specify the format of output to a file, you can use the template function of vault.

Vault CSI Provider

For this part, please refer to the following community program.

Community programme

Of course, the community has no reason not to be aware of the problem of native secret. Therefore, the community also has Kubernetes Secrets Store CSI Driver, a scheme to integrate secret into K8s through CSI interface.

The Secrets Store CSI driver (Secrets store. CSI. K8s. IO) allows K8s mount multiple secrets in the form of volume from the external KMS mount to the Pod.

To use the Secrets Store CSI Driver, the general process is as follows:

  • Define the SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault   # accepted provider options: azure or vault or gcp
  parameters:       # provider-specific parameters
  • Configure Volume for Pod
kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
  - image: k8s.gcr.io/e2e-test-images/busybox:1.29
    name: busybox
    command:
    - "/bin/sleep"
    - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "my-provider"

After the Pod is started, you can confirm the decrypted data:

$ kubectl exec secrets-store-inline -- ls /mnt/secrets-store/
foo

summary

The above summary is based on the information publicly available on the Internet and has not been personally experienced. Therefore, some places may be misunderstood. If you want to have an in-depth understanding, you need to confirm it yourself.

However, generally speaking, the community scheme may be the simplest, and the deployment is not very troublesome, but it has little to do with the native secret...

The Vault scheme is also mature and worthy of attention.

This article is reproduced from the "human guide", original text: https://tinyurl.com/y9warvpa , copyright belongs to the original author.

Hot Topics

©2025 Programmer Think