What stores Kubernetes in Etcd?

Introduction

Kubernetes uses Etcd to store information what’s happening on cluster – it means that master nodes read/write data from/to Etcd cluster to maintain cluster state. Etcd itself is simple distributed key-value store which uses Raft consensus algorithm.

As I like to dig deeper and I couldn’t find how looks data structure stored in Etcd I decided to make a little reverse engineering on my local Minikube cluster. To go trough that article by yourself you need installed Minikube on your machine.

Kubernetes version used: 1.10

How to get into Etcd in Minikube

Firstly we need to discover way how to see data stored in Etcd in Minikube – it’s not so trivial as Etcd has some security mechanisms which blocks us to just read data from any container on Kubernetes or local machine. That security is natural as Kubernetes store there e.g. secrets, private keys, etc. as plain text if encryption is not activated (default). Also writing data directly into Etcd gives us effectively admin rights on cluster.

Firtly start your minikube:

minikube start

And launch Kubernetes dashboard:

minikube dashboard

In kube-system namespace we can find etcd-minikube POD which in we are interested – in start command section we can see some useful things:

etcd
--advertise-client-urls=https://127.0.0.1:2379
--cert-file=/var/lib/localkube/certs/etcd/server.crt
--trusted-ca-file=/var/lib/localkube/certs/etcd/ca.crt
--peer-cert-file=/var/lib/localkube/certs/etcd/peer.crt
--peer-key-file=/var/lib/localkube/certs/etcd/peer.key
--listen-client-urls=https://127.0.0.1:2379
--client-cert-auth=true
--peer-client-cert-auth=true
--data-dir=/data/minikube
--key-file=/var/lib/localkube/certs/etcd/server.key
--peer-trusted-ca-file=/var/lib/localkube/certs/etcd/ca.crt

1. We see that client interface is bound to 127.0.0.1 so we are able to access Etcd only using localhost (not for example POD IP)
2. We see that Etcd listen on 2379 port what is default Etcd port
3. –client-cert-auth=true says us that certificate authentication is activated what means that we need trusted client certificate to get into Etcd

That’s a little strange – how any Kubernetes service can access Etcd if it’s listening only on localhost? So answer is in POD descriptor:

"hostNetwork": true

What means that Etcd’s docker container bind to host’s network interface. So who is connecting on localhost:2379 on Minikube? Let’s go into Minikube VM by using:

minikube ssh

In Minikube VM we have Sysdig available so we can check who is using that port:

sudo sysdig fd.port=2379

....
194074 10:30:39.093997519 1 etcd (3379) > read fd=20(<4t>127.0.0.1:57152->127.0.0.1:2379) size=1024 
194075 10:30:39.094001352 1 etcd (3379) < read res=-11(EAGAIN) data= 
194219 10:30:39.095750456 0 etcd (3236) > write fd=20(<4t>127.0.0.1:57152->127.0.0.1:2379) size=46 
194229 10:30:39.095821582 0 etcd (3236) < write res=46 data=....)......#..+jk.....I5o..N{z....e.9.;..wf... 
194240 10:30:39.095941041 1 kube-apiserver (3360) > read fd=16(<4t>127.0.0.1:57152->127.0.0.1:2379) size=2048 
194244 10:30:39.095983703 1 kube-apiserver (3360) < read res=46 data=....)......#..+jk.....I5o..N{z....e.9.;..wf... 
194247 10:30:39.096011985 1 kube-apiserver (3360) > read fd=16(<4t>127.0.0.1:57152->127.0.0.1:2379) size=2048 
194249 10:30:39.096014883 1 kube-apiserver (3360) < read res=-11(EAGAIN) data= 
.....

We see that kube-apiserver (as expected) communicate with Etcd over localhost (so kube-apiserver is also bound to host’s network interface). So let’s take a look on kubeapi starting command in Dashboard:

kube-apiserver
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
--enable-bootstrap-token-auth=true
--requestheader-extra-headers-prefix=X-Remote-Extra-
...
...
--etcd-servers=https://127.0.0.1:2379
--etcd-cafile=/var/lib/localkube/certs/etcd/ca.crt
--etcd-certfile=/var/lib/localkube/certs/apiserver-etcd-client.crt
--etcd-keyfile=/var/lib/localkube/certs/apiserver-etcd-client.key

What is important here:

1. We have confirmation that etcd and kube-apiserver communicate over localhost
2. CA file is required to accept Etcd server certificate
3. Kube-apiserver contains own certificates to authenticate in Etcd

To see data stored in Etcd we can borrow certificates from kube-apiserver and inject them into Etcd container as there is installed etcdctl – CLI which allows to access data in Etcd.

We need to find etcd and apiserver pods. Kubectl is automatically configured to target minikube:

kubectl get pods --namespace kube-system                                                                                              
NAME                                    READY     STATUS    RESTARTS   AGE
etcd-minikube                           1/1       Running   0          3m
kube-addon-manager-minikube             1/1       Running   2          4d
kube-apiserver-minikube                 1/1       Running   0          3m
kube-controller-manager-minikube        1/1       Running   0          3m
kube-dns-86f4d74b45-hpzx9               3/3       Running   9          4d
kube-proxy-tzlzs                        1/1       Running   0          2m
kube-scheduler-minikube                 1/1       Running   2          4d
kubernetes-dashboard-5498ccf677-fs5vk   1/1       Running   6          4d
storage-provisioner                     1/1       Running   6          4d

So we see etcd and apiserver pods – we need to copy certificates from apiserver to etcd:

kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/localkube/certs/apiserver-etcd-client.crt apiserver-etcd-client.crt
kubectl cp --namespace kube-system apiserver-etcd-client.crt etcd-minikube:var/lib/localkube/certs/

kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/localkube/certs/apiserver-etcd-client.key apiserver-etcd-client.key
kubectl cp --namespace kube-system apiserver-etcd-client.key etcd-minikube:var/lib/localkube/certs/

And go into Etcd container:

kubectl exec -it --namespace kube-system etcd-minikube sh

We need to set etcdctl tool to v3 API version using following environment variable:

export ETCDCTL_API=3

And test if we have access to Etcd:

/var/lib/localkube/certs # etcdctl --cacert="etcd/ca.crt" --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt endpoint status 
127.0.0.1:2379, 8e9e05c52164694d, 3.1.12, 2.2 MB, true, 3, 18349

Data structure description

To get all keys from Etcd we need to type:

/var/lib/localkube/certs # etcdctl --cacert="etcd/ca.crt" --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt get / --prefix --keys-only

Output is really big and divided into some namespaces – let’s look on them.

 

/registry/apiregistration.k8s.io/apiservices/{version}.{api name}

Contains definition of API Services in Kubernetes so we can find there all existing core APIs used by Kubernetes like /registry/apiregistration.k8s.io/apiservices/v1.batch or /registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io or custom APIs definition (see https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/aggregation.md). You can get information about APIs by reading value of that key in Etcd (you will get human-readable json) or in more friendly way by using kubectl get apiservice v1beta1.authorization.k8s.io -o json (the same value as in direct Etcd access)

/registry/clusterroles/{role name}

Contains definition of all cluster-wide roles in Kubernetes so we can find there things like /registry/clusterroles/cluster-admin or /registry/clusterroles/system:kube-scheduler. Data in Etcd is human readable but hard to understand – we can see there some actions like get, patch, update on some parts of API

/registry/clusterrolebindings/{entity name}

Contains binding between roles and users/groups/service accounts which are cluster-wide so we can find there things like /registry/clusterrolebindings/cluster-admin or /registry/clusterrolebindings/kubeadm:kubelet-bootstrap. Data in Etcd is human readable but hard to understand.

/registry/roles/{namespace}/{role name} and /registry/rolebindings/{namespace}/{entity name}

Same story as in cluster roles/bindings but scoped by namespace e.g. /registry/roles/kube-system/system:controller:token-cleaner

/registry/serviceaccounts/{namespace}/{name}

Definition of all service accounts

/registry/configmaps/{namespace}/{map name}

All configs maps stored as yamls

/registry/controllerrevisions/{namespace}/{pod}

I found ControllerRevision resource is used to provide rollback possilibities in DaemonSet and StatefulSet (https://kubernetes.io/docs/tasks/manage-daemon/rollback-daemon-set/). In Etcd we can find snapshot of pods spec.

/registry/daemonsets/{namespace}/{name} and /registry/deployments/{namespace}/{name} etc.

Under those keys Kubernetes stores information about different deployments like DaemonSet, Deployment, ReplicaSet, Job, etc. What’s interesting in case of deployment we see there last-applied-configuration described there https://kubernetes.io/docs/concepts/overview/object-management-kubectl/declarative-config/#merge-patch-calculation

/registry/minions/{node name}

Kubernetes nodes were previously called “minions” so in Etcd name is still not changed. We see there big amount of data describing node like:

• CPU cores
• Memory size
• Status of kubelet: e.g. kubelet has sufficient disk space available or kubelet has sufficient PID available
• Ip address
• Hostname
• Docker version
• Docker image/registry/ranges/servicenodeportss available on node

/registry/namespaces/{namespace}

Just defining namespace. There is also state of particular namespace like Active or Terminating.

/registry/pods/{namespace}/{pod name}

State of every pod running in cluster. Contains a lot of information like pod IP, mounted volumes, docker image etc.

/registry/ranges/serviceips

CIDR for services

/registry/ranges/servicenodeports

Ports range for exposing services

/registry/secrets/{namespace}/{pod}

All secrets in cluster stored as plain text in default mode. For encryption see https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

/registry/services/endpoints/{namespace}/{name}

Services definition. Kubernetes calculates which pods are selected by particular service and stores that information in service value so we can see pods ip addresses and names there.