More on Setting up etcd with Kubeadm

A while ago I wrote about using kubeadm to bootstrap an etcd cluster with TLS. In that post, I talked about one way to establish a secure etcd cluster using kubeadm and running etcd as systemd units. In this post, I want to focus on a slightly different approach: running etcd as static pods. The information on this post is intended to build upon the information already available in the Kubernetes official documentation, not serve as a replacement.

For reference, the Kubernetes official documentation has a write-up on using kubeadm to establish an etcd cluster with etcd running as static pods. For Kubernetes 1.12.x (the current version as of this writing), that information is here; for Kubernetes 1.11.x, that same information is here.

When using these instructions for use with Kubernetes 1.11.x, the official guide leaves something out that is very important: reconfiguring the kubelet to operate in a standalone fashion (without the Kubernetes control plane). This information is present in the 1.12.x documentation, but it applies to both versions.

Now, lest you think you can just follow the 1.12.x documentation for a 1.11.x cluster, you need to know that the kubeadm API changed from 1.11.x (where it is v1alpha2) to 1.12.x (where it is v1alpha3), and this means a number of changes elsewhere as well. This means that the kubeadm config files in the 1.12.x guide won’t work for 1.11.x. Additionally, I found that the 1.12.x guide leaves out one additional piece of information that might be necessary in order to get a working kubelet.

The extra step that is needed for 1.11.x (but that is included in the 1.12.x version of the documentation) is to provide an additional systemd drop-in file that configures the kubelet as a standalone kubelet. What the 1.12.x instructions are potentially missing—depending on your Linux distribution and/or the configuration of the Docker container runtime—is matching the kubelet’s cgroup driver to the runtime’s cgroup driver. If these two are mismatched, the kubelet will fail to start.

In my testing on Ubuntu 16.04 images prepared using Wardroom, the Docker container runtime was using the systemd cgroup driver. The kubelet, however, was not, and this caused the kubelet to fail even after adding the drop-in from the 1.12.x documentation.

If this affects you, you can amend the 1.12.x systemd drop-in to add a --cgroup-driver=systemd flag, so that the complete drop-in now looks like this:

[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cgroup-driver=systemd
Restart=always

Put this file in /etc/systemd/system/kubelet.service.d with a higher priority than any other drop-ins found there. I found that, per the 1.12.x instructions, using the filename 20-etcd-service-manager.conf was sufficient. Once this file is in place, then doing a systemctl daemon-reload and systemctl restart kubelet should be enough to get a running kubelet that can run etcd as a static pod.

The official docs—for both 1.11.x and 1.12.x—also assume that you’ll run all the kubeadm alpha phase commands from one specific host, then copy files around to the other hosts as appropriate. This will work, but if you do it this way the certificates generated by kubeadm will have an extra hostname attached to them: the hostname from the system generating the certificates. To avoid having this extra hostname attached to the certificates generated by kubeadm, you’ll need to instead copy the kubeadm configuration files to each node, then run the kubeadm alpha phase commands on each node. This also means that you’ll need to copy both the ca.crt and the ca.key generated on the first node to the second and third nodes. The instructions only say to copy the ca.crt file, but this is because it isn’t needed on the other nodes when you’re generating all the certificates on one node.

Aside from these two (very) minor issues, the procedures in the official docs will get you a working etcd cluster, properly secured with TLS certificates, and running as static pods under the kubelet (instead of as systemd units). Between the official docs plus these notes (which I’ll see about getting merged into the official docs) or my earlier article about running etcd as systemd units, you now have a couple of different approaches to setting up a properly-secured etcd cluster that can back your Kubernetes infrastructure.

Feel free to hit me up on Twitter if you have any questions, or if you feel I’ve made a mistake or misrepresented anything. Thanks!

UPDATE 2020-01-31: As of the Kubernetes 1.13 release, kubeadm alpha phase was replaced by kubeadm init phase, so be sure to adjust the commands above accordingly.

Metadata and Navigation