Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Reconstructing the Join Command for Kubeadm

If you’ve used kubeadm to bootstrap a Kubernetes cluster, you probably know that at the end of the kubeadm init command to bootstrap the first node in the cluster, kubeadm prints out a bunch of information: how to copy over the admin Kubeconfig file, and how to join both control plane nodes and worker nodes to the cluster you just created. But what if you didn’t write these values down after the first kubeadm init command? How does one go about reconstructing the proper kubeadm join command?

Fortunately, the values needed for a kubeadm join command are relatively easy to find or recreate. First, let’s look at the values that are needed.

Here’s the skeleton of a kubeadm join command for a control plane node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
--control-plane \
--certificate-key <certificate-key>

And here’s the skeleton of a kubeadm join command for a worker node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \

As you can see, the information needed for the worker node is a subset of the information needed for a control plane node.

Here’s how to find or recreate all the various pieces of information you need:

  • The value for <endpoint-ip-or-dns>:<port> can be retrieved from the “kubeadm-config” ConfigMap in the cluster’s “kube-system” namespace. Just run kubectl -n kube-system get cm kubeadm-config -o yaml and then look for the value specified for controlPlaneEndpoint.
  • For <valid-bootstrap-token>, there’s no way to find out the original token. It is, however, easy to create a new token. Just run kubeadm token create and note the output. Be aware that tokens have a default lifetime of 24 hours.
  • For <ca-cert-sha256-hash>, see this blog post.
  • For <certificate-key>, you can’t get whatever original value was output by kubeadm init. You can, though, generate another value. (This value is only good for two hours, so it’s far more likely you’d need to generate another value anyway.) Run kubeadm init phase upload-certs --upload-certs and make a note of the output of the command.

There you have it—how to easily reconstruct all the information needed to use kubeadm join to join a node (either control plane node or worker node) to a Kubernetes cluster. Note that this information also applies to gathering the information needed for a JoinConfiguration stanza in a kubeadm configuration file, like one used in this post to join control plane nodes or worker nodes to a cluster.

If I’ve missed anything here, feel free to contact me on Twitter. Thanks!

Setting up an AWS-Integrated Kubernetes 1.15 Cluster with Kubeadm

In this post, I’d like to walk through setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm. Over the last year or so, the power and utility of kubeadm has vastly improved (thank you to all the contributors who have spent countless hours!), and it is now—in my opinion, at least—at a point where setting up a well-configured, highly available Kubernetes cluster is pretty straightforward.

This post builds on the official documentation for setting up a highly available Kubernetes 1.15 cluster. This post also builds upon previous posts I’ve written about setting up Kubernetes clusters with the AWS cloud provider:

All of these posts are focused on Kubernetes releases prior to 1.15, and given the changes in kubeadm in the 1.14 and 1.15 releases, I felt it would be helpful to revisit the process again for 1.15. For now, I’m focusing on the in-tree AWS cloud provider; however, in the very near future I’ll look at using the new external AWS cloud provider.

As pointed out in the “original” September 2018 article, there are a number of high-level requirements needed in order to properly bootstrap an AWS-integrated Kubernetes cluster:

  1. The hostname of each node must match the EC2 Private DNS entry for the instance. By default this is something like ip-10-11-12-13.us-west-2.compute.internal, but the use of custom DNS domains is possible (a teammate has verified this and hopefully will be blogging about it soon).
  2. Each node must have an IAM instance profile that grants it access to an IAM role and policy with permissions to the AWS API.
  3. Resources used by the cluster must have specific AWS tags assigned to them.
  4. Specific entries are needed in the kubeadm configuration file used to bootstrap the cluster, join control plane nodes, and join worker nodes.

Let’s dig into each of these areas in a bit more detail. (Most, if not all, of this information is also included in the September 2018 post. Feel free to check that post for more information or details.)

Setting Node Hostnames

The hostname for the OS must match the EC2 Private DNS entry for that particular instance. By default, this is typically something like ip-10-11-12-13.us-west-2.compute.internal (change the numbers and the region to appropriately reflect the private IP address and region of the instance). The fastest/easiest way I’ve verified to make sure this is the case is with this command:

sudo hostnamectl set-hostname \
$(curl -s http://169.254.169.254/latest/meta-data/local-hostname)

Be sure to set the hostname before starting the bootstrapping process. Anecdotally, I’ve heard of some success putting this command in the user data for the instance, so that it runs automatically.

Creating and Assigning the IAM Instance Profile

The nodes in the cluster need to have specific permissions to AWS API objects in order for the AWS cloud provider to work. The cloud-provider-aws GitHub repository has full details on the specific permissions needed for control plane nodes and worker nodes. You’ll want to create the appropriate IAM roles, IAM policies, and IAM instance profiles so that the nodes have the necessary access. When creating the instances (either via the console, CLI, or some infrastructure-as-code tool), be sure to specify the IAM instance profile they should have assigned.

Assigning Tags to Resources

The AWS cloud provider needs resources to be tagged with a tag named kubernetes.io/cluster/<cluster-name> (the value is immaterial). Based on all the documentation I’ve been able to find, this tag is needed on all nodes, on exactly one security group (the nodes should be a member of this security group), and on all subnets and route tables. Failing to have things properly tagged with result in odd failure modes, like ELBs being automatically created in response to the creation of a Service object of type LoadBalancer, but instances never being populated for the ELB (for example).

Using Kubeadm Configuration Files

To use kubeadm to bootstrap a Kubernetes 1.15 cluster with the AWS cloud provider, you’ll want to use a kubeadm configuration file (Kubernetes 1.15 uses the “v1beta2” version of the kubeadm API; documentation for the API is here).

Three different configuration files are needed:

  1. A configuration file to be used to bootstrap the first/initial control plane node
  2. A configuration file used to join the additional control plane nodes
  3. A Configuration file used to join worker nodes

The sections below provide details on each of these configuration files, along with examples you can use to create your own.

Bootstrapping the First Control Plane Node

Here’s an example of a kubeadm configuration file you could use to bootstrap the first control plane node (some explanation is found below the example):

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: aws
clusterName: blogsample
controlPlaneEndpoint: cp-lb.us-west-2.elb.amazonaws.com
controllerManager:
  extraArgs:
    cloud-provider: aws
    configure-cloud-routes: "false"
kubernetesVersion: stable
networking:
  dnsDomain: cluster.local
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: aws

Let’s break this down just a bit:

  • The apiServer.extraArgs.cloud-provider and controllerManager.extraArgs.cloud-provider settings are what generate the --cloud-provider=aws flags for the API server and the controller manager (you can verify this by looking at their manifests in /etc/kubernetes/manifests after bootstrapping the control plane node). You need these settings here because the AWS cloud provider needs to be specified when you first bootstrap the node, not added afterward.
  • The nodeRegistration.kubeletExtraArgs.cloud-provider setting serves the same purpose, but for the Kubelet. Again, this has to be here when you first bootstrap the node; adding it afterward won’t work.
  • Because this is an HA cluster, controlPlaneEndpoint needs to point to the load balancer for your control plane nodes.
  • The clustername value isn’t technically necessary, but this allows you to specify individual clusters within a given set of AWS resources (make sure you use this name in the AWS tags on your resources).
  • Technically, the networking.dnsDomain and networking.serviceSubnet aren’t necessary either, but I’m including them here for completeness. networking.podSubnet is often required to be included for some CNI plugins (Calico, for example, requires this value to be set).
  • You’ll note there’s no etcd section—this tells kubeadm to create an etcd node local to the control plane node, in what we call a “stacked control plane” configuration. If you’re using an external etcd cluster, you’ll need to add the appropriate configuration to this file.

Set the values you need in this file, then bootstrap your first control plane node with kubeadm init --config=kubeadm.yaml (substituting the correct filename, of course).

Once you’ve verified that the first control plane node is up, go ahead and install your CNI plugin.

Adding More Control Plane Nodes

The 1.14 and 1.15 releases of Kubernetes added some significant functionality to kubeadm around the addition of control plane nodes. Thus, the addition of control plane nodes is dramatically simpler now.

Here’s a kubeadm configuration file you can use to join the second and third control plane nodes to the first node:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: 123456.a4v4ii39rupz51j3
    apiServerEndpoint: "cp-lb.us-west-2.elb.amazonaws.com:6443"
    caCertHashes: ["sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"]
nodeRegistration:
  name: ip-10-10-180-167.us-west-2.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws
controlPlane:
  localAPIEndpoint:
    advertiseAddress: 10.10.180.167
  certificateKey: "f6fcb672782d6f0581a1060cf136820acde6736ef12562ddbdc4515d1315b518"

Some additional information on some of these values may be helpful:

  • You’ll be using kubeadm join, not kubeadm init, so this needs to be a JoinConfiguration.
  • If you don’t know the value of a valid bootstrap token, you can create a new one with kubeadm token create and specify it for discovery.bootstrapToken.token.
  • The apiServerEndpoint should be the load balancer for the control plane.
  • If you don’t know the SHA256 hash of the CA certificate, see this post.
  • The value for certificateKey is only good for two hours, so you may need to use kubeadm init phase upload-certs --upload-certs on the initial control plane node if it has been more than two hours since the value was generated. This will generate a new certificate key value.
  • As outlined in this post, the controlPlane.localAPIEndpoint value is what tells kubeadm this is a control plane node you’re joining to the cluster.

Once you’ve set the values you need in this file, then just run kubeadm join --config=kubeadm.yaml (substituting the correct filename, of course). The node should join the cluster as a control plane node.

Adding Worker Nodes

Because adding worker nodes uses kubeadm join, you’ll find that the necessary kubeadm configuration here is similar to, but simpler, than the configuration for joining control plane nodes.

Here’s a sample configuration file that would work (additional notes about this file are found below the example):

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: 123456.a4v4ii39rupz51j3
    apiServerEndpoint: "cp-lb.us-west-2.elb.amazonaws.com:6443"
    caCertHashes: ["sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"]
nodeRegistration:
  name: ip-11-12-13-14.us-west-2.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws

As noted earlier, you’ll need a valid bootstrap token and the SHA256 hash of the CA certificate.

Once you have the correct values in this file, then run kubeadm join --config=kubeadm.yaml to join new worker nodes to the cluster (substitute the correct filename, of course).

Wrapping Up

At this point, you should have a fully functional Kubernetes cluster running on AWS, with a functioning AWS cloud provider.

If you run into issues getting this to work correctly for you, I encourage you to pop over to the Kubernetes Slack instance (in the “#kubeadm” channel) and ask any questions. I tend to be there, as are a lot of the kubeadm contributors. You’re also welcome to contact me on Twitter if you have any questions, comments, or corrections. (If you are contacting me for technical assistance, I do ask that you at least try some basic troubleshooting first. Thanks!)

Enjoy!

Converting Kubernetes to an HA Control Plane

While hanging out in the Kubernetes Slack community, one question I’ve seen asked multiple times involves switching a Kubernetes cluster from a non-HA control plane (single control plane node) to an HA control plane (multiple control plane nodes). As far as I am aware, this isn’t documented upstream, so I thought I’d walk readers through what this process looks like.

I’m making the following assumptions:

  • The existing single control plane node was bootstrapped using kubeadm. (This means we’ll use kubeadm to add the additional control plane nodes.)
  • The existing single control plane node is using a “stacked configuration,” in which both etcd and the Kubernetes control plane components are running on the same nodes.

I’d also like to point out that there are a lot of different configurations and variables that come into play with a process like this. It’s (nearly) impossible to cover them all in a single blog post, so this post attempts to address what I believe to be the most common situations.

With those assumptions and that caveat in mind, the high-level overview of the process looks like this:

  1. Create a load balancer for the control plane.
  2. Update the API server’s certificate.
  3. Update the kubelet on the existing control plane.
  4. Update other control plane components.
  5. Update worker nodes.
  6. Add control plane nodes.

The following sections will explore each of these steps in a bit more detail. First, though, a disclaimer.

Disclaimer

In all reality, the best way to “upgrade” a non-HA control plane to an HA control plane is to build a new HA control plane and migrate all your workloads (perhaps using something like Velero). This better aligns with “cluster as cattle” thinking (more appropriately known as immutable infrastructure). With that in mind, I’m presenting the following information more as a means of learning a bit more about how Kubernetes works. I do not recommend using this procedure on any cluster that will ever be considered a production cluster.

Create a Load Balancer for the Control Plane

The first step is to create a load balancer for the control plane (a load balancer is required when using multiple control plane nodes). Since the specifics of how to set up and configure a load balancer will vary from solution to solution, I won’t try to include the details here other than to mention two high-level requirements:

  • You should be using a Layer 4 load balancer (TCP instead of HTTP/HTTPS).
  • Health checks should be configured as SSL health checks instead of TCP health checks (this will weed out spurious “TLS handshake errors” in the API server’s logs).

It is also be a good idea at this time to create a DNS CNAME entry to point to your load balancer (highly recommended). This gives you some additional flexibility in the event you need to swap out or reconfigure your load balancing solution, as the DNS CNAME remains constant even if the names to which it resolves change behind the scenes. (The name remaining constant is important, as you’ll see in a moment.)

Whether or not you create a DNS CNAME, make a note of the IP address(es) and DNS names you’ll use to connect to the cluster through the load balancer, as you’ll need those in the next step.

Because everything in the cluster still points directly to the single control plane node, adding in the load balancer won’t affect anything unless the load balancer is inline (requiring traffic to go through the load balancer to reach the single control plane node). At this point, I’d recommend sticking with a solution that still allows the rest of the cluster to reach the single control plane node directly.

Update the API Server’s Certificate

The next step is to update the API server’s TLS certificate to account for the IP address(es) and/or DNS names that will be used to reach the control plane through the load balancer (see, I said you’d need them in the next step!). The API server (one of the three Kubernetes control plane components, the other two being the controller manager and the scheduler) uses a TLS certificate to both provide authentication as well as to encrypt control plane traffic. This certificate needs to have a proper Subject Alternative Name (SAN) that matches whatever IP address or DNS name is being used to communicate with the API server. Otherwise, communications with the API server will result in an error, and that (ultimately) break your cluster.

To address this, you’ll need to add one or more new SANs to the API server’s certificate. I wrote about this process in this blog post, so follow the instructions in that post to update your API server certificate.

Once you’ve completed this process, I recommend you update your Kubeconfig file (as outlined in the “Verifying the Change” section of the blog post for updating your certificate) to point to an IP address or DNS name that will direct the traffic through the load balancer, and then test kubectl access to your cluster. Even though the rest of the cluster is still pointing directly to the single control plane node, using kubectl through the load balancer should still work as expected. If it doesn’t work, stop and troubleshoot the issue before proceeding. You’ll want to be sure that access to the API server through the load balancer is working before you continue.

Update the Kubelet on the Control Plane Node

The Kubelet on the existing control plane node communicates with the API server, as do all the other components of the cluster. Once you’ve verified that access to the API server through the load balancer works, the next step is to update the Kubelet to access the API server through the load balancer as well.

Much like a user does, the Kubelet uses a Kubeconfig file to know how to find the control plane and authenticate to it. This file is found in /etc/kubernetes/kubelet.conf, which you can verify by looking at the systemd drop-in added to the Kubelet service by kubeadm (the file is /etc/systemd/system/kubelet.service.d/10-kubeadm.conf).

In order to have the Kubelet communicate with the API server through the load balancer, you’ll need to edit this Kubeconfig file for the Kubelet (again, the file in question is /etc/kubernetes/kubelet.conf) and change the server: line to point to an IP address or DNS name for the load balancer (and for which there is a corresponding SAN on the API server’s certificate). Note you can also use export KUBECONFIG=/etc/kubernetes/kubelet.conf and kubectl config set clusters.default-cluster.server to make this change. (Personally, I find editing the file easier.)

Once you made this change, restart the Kubelet with systemctl restart kubelet, and then check the logs for the Kubelet to be sure it is working as expected.

Update Control Plane Components

The fourth step is to update the other control plane components to communicate with the API server through the load balancer. Like the Kubelet, both the controller manager and the scheduler (two other components of the Kubernetes control plane along with the API server) use Kubeconfig files to communicate with and authenticate to the API server. Just as you updated the Kubeconfig file (by modifying the server: line for the cluster being modified to point to the load balancer) used by the Kubelet, you’ll also need to update the Kubeconfig files that the controller manager and scheduler use to connect to the API server.

The files that need to be modified are:

/etc/kubernetes/controller-manager.conf
/etc/kubernetes/scheduler.conf

These files are standard Kubeconfig files. The only line that needs to be changed is the server: line that specifies the API endpoint (this is currently probably pointing to the IP address or hostname of the single control plane node). Edit each of these files to point to an IP address or DNS name for the load balancer (and for which a SAN exists on the API server certificate).

For these components to pick up the change, you’ll need to restart them. Assuming that Docker is the container runtime in use, these commands will kill the container for each component:

docker kill $(docker ps | grep kube-controller-manager | \
grep -v pause | cut -d' ' -f1)
docker kill $(docker ps | grep kube-scheduler | grep -v pause | \
cut -d' ' -f1)

The Kubelet will then restart them automatically, and they’ll pull in the changes to their respective Kubeconfig files. Once the controller manager and scheduler have restarted, check the logs for each (using either docker logs <containerID> or kubectl logs -n kube-system <podName>) to make sure that they are operating correctly.

At this point, you’ve created and configured a load balancer for the control plane, updated the API server’s certificate to account for the load balancer, updated the Kubelet to point to the load balancer, and updated the controller manager and scheduler to point to the load balancer.

Update Worker Nodes

The only component running on the worker nodes that needs to be updated is the Kubelet configuration. Follow the instructions in the section “Update the Kubelet on the Control Plane Node” to update the Kubelet on the worker nodes as well.

Update the In-Cluster Configuration

As pointed out in this post, kubeadm stores some configuration in a ConfigMap named “kubeadm-config” in the “kube-system” namespace. kubeadm uses this information when performing cluster upgrades, or when joining nodes to the cluster. Since we’ve now added a load balancer in front of the control plane, we need to update this ConfigMap with the correct information. (You’ll be adding control plane nodes to the cluster shortly, so having the correct information in this ConfigMap is important.)

First, pull the current configuration from the ConfigMap with this command:

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

Currently, the controlPlaneEndpoint is most likely empty (just a pair of double quotes). Edit this value to point to the DNS CNAME of the load balancer you created and configured for the control plane. (As I mentioned earlier, the use of a DNS CNAME for the control plane load balancer is recommended.)

You should also be able to see the certSANs section that was added as part of adding a SAN to the API server’s certificate (assuming you followed the directions to update the ConfigMap when updating the API server certificate).

Once you’ve edited the file, upload it back to the cluster with this command:

kubeadm config upload from-file --config kubeadm.yaml

You also need to update the “cluster-info” ConfigMap in the “kube-public” namespace, which contains a Kubeconfig file with a server: line that points to the single control plane node. It’s easist to just use kubectl -n kube-public edit cm cluster-info and update the server: line to point to the load balancer for the control plane. (Thanks to Fabrizio Pandini for pointing this out!)

Only one step remains now: actually adding more control plane nodes to make the control plane highly available.

Add Control Plane Nodes

To add additional control plane nodes—and you should be adding two additional nodes, for a total of three, which allows etcd to reach quorum—you can follow the instructions from the Kubernetes web site (see here for version 1.15 or here for version 1.14). In particular, see the “Steps for the rest of the control plane nodes” under the “Stacked control plane and etcd nodes” section.

There are a few caveats/considerations to keep in mind:

  • These instructions assume you are joining the additional control plane nodes immediately or nearly immediately after creating the control plane. In this case, however, the control plane may have been up for quite a while. Therefore, you’re going to need to upload the certificates again (and generate a new certificate key) using kubeadm init phase upload-certs --upload-certs (for 1.15) or kubeadm init phase upload-certs --experimental-upload-certs (for 1.14). This will generate a new certificate key, which you’ll need (it’s only good for 2 hours).
  • For the same reason as above, you’ll probably also need to generate a new bootstrap token (the default lifetime of a token is 24 hours). You can do this with kubeadm token create.
  • Finally, you may not know the SHA256 hash of the CA certificate. Fortunately, I have this covered for you as well; see this post for instructions.

Once you have a valid certificate key, a valid bootstrap token, and the correct SHA256 hash of the CA certificate, you can join a new control plane node with this command:

kubeadm join <DNS CNAME of load balancer>:6443 \
--token <bootstrap-token> \
--discovery-token-ca-cert-hash sha256:<CA certificate hash> \
--control-plane --certificate-key <certificate-key>

(If you’re using 1.14, replace --control-plane with --experimental-control-plane.)

Once you’ve added two more control plane nodes (for a total of three), you now have a highly available Kubernetes control plane. Go pat yourself on the back.

Reminder: I do not recommend using this procedure for any sort of cluster that is or ever will be considered “production.” See the “Disclaimer” section above.

If you have any questions, comments about the post, or corrections/suggestions, please don’t hesitate to contact me on Twitter. Or, you can find me on the Kubernetes Slack community (I hang out a lot in the “#kubeadm” channel). I hope this information is useful to you!

Technology Short Take 117

Welcome to Technology Short Take #117! Here’s my latest gathering of links and articles from the around the World Wide Web (an “old school” reference for you right there). I’ve got a little bit of something for most everyone, except for the storage nerds (I’m leaving that to my friend J Metz this time around). Here’s hoping you find something useful!

Networking

Servers/Hardware

Security

Cloud Computing/Cloud Management

  • Dan Finneran has a great introductory post on first steps with govmomi.
  • Iman Tumorang shares how to use a private instance of Google Container Registry (GCR) from a Kubernetes cluster.
  • Mohamed Labouardy has a write-up showing how to pull together various Google services, including GKE, Cloud Build, and GCR along with Terraform and Packer, to build a CI/CD workflow.
  • This is probably the strongest use case I’ve seen so far for using a general-purpose programming language for infrastructure-as-code. The linked article uses TypeScript with Pulumi, but the concept—being able to extend existing concepts via a general purpose programming language to add new functionality—applies equally well to other environments as well.

Operating Systems/Applications

Storage

Nothing from me this time around, but if you’re really hurting for some storage news and views, check out Storage Short Take #12 from J Metz.

Virtualization

Career/Soft Skills

  • James Beswick explores the cloud skill shortage and why it exists even though there are plenty of folks who are certified. The “TL;DR” is, in my mind, captured in this statement: “They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly.” I’d say this is a strong indicator that experienced IT folks who are able to assimilate new skills are going to have an advantage moving forward.

OK, enough is enough, time to wrap it up. As always, feel free to engage with me on Twitter if you have any questions, corrections, suggestions, or other feedback. Thanks!

Accessing the Docker Daemon via an SSH Bastion Host

Today I came across this article, which informed me that (as of the 18.09 release) you can use SSH to connect to a Docker daemon remotely. That’s handy! The article uses docker-machine (a useful but underrated tool, I think) to demonstrate, but the first question in my mind was this: can I do this through an SSH bastion host? Read on for the answer.

If you’re not familiar with the concept of an SSH bastion host, it is a (typically hardened) host through which you, as a user, would proxy your SSH connections to other hosts. For example, you may have a bunch of EC2 instances in an AWS VPC that do not have public IP addresses. (That’s reasonable.) You could use an SSH bastion host—which would require a public IP address—to enable SSH access to otherwise inaccessible hosts. I wrote a post about using SSH bastion hosts back in 2015; give that post a read for more details.

The syntax for connecting to a Docker daemon via SSH looks something like this:

docker -H ssh://user@host <command>

So, if you wanted to run docker container ls to list the containers running on a remote system, you’d run this command:

docker -H ssh://slowe@10.11.12.13 container ls

Note that there’s no way to pass sudo via this syntax, so you’ll need to ensure that the user you specify on the command line has the ability to run docker commands without privilege escalation. (You could add them to the “docker” group on the remote system, for example.)

Now, what about doing this through an SSH bastion host? Let’s say you have this configuration stanza in your ~/.ssh/config file (the IP address shown below formerly belonged to an EC2 instance in one of my AWS accounts, but has since been terminated—do not try to connect to this system!):

Host bastion-host
  HostName 18.191.181.211
  User ubuntu
  IdentityFile ~/.ssh/sshkey_rsa

Host 10.100.15.*
  User ubuntu
  IdentityFile ~/.ssh/sshkey_rsa
  ProxyCommand ssh bastion-host -W %h:%p

Assuming there was an instance at 10.100.15.25, this configuration would allow you to simply run ssh 10.100.15.25 and it would attempt to connect as user “ubuntu”, using the specified private key, through the SSH bastion host defined as “bastion-host”. (This is all described in detail in a number of posts all over the Internet, as well as here, here, here, and here on this site).

So, if you have a working SSH bastion configuration, the cool thing about accessing a remote Docker daemon over SSH is that the bastion host configuration is totally transparent and just works.

Using the working configuration described above, this command will work straight out of the box:

docker -H ssh://10.100.15.25 container ls

SSH will connect to the remote Docker daemon via the configured bastion host, with no additional effort or configuration required on your part. (All you need is a working SSH bastion host configuration already in place.)

Nifty, eh?

Recent Posts

Decoding a Kubernetes Service Account Token

Recently, while troubleshooting a separate issue, I had a need to get more information about the token used by Kubernetes Service Accounts. In this post, I’ll share a quick command-line that can fully decode a Service Account token.

Read more...

Adding a Name to the Kubernetes API Server Certificate

In this post, I’m going to walk you through how to add a name (specifically, a Subject Alternative Name) to the TLS certificate used by the Kubernetes API server. This process of updating the certificate to include a name that wasn’t included could find use for a few different scenarios. A couple of situations come to mind, such as adding a load balancer in front of the control plane, or using a new or different URL/hostname used to access the API server (both situations taking place after the cluster was bootstrapped).

Read more...

VMworld 2019 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. I didn’t attend the conference last year, but organized a prayer time nevertheless (and was able to join one morning for prayer). This year, now that I’m back at VMware (via the Heptio acquisition) and speaking at the conference, I’d once again like to coordinate a time for believers to meet. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.

Read more...

Spousevitivities at VMworld 2019

This year VMworld—VMware’s annual user conference—moves back to San Francisco from Las Vegas. Returning to the Bay Area with VMworld is Spousetivities, which is happening again this year for the 11th year at VMworld. Better get your tickets sooner rather than later, there’s quite a good chance these activities will sell out!

Read more...

Calculating the CA Certificate Hash for Kubeadm

When using kubeadm to set up a new Kubernetes cluster, the output of the kubeadm init command that sets up the control plane for the first time contains some important information on joining additional nodes to the cluster. One piece of information in there that (until now) I hadn’t figured out how to replicate was the CA certificate hash. (Primarily I hadn’t figured it out because I hadn’t tried.) In this post, I’ll share how to calculate the CA certificate hash for kubeadm to use when joining additional nodes to an existing cluster.

Read more...

Building Jsonnet from Source

I recently decided to start working with jsonnet, a data templating language and associated command-line interface (CLI) tool for manipulating and/or generating various data formats (like JSON, YAML, or other formats; see the Jsonnet web site for more information). However, I found that there are no prebuilt binaries for jsonnet (at least, not that I could find), and so I thought I’d share here the process for building jsonnet from source. It’s not hard or complicated, but hopefully sharing this information will streamline the process for others.

Read more...

Technology Short Take 116

Welcome to Technology Short Take #116! This one is a bit shorter than usual, due to holidays in the US and my life being busy. Nevertheless, I hope that I managed to capture something you find useful or helpful. As always, your feedback is welcome, so if you have suggestions, corrections, or comments, you’re welcome to contact me via Twitter.

Read more...

Technology Short Take 115

Welcome to Technology Short Take #115! I’m back from my much-needed vacation in Bali, and getting settled back into work and my daily routine (which, for the last few weeks, was mostly swimming in the pool and sitting on the beach). Here’s a fresh new collection of links and articles from the around the web to propel myself back into blogging. I hope you find something useful here!

Read more...

Blogging Break

I wanted to let readers know that there will be a break in my blogging over the next few weeks. Crystal and I are celebrating our 20th wedding anniversary and have decided to take a very long trip to someplace very far away from civilization so that we can relax, unplug, and simply enjoy each other’s company.

Read more...

Technology Short Take 114

Welcome to Technology Short Take #114! There will be a longer gap than usual before the next Tech Short Take (more details to come on Monday), but in the meantime here’s some articles and links to feed your technical appetite. Enjoy!

Read more...

The Linux Migration: Preparing for the Migration

As far back as 2012, I was already thinking about migrating away from Mac OS X (now known as macOS). While the migration didn’t start in earnest until late 2016, a fair amount of work happened in advance of the migration. Since I’ve had a number of folks ask me about migrating to Linux, I thought I’d supplement my Linux migration series with a “prequel” about some of the work that happened to prepare for the migration.

Read more...

A Sandbox for Learning Pulumi

I recently started using Pulumi, a way of using a general purpose programming language for infrastructure-as-code projects. I’ve been using Pulumi with JavaScript (I know, some folks would say I should question my life decisions), and while installing Pulumi itself is pretty low-impact (a small group of binaries) there are a number of dependencies that need to be installed when using Pulumi with JavaScript. As I’m a stickler for keeping my primary system very “clean” with regard to installed packages and software, I thought I’d create a means whereby I can easily spin up a “sandbox environment” for learning Pulumi.

Read more...

Technology Short Take 113

Welcome to Technology Short Take #113! I hope the collection of links and articles I’ve gathered for you contains something useful for you. I think I have a pretty balanced collection this time around; there’s a little bit of something for almost everyone. Who says you can’t please everyone all the time?

Read more...

Technology Short Take 112

Welcome to Technology Short Take #112! It’s been quite a while since the last one, as life and work have been keeping me busy. I have, however, finally managed to pull together this list of links and articles from around the Internet, and I hope that something I’ve included here proves useful to readers.

Read more...

Using Kubeadm to Add New Control Plane Nodes with AWS Integration

In my recent post on using kubeadm to set up a Kubernetes 1.13 cluster with AWS integration, I mentioned that I was still working out the details on enabling AWS integration (via the AWS cloud provider) while also using new functionality in kubeadm (specifically, the --experimental-control-plane flag) to make it easier to join new control plane nodes to the cluster. In this post, I’ll share with you what I’ve found to make this work.

Read more...

Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!