Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Setting up K8s on AWS with Kubeadm and Manual Certificate Distribution

Credit for this post goes to Christian Del Pino, who created this content and was willing to let me publish it here.

The topic of setting up Kubernetes on AWS (including the use of the AWS cloud provider) is a topic I’ve tackled a few different times here on this site (see here, here, and here for other posts on this subject). In this post, I’ll share information provided to me by a reader, Christian Del Pino, about setting up Kubernetes on AWS with kubeadm but using manual certificate distribution (in other words, not allowing kubeadm to distribute certificates among multiple control plane nodes). As I pointed out above, all this content came from Christian Del Pino; I’m merely sharing it here with his permission.

This post specifically builds upon this post on setting up an AWS-integrated Kubernetes 1.15 cluster, but is written for Kubernetes 1.17. As mentioned above, this post will also show you how to manually distribute the certificates that kubeadm generates to other control plane nodes.

What this post won’t cover are the details on some of the prerequisites for making the AWS cloud provider function properly; specifically, this post won’t discuss:

  • Setting node hotnames to match EC2’s Private DNS entry
  • Creating and assigning the IAM instance profile to allow access to the AWS APIs
  • Assigning the correct tags to AWS resources as required by the AWS cloud provider

I encourage you to refer to the link above for details on these specific areas.

Now, let’s take a look at the process that Christian tested and shared with me.

Bootstrapping the First Control Plane Node

To build the first control plane node, you need to setup a configuration file that will be used by kubeadm to perform an initialization. (Again, this assumes all necessary prerequisites described above have been satisfied.) Here is a sample configuration file:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: aws
clusterName:
controlPlaneEndpoint: cp-nlb.elb.us-east-2.amazonaws.com
controllerManager:
  extraArgs:
    cloud-provider: aws
    configure-cloud-routes: "false"
kubernetesVersion: v1.17.0
networking:
  podSubnet: 192.168.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: aws

The explanation of this configuration file is already pretty well-covered in this earlier post.

Once you have the configuration file set to your liking, run the following command to setup your first control plane:

kubeadm init --config=init.yaml

(Replace init.yaml with the correct filename, of course.)

Assuming the process of initializing the first control plane node works as expected, then the next step is to install a CNI plugin, as documented here.

To confirm that your first control plane is up and ready, just run this command:

kubectl get nodes

If this command completes successfully and shows the first control plane node as “Ready,” then you are now ready to add more control plane nodes. However, since the --upload-certs flag was not specified during the kubeadm init command, the certificates generated on the first control plane node will not be shared across all the control plane nodes automatically. The next section explains how to manually copy the appropriate files across to additional control plane nodes.

Manually Distributing Certificates

There are a couple different reasons why a user might prefer to manually distribute certificates. First, the encryption key used by the --upload-certs flag expires after two hours. Therefore, it has been more than two hours, you need to either run kubeadm init phase upload-certs or manually copy the certificates across to other nodes. Second, you may want to use automation tools to deploy your cluster. One way to copy the certificates is to use ssh and scp, as described in the kubeadm documentation here.

You could also use S3 as a means of distributing certificates (more on that shortly).

Regardless of the method, here is the exact list of certificates that need to be copied over from the first control plane node:

/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

If you’d like to use scp, then simply copy these files across to the EC2 instance(s) you’d like to use as additional control plane nodes.

Since this cluster will be using the AWS cloud provider, EC2 instances have to be booted with an IAM instance profile that enables access to the AWS APIs. If that IAM instance profile includes S3 permissions, and if you can install the AWS CLI tools on the EC2 instances, you can use the aws s3 cp command to upload the certificates to a bucket of your choice (just ensure the bucket permissions are appropriately secured). Once the certificates are uploaded to S3 (or copied over), you can proceed to add more control plane nodes.

Adding Additional Control Plane Nodes

Before adding more control planes, it is necessary to pull down the certificates from the S3 bucket using the aws s3 cp command into the /etc/kubernetes/pki directory (if you didn’t copy them across directly using scp). You must keep the same structure as in the first control plane:

/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

Once the certificates are in place, you can build the configuration file for your joining control planes:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: 123456.i2v5ub39hrvf51j3
    apiServerEndpoint: cp-nlb.elb.us-east-2.amazonaws.com
    caCertHashes:
      - "sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"
nodeRegistration:
  name: ip-10-10-100-155.us-east-2.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws
controlPlane:
  localAPIEndpoint:
    advertiseAddress: 10.10.100.155

Refer back to the earlier post for information on the parameters in this configuration file. The key difference here is that the configuration file does not include the certificateKey parameter since the original kubeadm init command did not include the --upload-certs flag.

Once the file is complete, proceed to run the following to join your additional control plane nodes:

kubeadm join --config=cp-join.yaml

(Replace cp-join.yaml with the correct filename you used.)

Once run, the new control plane node should join the cluster. You can confirm that the additional control planes have joined the cluster by running kubectl get nodes and make sure the additional nodes are reporting as “Ready”.

Adding Worker Nodes

Adding worker nodes is simple. There are no certificates to copy and the kubeadm configuration to join the cluster is similar to that of the joining control plane nodes. Here is a sample:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: 123456.i2v5ub39hrvf51j3
    apiServerEndpoint: "cp-nlb.elb.us-east-2.amazonaws.com"
    caCertHashes:
      - "sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"
nodeRegistration:
  name: ip-10-10-100-133.us-east-2.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws

As outlined here, you’ll need to be sure your version of this file uses the correct bootstrap token, the correct API server endpoint, the correct CA certificate hash, and the correct node name. Once the file is complete, you will run the following to join your worker nodes:

kubeadm join --config=worker-join.yaml

(Substitute the filename you used for worker-join.yaml in the above command.)

Upon running this command, the worker node should join the cluster. Once again, you can use kubectl get nodes to verify that the node(s) have joined the cluster.

Summary

As you can see above, the process that Christian shared with me is extremely similar to what I shared here, with the exception of manually distributing the certificates (and Christian mentions a couple very good reasons above why manual certificate distribution may be preferred). This post also shows that the process for establishing an AWS-integrated Kubernetes cluster is consistent for versions 1.15, 1.16, and 1.17 (you just need to change the kubernetesVersion value in the configuration file you use to bootstrap the first control plane node).

I hope this information is helpful. Thanks to Christian for sharing this information with me and allowing me to publish it here. If you have any questions, feel free to hit me on Twitter or find me on the Kubernetes Slack instance.

Building an Isolated Kubernetes Cluster on AWS

In this post, I’m going to explore what’s required in order to build an isolated—or Internet-restricted—Kubernetes cluster on AWS with full AWS cloud provider integration. Here the term “isolated” means “no Internet access.” I initially was using the term “air-gapped,” but these aren’t technically air-gapped so I thought isolated (or Internet-restricted) may be a better descriptor. Either way, the intent of this post is to help guide readers through the process of setting up a Kubernetes cluster on AWS—with full AWS cloud provider integration—using systems that have no Internet access.

At a high-level, the process looks something like this:

  1. Build preconfigured AMIs that you’ll use for the instances running Kubernetes.
  2. Stand up your AWS infrastructure, including necessary VPC endpoints for AWS services.
  3. Preload any additional container images, if needed.
  4. Bootstrap your cluster using kubeadm.

It’s important to note that this guide does not replace my earlier post on setting up an AWS-integrated Kubernetes cluster on AWS (written for 1.15, but valid for 1.16 and 1.17). All the requirements in that post still apply here. If you haven’t read that post or aren’t familiar with the requirements for setting up a Kubernetes cluster with the AWS cloud provider, I strongly recommend you go read the linked article. As I walk through the four steps I outlined above, I’ll be supplementing the information provided in that blog post.

Build Preconfigured AMIs

Without getting into how the acronym is pronounced (OK, fine, I’m on the three syllable side), the first step in building an isolated Kubernetes cluster is creating preconfigured AMIs that have all the necessary components already installed. It may seem obvious, but once you launch an instance based on the AMI, you won’t able to install additional packages—because you won’t have Internet access. Hence, your images must have everything they could need already done:

  • All necessary packages and dependencies need to be installed
  • All required persistent configuration changes (like disabling swap or enabling IP routing) needs to be done
  • All the core Kubernetes container images need to be preloaded/already pulled

There are any number of ways to accomplish these things; the means that you take isn’t really important. Two examples you can use as inspiration or possibly as a source for your own fork are Wardroom and the Kubernetes “image-builder” repository (specifically, the section of the repository focused on building Cluster API [CAPI] images). Both Wardroom and the CAPI portion of the “image-builder” repository are based on Packer and Ansible. For my testing, I used AMIs built using the Kubernetes “image-builder” repository.

Once your AMIs have been created, then you’re ready to move to the next step: standing up the necessary AWS infrastructure.

Stand up AWS Infrastructure

I won’t go into too much detail here; this is already well-covered in other posts (see the August 2019 article or my original September 2018 article on setting up the AWS cloud provider). I will add a few salient points that are specific to setting up an isolated Kubernetes cluster:

  • All of the Kubernetes nodes will land on private subnets, for which the setting to automatically assign a public IP address is turned off (disabled).
  • You’ll need at least one public subnet (for a bastion host, or whatever you’re going to use to gain access to the private subnets).
  • Your public subnet will need an Internet gateway or NAT gateway for Internet access. The private subnets won’t need anything, naturally.
  • If you want to be able to use an Internet-facing ELB to expose Kubernetes Services running on the Internet-restricted cluster (using a Service of type LoadBalancer, for example), you’ll want to be sure that a public subnet exists in each availability zone (AZ) where a private subnet exists that might contain Kubernetes nodes.
  • The private subnets won’t have Internet access, but the AWS cloud provider needs to make a call to the EC2 and Elastic Load Balancing (ELB) API endpoints. By default, these are URLs that resolve to a public IP address. The instances won’t be able to connect to the public IP addresses associated with these endpoints, and thus the AWS cloud provider will fail. To fix this, you need two VPC endpoints in the VPC where you are building the isolated cluster. One of the endpoints is for the EC2 service, and the second is for the ELB service. Both of these will be for the region where you’re building the isolated Kubernetes cluster, and both of these VPC endpoints need to have private DNS enabled. (I wrote about how to create a VPC endpoint programmatically using Pulumi in this post, in case you want to tackle this with an infrastructure-as-code approach.) Note that the AWS cloud provider’s Elastic Block Store (EBS) integration uses the EC2 APIs.
  • The AWS resources need to have the correct tags assigned for the AWS cloud provider to work. This is explained in more detail in the linked posts above, as are the IAM role and policy requirements for the AWS cloud provider to work.

You’ll want to be sure that the VPC endpoints are properly registered with private DNS entries before you proceed. Use dig (or whatever DNS lookup tool may be installed on your instances) to verify that the instances can resolve ec2.<region>.amazonaws.com and elasticloadbalancing.<region>.amazonaws.com, receiving back IP addresses that are part of the private subnets where your instances reside.

Preload Additional Container Images

Out of all the various tools I’ve seen for building preconfigured images—to be fair, I’m sure I haven’t seen all of them, or even a significant fraction of them—none of them handle preloading/pulling container images other than those required by Kubernetes. In particular, none of them address the additional container images needed by your CNI plugin.

This is not a fault of these tools. The vast majority of the time, users are a) building clusters that have Internet access, and therefore it isn’t necessary to preload the CNI plugin’s container images; and b) the choice of CNI plugin is generally a “run-time” decision, not a “build-time” decision, meaning that most of the time users won’t/don’t know in advance when building a preconfigured image which CNI plugin they will use.

In this case, though, you have to know which CNI plugin you’re using because you can’t simply rely on Kubernetes to pull whatever images are needed (it won’t work since there’s no Internet access). So you have two options:

  1. Modify whatever tool you’re using to build your preconfigured images to also preload the container images needed by your CNI plugin of choice.
  2. Manually load the container images on instances by using a bastion host or VPN from a system that does have Internet access. (It’s pretty likely you’ll have either an SSH bastion host or a VPN into the private subnets, since that will be the most expedient way to gain access to the instances running on those private subnets.)

For the second option, I recently documented the process for preloading container images for use by Kubernetes using the CLI on containerd-based instances (the Kubernetes “image-builder” project creates machine images running containerd). As noted in that article, you can use Docker on your personal laptop (or some other system) to do docker pull and then docker save to get the images, then use ctr and crictl on the back-end Kubernetes nodes to import the images.

Regardless of which option you take—preloading the container images into the preconfigured machine images, or manually loading them in via some other system with connectivity—you’ll want to be sure all necessary container images are present on the instances before proceeding to the next step. For containerd-based systems, use crictl images to show all the container images that Kubernetes will be able to see when it is bootstrapped.

The Kubernetes-related images that should be present when you run this command are:

k8s.gcr.io/coredns
k8s.gcr.io/etcd
k8s.gcr.io/kube-apiserver
k8s.gcr.io/kube-controller-manager
k8s.gcr.io/kube-proxy
k8s.gcr.io/kube-scheduler
k8s.gcr.io/pause

You’ll also need to see listed some container images for your CNI plugin. For recent versions of Calico, for example, you should see:

calico/cni
calico/kube-controllers
calico/node
calico/pod2daemon-flexvol

Once you’ve verified that both the Kubernetes-related container images and the container images required for your CNI plugin are present, then you’re ready to proceed.

Bootstrap the Cluster

You’re now ready to bootstrap your isolated Kubernetes cluster on AWS. You can use the kubeadm configuration files in the August 2019 article for your nodes as a basis/starting point.

Note that in this particular case, I haven’t yet tested HA configurations for isolated Kubernetes clusters, so you’d want to remove the controlPlaneEndpoint line from the configuration file for your control plane node. That would make the kubeadm configuration file for the control plane node (there will be only one) look something like this:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: aws
clusterName: blogsample
controllerManager:
  extraArgs:
    cloud-provider: aws
    configure-cloud-routes: "false"
kubernetesVersion: stable
networking:
  dnsDomain: cluster.local
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: aws

When adding worker nodes, change the discovery.bootstrapToken.apiServerEndpoint to point to the DNS name for the control plane node, making their kubeadm configuration file look something like this:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: 123456.a4v4ii39rupz51j3
    apiServerEndpoint: "ip-1-2-3-4.us-west-2.compute.internal:6443"
    caCertHashes: ["sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"]
nodeRegistration:
  name: ip-11-12-13-14.us-west-2.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws

Obviously, the sample configuration file above would need to have a valid bootstrap token and valid CA certificate hash, as output by kubeadm init on the control plane node. If you don’t know these values, this blog post explains how to get them.

Within a few minutes, you should have a non-HA (single control plane node) Kubernetes cluster up and running, with full AWS cloud provider integration. (Note you’ll need to define a StorageClass for dynamic volume provisioning to work; this isn’t done by default).

How I Tested

I tested this using Pulumi to create all necessary AWS resources, including VPC, subnets, Internet gateway (for the public subnets), VPC endpoints, security groups, and instances. I used AMIs built using the Kubernetes “image-builder” repository, and used an SSH bastion to gain access to the private instances. I preloaded the container images needed by the Calico CNI plugin manually (not baked into the image), and then used kubeadm to bootstrap the cluster. The operating system on the instances was Ubuntu 18.04.

Wrapping it up

Hopefully this walk-through is helpful in describing some of the challenges of setting up an isolated Kubernetes cluster on AWS. While this procedure lets you get an isolated cluster up and running, there are a whole host of considerations for operating an isolated cluster:

  • How are you going to get container images into the cluster? (I have a simple idea about how to handle this, thanks to a conversation with my colleague Duffie Cooley.)
  • What applications are going to run on the cluster, and how will users (or other applications) access them?
  • Kubernetes Services of type LoadBalancer will spawn an ELB, but by default those ELBs will be externally accessible. Is that what’s intended? (If not, you’ll want to be sure the service.beta.kubernetes.io/aws-load-balancer-internal annotation is present in your Service manifest.)

I’d strongly encourage readers to be sure to think through these questions—among others—before proceeding, and make sure that this design fits your specific use case.

If anyone has any questions, feel free to hit me up on the Kubernetes Slack instance; I can be found hanging out in the #kubeadm and #kubernetes-users channels, among others. (Sign up here if you’re not already a registered user of the Kubernetes Slack instance.) Alternately, feel free to contact me via Twitter. I’d love to hear any feedback, suggestions for improvement, or corrections. Thanks!

Creating an AWS VPC Endpoint with Pulumi

In this post, I’d like to show readers how to use Pulumi to create a VPC endpoint on AWS. Until recently, I’d heard of VPC endpoints but hadn’t really taken the time to fully understand what they were or how they might be used. That changed when I was presented with a requirement for the AWS EC2 APIs to be available within a VPC that did not have Internet access. As it turns out—and as many readers are probably already aware—this is one of the key use cases for a VPC endpoint (see the VPC endpoint docs). The sample code I’ll share below shows how to programmatically create a VPC endpoint for use in infrastructure-as-code use cases.

For those that aren’t familiar, Pulumi allows users to use one of a number of different general-purpose programming languages and apply them to infrastructure-as-code scenarios. In this example, I’ll be using TypeScript, but Pulumi also supports JavaScript and Python (and Go is in the works). (Side note: I intend to start working with the Go support in Pulumi when it becomes generally available as a means of helping accelerate my own Go learning.)

Here’s a snippet of TypeScript code that will create a VPC endpoint (I’ve randomized the resource IDs of the AWS resources below):

const ec2ApiEndpoint = new aws.ec2.VpcEndpoint("test-vpc-endpoint", {
    privateDnsEnabled: true,
    serviceName: "com.amazonaws.us-west-2.ec2",
    securityGroupIds: [ "sg-1db01e1f9be081d76" ],
    subnetIds: [ "subnet-0db757befe2c48624" ],
    vpcEndpointType: "Interface",
    vpcId: "vpc-0e2c348614e845763",
    tags: {
        Name: "test-vpc-endpoint"
    }
});

Let’s walk through this a bit (the full API reference is here):

  • The privateDnsEnabled: true line enables us to use the “standard” EC2 API URL endpoint within the VPC and have it resolve to a private IP address out of the VPC/subnets. This relies on Route 53, and is necessary for use cases where you (the user) don’t have any control over how the API endpoint is being called from within the VPC (it may be a bit of commercial software you can’t modify). In this case, any software trying to reach “ec2.us-west-2.amazonaws.com” will resolve to a private IP address instead of the public IP address.
  • The serviceName specifies which service is available via this endpoint (naturally, this is for EC2).
  • The vpcEndpointType specifies “Interface”, which means that an ENI is created to allow traffic to reach the service represented by this endpoint. The other option is “Gateway”, which would require you to modify route tables, and is only supported for some services (like S3). This article provides an example of a gateway endpoint.

The other settings are all pretty self-explanatory. This being TypeScript, you could—of course—use references to variables or outputs from other portions of your Pulumi code instead of hard-coding values as I have in this example (and that would be highly recommended). This could be easily incorporated into a large portion of code that created a new VPC, new subnets, etc.

And that’s it! I know this is a fairly simple example, but I hadn’t found very many articles showing how this is done, and thought that sharing this may prove helpful to others. Feel free to contact me on Twitter if you have corrections (in the event I made a mistake), suggestions for improvement, or other feedback.

Manually Loading Container Images with containerD

I recently had a need to manually load some container images into a Linux system running containerd (instead of Docker) as the container runtime. I say “manually load some images” because this system was isolated from the Internet, and so simply running a container and having containerd automatically pull the image from an image registry wasn’t going to work. The process for working around the lack of Internet access isn’t difficult, but didn’t seem to be documented anywhere that I could readily find using a general web search. I thought publishing it here may help individuals seeking this information in the future.

For an administrator/operations-minded user, the primary means of interacting with containerd is via the ctr command-line tool. This tool uses a command syntax very similar to Docker, so users familiar with Docker should be able to be productive with ctr pretty easily.

In my specific example, I had a bastion host with Internet access, and a couple of hosts behind the bastion that did not have Internet access. It was the hosts behind the bastion that needed the container images preloaded. So, I used the ctr tool to fetch and prepare the images on the bastion, then transferred the images to the isolated systems and loaded them. Here’s the process I followed:

  1. On the bastion host, first I downloaded (pulled) the image from a public registry using ctr image pull (the example I’ll use here is for the Calico node container image, used by the Calico CNI in Kubernetes clusters):

    ctr image pull docker.io/calico/node:v3.11.2
    

    (Note that sudo may be needed for all these ctr commands; that will depend on your system configuration.)

    I repeated this process for all the other images the isolated systems also needed.

    If you have a system (like your local laptop) running Docker, then you can use docker pull here instead; just note that you may need to adjust the path/URL to the image/image registry.

  2. Still on the bastion host, I exported the pulled images to standalone archives:

    ctr image export calico-node-v3.11.2.tar \
    docker.io/calico/node:v3.11.2
    

    The general format for this command looks like this:

    ctr image export <output-filename> <image-name>
    

    If you don’t know what the image name (according to containerd) is, use ctr image ls.

    If you’re using a system with Docker installed (maybe you’re using your local laptop), then docker save <image-name> -o <output-filename will get you an image you can use in the subsequent steps.

  3. After transferring the standalone archives to the other systems (using whatever means you prefer; I used scp), then load (or import) the images into containerd with this command:

    ctr image import <filename-from-previous-step>
    

    Repeat as needed for additional images. It appears, by the way, that using wildcards in the ctr image import command won’t work; I had to manually specify each individual file for import.

    If you need these images to be available to Kubernetes, you must be sure to add the -n=k8s.io flag to the ctr image import command, like this:

    ctr -n=k8s.io images import <filename-from-previous-step>
    
  4. Verify that the image(s) are present and recognized by containerd using ctr image ls.

    If you specified the k8s.io namespace when importing the images in the previous step—so as to make the images available to Kubernetes—then you can verify that CRI (Container Runtime Interface, the means by which Kubernetes talks to containerd) sees these images by running crictl images (again, sudo may be required, based on your configuration).

That should do it for you!

How I Tested

To test this procedure, I used multiple Ubuntu 18.04 instances on AWS. One instance had Internet access (the bastion host); the others did not. The instances had been built from AMIs prepared using Packer and Ansible with playbooks from the Kubernetes “image-builder” repository (specifically, the playbooks used to build the Kubernetes Cluster API images), so containerd and all the other CLI tools mentioned in this article were already installed and available for use.

Additional Resources

I did find a few sites with some containerd information while searching, and wanted to include them here for credit and additional context:

Getting started with Containerd

crictl User Guide

I hope the information in this post is helpful. If you have questions, comments, suggestions, or corrections, you are welcome to contact me on Twitter, or drop me an e-mail (my e-mail address isn’t too hard to find).

UPDATE: I’ve added a few Docker commands to steps #1 and #2 above for users who may be using a Docker-powered system, like their local laptop, to retrieve images from the Internet that will be transferred to the containerd-powered systems for importing.

Thinking and Learning About API Design

In July of 2018 I talked about Polyglot, a very simple project I’d launched whose only purpose was simply to bolster my software development skills. Work on Polyglot has been sporadic at best, coming in fits and spurts, and thus far focused on building a model for the APIs that would be found in the project. Since I am not a software engineer by training (I have no formal training in software development), all of this is new to me, and I’ve found myself encountering lots of questions about API design along the way. In the interest of helping others who may be in a similar situation, I thought I’d share a bit here.

I initially approached the API in terms of how I would encode (serialize?) data on the wire using JSON (I’d decided on using a RESTful API with JSON over HTTP). Starting with how I anticipated storing the data in the back-end database, I created a representation of how a customer’s information would be encoded (serialized) in JSON:

{
    "customers": [
        {
            "customerID": "5678",
            "streetAddress": "123 Main Street",
            "unitNumber": "Suite 123",
            "city": "Anywhere",
            "state": "CO",
            "postalCode": "80108",
            "telephone": "3035551212",
            "primaryContactFirstName": "Scott",
            "primaryContactLastName": "Lowe"
        }
    ]
}

It’s pretty obvious, though, that this representation is closely tied to how I anticipate I’ll store customer data in the back-end database, right down to the field names. One of the books I’ve been reading is Building Microservices, an O’Reilly book by Sam Newman (see here; I’ll post a review of the book once I complete it). One of the things Newman cautions readers against in his book is exposing implementation details internal to a service to other services—did this qualify? Newman also discussed hypermedia as the engine of application state (HATEOAS, what an acronym!), but I’ll be honest in admitting that it didn’t click with me at first.

While still working through Newman’s book, I added REST API Design Rulebook by Mark Masse to the mix (see my review of the book). Masse talked extensively about HATEOAS, but—as you can see in my review of the book—it still didn’t click for me.

Next up, I added RESTful Web APIs by Leonard Richardson, Sam Ruby, and Mike Amundsen to my reading list (here’s the O’Reilly page for the book; I’m still reading the book so no review yet). Whether by repetition or by seeing the information presented in a different context, the discussion of HATEOAS and hypermedia by Richardson et al. finally made sense. It also unlocked some of the ideas presented by Masse and Newman earlier, so that I was able to really understand that I’d made several critical mistakes in my approach thus far:

  • I chose a media type (JSON) before fully understanding the particulars of my application without realizing the constraints that media type creates.
  • The client representation was too closely tied to the back-end/server-side implementation, making things far too brittle (what would happen if I needed to change the database schema?).
  • I invented new field names instead of re-using properties and fields that are already well-known and well-accepted (like re-using information from an appropriate microformat).
  • The client representation doesn’t provide any information on state transitions (How is a customer record updated? How is a customer record deleted?) or relationships between resources (What’s the relationship between an order record and a customer record? Or vice versa?).

It’s clear that something more than just encoding data in JSON is required.

The question, for me, is how much more? And what to include, exactly? Richardson et al. seem to advocate the Collection+JSON format, but it’s only a personal standard (see his book for definitions of various standards) and not necessarily widely accepted/used. However, even within Collection+JSON, the specific links provided as hypermedia are left as “implementation details.” So what sort of information makes sense?

Given that Polyglot is intended to be a simple order entry/lookup system, a few potential ideas come to mind:

  • All discussions of HATEOAS and hypermedia agree that a “self” link is needed that provides the URL to the resource (customer record or order record) being returned.
  • It seems reasonable to return “related” information. So, if viewing an order record, provide a link to the associated customer record (for example), the previous order, the next order, etc.

Clearly I have lots of work to do!

Now comes the task of shaping the limited work I’ve done so far into something more usable. I also need to finish reading RESTful Web APIs; although the content was difficult for me to grok at first, recently I’ve found that I’m beginning to more fully understand the concepts the authors are presenting. I still have lots of questions—how do things like OpenAPI fit in here?—so if you have answers, feel free to hit me on Twitter. This is definitely a learning exercise for me, so I welcome any other resources I should be using to help guide my way.

Recent Posts

Technology Short Take 123

Welcome to Technology Short Take #123, the first of 2020! I hope that everyone had a wonderful holiday season, but now it’s time to jump back into the fray with a collection of technical articles from around the Internet. Here’s hoping that I found something useful for you!

Read more...

Removing Unnecessary Complexity

Recently, I’ve been working to remove unnecessary complexity from my work environment. I wouldn’t say that I’m going full-on minimalist (not that there’s anything wrong with that), but I was beginning to feel like maintaining this complexity was taking focus, willpower, and mental capacity away from other, more valuable, efforts. Additionally, given the challenges I know lie ahead of me this year (see here for more information), I suspect I’ll need all the focus, willpower, and mental capacity I can get!

Read more...

Looking Back: 2019 Project Report Card

As has been my custom over the last five years or so, in the early part of the year I like to share with my readers a list of personal projects for the upcoming year (here’s the 2019 list). Then, near the end of that same year or very early in the following year, I evaluate how I performed against that list of personal projects (for example, here’s my project report card for 2018). In this post, I’ll continue that pattern with an evaluation of my progress against my 2019 project list.

Read more...

New Year, New Adventure

I’ll skip the build-up and jump straight to the whole point of this post: a once-in-a-lifetime opportunity has come up and I’m embarking on a new adventure starting in early 2020. No, I’m not changing jobs…but I am changing time zones.

Read more...

Technology Short Take 122

Welcome to Technology Short Take #122! Luckily I did manage to get another Tech Short Take squeezed in for 2019, just so all my readers could have some reading materials for the holidays. I’m kidding! No, I mean I really am kidding—don’t read stuff over the holidays. Spend time with your family instead. The investment in your family will pay off in later years, trust me.

Read more...

Technology Short Take 121

Welcome to Technology Short Take #121! This may possibly be the last Tech Short Take of 2019 (not sure if I’ll be able to squeeze in another one), so here’s hoping that you find something useful, helpful, or informative in the links that I’ve collected. Enjoy some light reading over your festive holiday season!

Read more...

Technology Short Take 120

Welcome to Technology Short Take #120! Wow…hard to believe it’s been almost two months since the last Tech Short Take. Sorry about that! Hopefully something I share here in this Tech Short Take is useful or helpful to readers. On to the content!

Read more...

KubeCon 2019 Day 3 and Event Summary

Keynotes Bryan Liles kicked off the day 3 morning keynotes with a discussion of “finding Kubernetes’ Rails moment”—basically focusing on how Kubernetes enables folks to work on/solve higher-level problems. Key phrase from Bryan’s discussion (which, as usual, incorporated the humor I love to see from Bryan): “Kubernetes isn’t the destination. Kubernetes is the vehicle that takes us to the destination.” Ian Coldwater delivered a talk on looking at Kubernetes from the attacker’s point of view, and using that perspective to secure and harden Kubernetes.Read more...

KubeCon 2019 Day 2 Summary

Keynotes This morning’s keynotes were, in my opinion, better than yesterday’s morning keynotes. (I missed the closing keynotes yesterday due to customer meetings and calls.) Only a couple of keynotes really stuck out. Vicki Cheung provided some useful suggestions for tools that are helping to “close the gap” on user experience, and there was an interesting (but a bit overly long) session with a live demo on running a 5G mobile core on Kubernetes.Read more...

KubeCon 2019 Day 1 Summary

This week I’m in San Diego for KubeCon + CloudNativeCon. Instead of liveblogging each session individually, I thought I might instead attempt a “daily summary” post that captures highlights from all the sessions each day. Here’s my recap of day 1 at KubeCon + CloudNativeCon.

Read more...

Using Kustomize with Cluster API Manifests

A topic that’s been in the back of my mind since writing the Cluster API introduction post is how someone could use kustomize to modify the Cluster API manifests. Fortunately, this is reasonably straightforward. It doesn’t require any “hacks” like those needed to use kustomize with kubeadm configuration files, but similar to modifying kubeadm configuration files you’ll generally need to use the patching functionality of kustomize when working with Cluster API manifests. In this post, I’d like to take a fairly detailed look at how someone might go about using kustomize with Cluster API.

Read more...

Programmatically Creating Kubernetes Manifests

A while ago I came across a utility named jk, which purported to be able to create structured text files—in JSON, YAML, or HCL—using JavaScript (or TypeScript that has been transpiled into JavaScript). One of the use cases was creating Kubernetes manifests. The GitHub repository for jk describes it as “a data templating tool”, and that’s accurate for simple use cases. In more complex use cases, the use of a general-purpose programming language like JavaScript in jk reveals that the tool has the potential to be much more than just a data templating tool—if you have the JavaScript expertise to unlock that potential.

Read more...

Spousetivities in Barcelona at VMworld EMEA 2019

Barcelona is probably my favorite city in Europe—which works out well, since VMware seems to have settled on Barcelona at the destination for VMworld EMEA. VMworld is back in Barcelona again this year, and I’m fortunate enough to be able to attend. VMworld in Barcelona wouldn’t be the same without Spousetivities, though, and I’m happy to report that Spousetivities will be in Barcelona. In fact, registration is already open!

Read more...

Using Kustomize with Kubeadm Configuration Files

Last week I had a crazy idea: if kustomize can be used to modify YAML files like Kubernetes manifests, then could one use kustomize to modify a kubeadm configuration file, which is also a YAML manifest? So I asked about it in one of the Kubernetes-related channels in Slack at work, and as it turns out it’s not such a crazy idea after all! So, in this post, I’ll show you how to use kustomize to modify kubeadm configuration files.

Read more...

Technology Short Take 119

Welcome to Technology Short Take #119! As usual, I’ve collected some articles and links from around the Internet pertaining to various data center- and cloud-related topics. This installation in the Tech Short Takes series is much shorter than usual, but hopefully I’ve managed to find something that proves to be helpful or informative! Now, on to the content!

Read more...

Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!