Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Follow Up: Bootstrapping Servers into Ansible

Seven years ago, I wrote a quick post on bootstrapping servers into Ansible. The basic gist of the post was that you can use variables on the Ansible command-line to specify hosts that aren’t part of your inventory or log in via a different user (useful if the host doesn’t yet have a dedicated Ansible user account because you want to use Ansible to create that account). Recently, though, I encountered a situation where this approach doesn’t work, and in this post I’ll describe the workaround.

In one of the Slack communities I frequent, someone asked about using the approach described in the original blog post. However, they were having issues connecting. Specifically, this error was cropping up in the Ansible output (names have been changed to protect the innocent):

fatal: []: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,password).", "unreachable": true}

Now, this is odd, because the Ansible command-line being executed included the parameters I mentioned in the original blog post:

ansible-playbook bootstrap.yml -i inventory/hosts -K --extra-vars " user=john"

For some reason, though, it was ignoring that parameter and attempting to connect as ansible. At first, I thought this was just an SSH error, but the person assured me that they were definitely able to SSH into the specified host as the specified user.

That was when we found the difference between this environment and the environment I’d used: in this environment, the ansible_user variable was defined in the specified inventory file. I hadn’t defined ansible_user. Could that be it?

I asked the person to try specifying ansible_user instead of user on the command line, like so:

ansible-playbook bootstrap.yml -i inventory/hosts -K --extra-vars " ansible_user=john"

It worked!

So, the key takeaway is: if you’re thinking of using this bootstrapping technique with your Ansible environment, be sure to specify ansible_user on the Ansible command-line if you’ve defined ansible_user in your inventory. Otherwise, you should be able to just use user.

Credit for finding this issue with the approach I described in the original blog post, and for testing the workaround, goes to James Cox. Thanks James! As always, feel free to reach out to me on Twitter—or find me in any of the Slack communities I regularly visit—and ask questions, provide feedback, or just say hello. I’d love to hear from you.

Technology Short Take 151

Welcome to Technology Short Take #151, the first Technology Short Take of 2022. I hope everyone had a great holiday season and that 2022 is off to a wonderful start! I have a few more links than normal this time around, although I didn’t find articles in a couple categories. Don’t worry—I’ll keep my eyes peeled and my RSS reader ready to pull in new articles in those categories for next time. And now for the content!




Cloud Computing/Cloud Management

Operating Systems/Applications

  • I recently had a need to do a multicast DNS lookup, and this article was critical in figuring out how to use dig to do it.
  • If you need a quick start for HashiCorp Vault, this one worked really well for me. I found it easier/better than the documentation on the HashiCorp web site, in fact.
  • Dennis Felsing shares some thoughts on switching to macOS after 15 years on Linux.
  • Running Docker on an M1 Max-based system? This article may provide some useful information.
  • Here’s some information on why Microsoft Exchange stopped delivering e-mail messages on January 1, 2022.
  • Mark Brookfield has started a series on automating SSL certificate issuance and renewal using HashiCorp Vault Agent, here’s part 1.
  • Julia Evans has a quick post on finding a domain’s authoritative name servers. I was already familiar with this process, but I appreciate that there are lots of folks out there who may not have had to ever do this.
  • Here’s a handy list of secret phone codes you can use. Some of these I already knew, but a few of them were new to me. Neat.
  • The founder of the Nginx project recently stepped away from the project. Read more here.
  • BIOS updates without a reboot, and under Linux first? Yes please!

That’s all for now. I hope this was useful in some way! If you have any feedback for me—constructive criticism, praise, suggestions for where I can find more articles (especially if the site supports RSS!), feel free to reach out. I’d love to hear from you! You can reach me on Twitter, or hit me up in any one of a number of different Slack communities.

Getting Certificate Details from HashiCorp Vault

It seems there are lots of tutorials on setting up a PKI (public key infrastructure) using HashiCorp Vault. What I’ve found missing from most of these tutorials, however, is how to get details on certificates issued by a Vault-driven PKI after the initial creation. For example, someone other than you issued a certificate, but now you need to get the details for said certificate. How is that done? In this post, I’ll show you a couple ways to get details on certificates issued and stored in HashiCorp Vault.

For the commands and API calls I’ve shared below, I’m using “pki” as the name/path you (or someone else) assigned to a PKI secrets engine within Vault. If you’re using a different name/path, then be sure to substitute the correct name/path as appropriate.

To use the Vault CLI to see the list of certificates issued by Vault, you can use this command:

vault list pki/certs

This will return a list of the serial numbers of the certificates issued by this PKI. Looking at just serial numbers isn’t terribly helpful, though. To get more details, you first need to read the certificate details (note singular “cert” here versus plural “certs” in the previous command):

vault read pki/cert/<serial-num>

Now the output is a formatted response that includes the PEM-encoded certificate. That’s a bit more useful, but still not quite all the way to seeing all the details about the certificate. Fortunately, the Vault CLI supports a -format=json parameter, and you can couple that with the trusty tool jq to parse the output (see this post if you’re not familiar with jq):

vault read -format=json pki/cert/<serial-num> | jq -r '.data.certificate'

Finally! One more step: pipe the output of the command through openssl x509, like this:

vault read -format=json pki/cert/<serial-num> | \
jq -r '.data.certificate' | \
openssl x509 -in - -noout -text

Bam! Now you have all the details on the certificate.

It’s also possible to do this via Vault’s HTTP API. Using curl, you can retrieve the list of certificate serial numbers with this API call:

curl -s --request LIST https://<vault-ip-or-hostname>/v1/pki/certs

Armed with a serial number, you can then retrieve a specific certificate:

curl -s https://<vault-ip-or-hostname>/v1/pki/cert/<serial-num>

And then, naturally, you can use jq and openssl to see the certificate details:

curl -s https://<vault-ip-or-hostname>/v1/pki/cert/<serial-num> | \
jq -r '.data.certificate' | \
openssl x509 -in - -noout -text

Per the Vault API docs, the endpoints for retrieving the list of certificates or retrieving a specific certificate are unauthenticated endpoints, so there’s no need to include the “X-Vault-Token” header with your API requests.

I hope this is useful. If you have questions—or corrections, in the event I explained something incorrectly above—feel free to reach out to me on Twitter, or find me in any one of a number of Slack communities.

Using Test-Driven Development for Kustomize Overlays

I am by no means a developer (not by a long shot!), but I have been learning lots of development-related things over the last several years and trying to incorporate those into my workflows. One of these is the idea of test-driven development (see Wikipedia for a definition and some additional information), in which one writes tests to validate functionality before writing the code to implement said functionality (pardon the paraphrasing). In this post, I’ll discuss how to use conftest to (loosely) implement test-driven development for Kustomize overlays.

If you’re unfamiliar with Kustomize, then this introductory article I wrote will probably be useful.

For the discussion around using the principles of test-driven development for Kustomize overlays, I’ll pull in a recent post I did on creating reusable YAML for installing Kuma. In that post, I pointed out four changes that needed to be made to the output of kumactl install control-plane to make it reusable:

  1. Remove the caBundle value for all webhooks.
  2. Annotate all webhooks so that cert-manager will inject the correct caBundle value.
  3. Add a volume and volume mount to the “kuma-control-plane” Deployment.
  4. Change one of the environment variables for the “kuma-control-plane” Deployment to reference the volume added in step 3.

Since I know what specific changes I need to see in the original YAML, I should be able to write some sort of test that tells me if that change has been made or not. I can run the test against the original YAML, knowing that it will fail, before writing the necessary Kustomize patches to make the change. Then I can test again to verify if the Kustomize patches worked as expected or anticipated.

This is where conftest comes in. The conftest command-line utility uses the Rego policy language from Open Policy Agent, and is intended to allow users to do exactly what I’m trying to do. Specifically, it allows me to run a series of tests against YAML manifests to ensure that the test conditions (whatever those may be) are satisfied. The “test conditions” here are that the necessary changes have been made in order to make the installation YAML reusable.

I’ll start with the first change (removing the caBundle value). Here’s a Rego stanza that will test for the presence of a caBundle value (I’ll explain the code below):

warn[msg] {
    input.kind == "ValidatingWebhookConfiguration"
    some i; input.webhooks[i].clientConfig.caBundle
    msg = "caBundle value not removed for a validating webhook"

Before moving on to the other tests, let’s break this one down a bit:

  • The warn[msg] header tells conftest to warn with the supplied message if the following conditions are true. The conditions within the curly braces act as a logical AND; all of them must be true in order to cause conftest to warn with the supplied message.
  • The first line checks to see if input.kind is equal to ValidatingWebhookConfiguration. The input here is whatever YAML being fed to conftest, so input.kind refers to the top-level kind object in the input YAML.
  • The second line is Rego’s form of iteration. Rego will essentially iterate over all the webhooks in the input YAML and look for the presence of clientConfig.caBundle in each webhook. If it finds at least one webhook with clientConfig.caBundle present, then this condition is true.
  • The last line in the curly braces sets the message to be displayed by conftest if all the conditions are true.

I can place this test into a file, which conftest expects to be in a subfolder named “policy”, and then pipe the output of kumactl install control-plane through it like this:

kumactl install control-plane | conftest test -

This will produce a warning, of course, but that’s exactly what we wanted—we want to the test to indicate that the caBundle hasn’t been removed.

The test above checked for the caBundle value for validating webhooks; this slightly modified version does the same thing for mutating webhooks:

warn[msg] {
    input.kind == "MutatingWebhookConfiguration"
    some i; input.webhooks[i].clientConfig.caBundle
    msg = "caBundle value not removed for a validating webhook"

Let’s look at another change and how we can test for it. What about change #4 from above—changing an environment variable being passed to the “kuma-control-plane” Deployment? Here’s a Rego test for that:

warn[msg] {
    input.kind == "Deployment"
    env := input.spec.template.spec.containers[0].env[_] == "KUMA_RUNTIME_KUBERNETES_INJECTOR_CA_CERT_FILE"
    not env.value == "/var/run/secrets/"
    msg = "KUMA_RUNTIME_KUBERNETES_INJECTOR_CA_CERT_FILE is not set correctly"

The syntax here is a bit different, so let’s discuss it before moving on:

  • The header and the first line inside the braces are similar/the same as what I explained above in the previous example.
  • The second line in the braces creates a new variable, called env, and assigns it the contents of the env array of values under spec.template.spec.container[0] (i.e., the first container defined in the Deployment). The env[_] syntax is like another form of iteration, allowing Rego to perform the next two checks on all the contents of the array of values.
  • The next two lines check for a) an entry whose name field is set to KUMA_RUNTIME_KUBERNETES_INJECTOR_CA_CERT_FILE and b) whose value field is NOT set to the specified path. Recall that conditions within the braces are like a logical AND, so specifying both of these conditions allows the test to find instances where the environment variable is defined incorrectly (i.e., with a different value than what is expected/desired).
  • The test closes out with setting the message conftest will display if all the conditions are true.

The test for change #2 is reasonably straightforward:

warn[msg] {
    input.apiVersion == ""
    not input.metadata.annotations[""]
    msg = "cert-manager CA injection annotation missing from a webhook"

The only notable syntax note here is how Rego would reference the name of the annotation, which is “”. If a field name contains only letters or numbers, then the input.metadata.annotations syntax (separating each field with a period) is sufficient. When a field contains something other than letters and numbers, then you need to move to the input.metadata.annotations[""] syntax, where the field name is referenced in brackets and quotes.

Finally, I can write the tests for the last change: adding a volume and mounting it to a path in the container. This change is a bit more complex. In this case, the tests need to flag three different conditions:

  1. If the volume isn’t present
  2. If the volume is present, but configured to point to the wrong entity (it’s supposed to point to a specific Secret)
  3. If the volume isn’t mounted to the correct path in the container

As you might expect, then, I need three tests. In this case, I’m going to use a slightly different syntax that allows me to define a condition and then reference it later in a rule. First, let’s set up the condition:

general_ca_vol_present = true {
    input.kind == "Deployment"
    vols := input.spec.template.spec.volumes[_] == "general-ca-crt"

By now, this syntax should look pretty familiar to you, with the exception of the header. This defines a condition—a subroutine or function, if you will—called general_ca_vol_present. This function checks to ensure that the volume does exist in the YAML manifest. I’ll reference this in a rule in just a moment.

<aside>One side note here: a peculiarity of Rego is that I can’t test if the volume doesn’t exist. If you check for not == "general-ca-crt", this check is true because there are other volumes whose name is not “general-ca-crt”. This may require to you write your tests slightly differently than you might expect at first.</aside>

Next, a custom function to see if the volume is configured incorrectly (it doesn’t point to the correct Secret):

general_ca_vol_incorrect = true {
    input.kind == "Deployment"
    vols := input.spec.template.spec.volumes[_] == "general-ca-crt"
    not vols.secret.secretName == "kuma-root-ca"

Finally, a function to check to see if it is mounted into the container correctly:

general_ca_crt_volume_mounted = true {
    input.kind == "Deployment"
    mounts := input.spec.template.spec.containers[0].volumeMounts[_]
    mounts.mountPath == "/var/run/secrets/" == "general-ca-crt"
    mounts.readOnly == true

With the functions defined, I can now write the rules:

warn[msg] {
    input.kind == "Deployment"
    not general_ca_vol_present
    msg = "Additional volume not defined for CA of custom TLS certificate"

warn[msg] {
    input.kind == "Deployment"
    msg = "Additional volume for CA of custom TLS certificate not configured correctly"

warn[msg] {
    input.kind == "Deployment"
    not general_ca_crt_volume_mounted
    msg = "Additional volume for CA of custom TLS certificate not mounted or incorrectly mounted in container"

With all the tests in place, running kumactl install control-plane | conftest test - will generate warnings, as expected. From here, I can start to write Kustomize configurations that will make the necessary changes, and run them back through the tests like this (this command assumes the Kustomize configuration is in a directory named “kuma” under the current directory):

kustomize build kuma | conftest test -

When conftest finally reports zero warnings, I’ll know my Kustomize configuration is making the changes I coded in my tests. Bam—I’ve just used the principles behind test-driven development in the creation of a Kustomize configuration.

In this particular case, a sample Kustomize configuration that modifies the base YAML output by kumactl install control-plane is found in this GitHub repository. I’ve also added a Rego file with all the tests described in this post to the repository; it’s found in the “policy” directory. You can use conftest to verify the sample Kustomize configuration using the Rego file in the “policy” directory.

Additional Resources

As mentioned above, this GitHub repository—which was created to accompany the original reusable Kuma installation YAML blog post—has both a sample Kustomize configuation and a Rego policy file you can use with conftest. Feel free to take a closer look at either or both, or use them as the basis for something of your own.

If you have any questions, feel free to reach out to me on Twitter. I’d be happy to help, if I’m able. For help with Rego, OPA, or conftest, you can also check out the Open Policy Agent Slack community.

Technology Short Take 150

Welcome to Technology Short Take #150! This is the last Technology Short Take of 2021, so hopefully I’ll close the year out “with a bang” with this collection of links and articles on various technology areas. Bring on the content!


  • Ivan Pepelnjak has a post on running network automation tools in a container. In fact, he’s already built some container images, and the post has information on running tools from his prebuilt container image. Well worth reading!
  • Tom Hollingsworth likens networking disaggregation to “cutting the cord” and switching away from cable.



  • Nicholas Weaver (no, not that Nick Weaver) discusses the Log4Shell vulnerability.
  • The Log4J vulnerability and associated exploits has been on many folks' minds, so it’s only natural that many security companies have been looking into how to mitigate this attack vector. Aqua Security has a write-up on some of their analysis here.
  • This is an older post, but it doesn’t look like I’ve linked to it before, so I thought I’d include it here. Some Azure folks from Microsoft have published a threat matrix for Kubernetes. This is useful, in my opinion, because now platform operators have a “starting point” on the specific threats they need to try to mitigate.
  • I found this article on NSO/Pegasus a very interesting read. There’s also some great information in this Google Project Zero blog post.

Cloud Computing/Cloud Management

Operating Systems/Applications


  • Jeff Geerling describes his backup plan. (Jeff, if you aren’t aware, is a fairly prolific YouTube content creator.)


Career/Soft Skills

  • Katarina Brookfield recently shared her experience with the Certified Kubernetes Administrator (CKA) exam; you can find her notes here. Oh, and while we’re at it—you may find Curtis Collicutt’s experience with the Certified Kubernetes Security (CKS) specialist exam helpful as well.
  • This post has a list of “10 ways to prove yourself during remote work.” It seems to be primarily targeted at leadership-type roles, but readers may find a few useful tips here.

That’s a wrap! Not only for this Technology Short Take, but also for 2021. I hope that 2022 brings you much success and joy. As always, feel free to contact me on Twitter if you’d like to provide feedback, ask a question, or just say hello. I’d love to hear from you!

Recent Posts

Review: OWC Thunderbolt 3 Dock

About six months ago I purchased an OWC Thunderbolt 3 Dock to replace my Anker PowerElite Thunderbolt 3 Dock (see my review here). While there was nothing necessarily wrong with the Anker PowerElite, it lacked a digital audio port that I could use to send audio to a soundbar positioned under my monitor. (I’d grown accustomed to using a soundbar when my 2012 Mac Pro was my primary workstation.) In this post, I’ll provide a brief review of the OWC Thunderbolt 3 Dock.


Technology Short Take 149

Welcome to Technology Short Take #149! I’ll have one more Technology Short Take in 2021, scheduled for three weeks from now (on the last day of the year!). For now, though, I have a small collection of articles and links for your reading pleasure—not as many as I usually include in a Technology Short Take, but better than nothing at all (I hope!). Enjoy!


Technology Short Take 148

Welcome to Technology Short Take #148, aka the Thanksgiving Edition (at least, for US readers). I’ve been scouring RSS feeds and various social media sites, collecting as many useful links and articles as I can find: from networking hardware and networking CI/CD pipelines to Kernel TLS and tricks for improving your working memory. That’s quite the range! I hope that you find something useful here.


Using Kustomize Components with Cluster API

I’ve been using Kustomize with Cluster API (CAPI) to manage my AWS-based Kubernetes clusters for quite a while (along with Pulumi for managing the underlying AWS infrastructure). For all the time I’ve been using this approach, I’ve also been unhappy with the overlay-based approach that had evolved as a way of managing multiple workload clusters. With the recent release of CAPI 1.0 and the v1beta1 API, I took this opportunity to see if there was a better way. I found a different way—time will tell if it is a better way. In this post, I’ll share how I’m using Kustomize components to help streamline managing multiple CAPI workload clusters.


Technology Short Take 147

Welcome to Technology Short Take #147! The list of articles is a bit shorter than usual this time around, but I’ve still got a good collection of articles and posts covering topics in networking, hardware (mostly focused on Apple’s processors), cloud computing, and virtualization. There’s bound to be something in here for most everyone! (At least, I hope so.) Enjoy your weekend reading!


Influencing Cluster API AMI Selection

The Kubernetes Cluster API (CAPI) project—which recently released v1.0—can, if you wish, help manage the underlying infrastructure associated with a cluster. (You’re also fully able to have CAPI use existing infrastructure as well.) Speaking specifically of AWS, this means that the Cluster API Provider for AWS is able to manage VPCs, subnets, routes and route tables, gateways, and—of course—EC2 instances. These EC2 instances are booted from a set of AMIs (Amazon Machine Images, definitely pronounced “ay-em-eye” with three syllables) that are prepared and maintained by the CAPI project. In this short and simple post, I’ll show you how to influence the AMI selection process that CAPI’s AWS provider uses.


Creating Reusable Kuma Installation YAML

Using CLI tools—instead of a “wall of YAML”—to install things onto Kubernetes is a growing trend, it seems. Istio and Cilium, for example, each have a CLI tool for installing their respective project. I get the reasons why; you can build logic into a CLI tool that you can’t build into a YAML file. Kuma, the open source service mesh maintained largely by Kong and a CNCF Sandbox project, takes a similar approach with its kumactl tool. In this post, however, I’d like to take a look at creating reusable YAML to install Kuma, instead of using the CLI tool every time you install.


Using the External AWS Cloud Provider for Kubernetes

In 2018, after finding a dearth of information on setting up Kubernetes with AWS integration/support, I set out to try to establish some level of documentation on this topic. That effort resulted in a few different blog posts, but ultimately culminated in this post on setting up an AWS-integrated Kubernetes cluster using kubeadm. Although originally written for Kubernetes 1.15, the process described in that post is still accurate for newer versions of Kubernetes. With the release of Kubernetes 1.22, though, the in-tree AWS cloud provider—which is what is used/described in the post linked above—has been deprecated in favor of the external cloud provider. In this post, I’ll show how to set up an AWS-integrated Kubernetes cluster using the external AWS cloud provider.


Kustomize Transformer Configurations for Cluster API v1beta1

The topic of combining kustomize with Cluster API (CAPI) is a topic I’ve touched on several times over the last 18-24 months. I first touched on this topic in November 2019 with a post on using kustomize with CAPI manifests. A short while later, I discovered a way to change the configurations for the kustomize transformers to make it easier to use it with CAPI. That resulted in two posts on changing the kustomize transformers: one for v1alpha2 and one for v1alpha3 (since there were changes to the API between versions). In this post, I’ll revisit kustomize transformer configurations again, this time for CAPI v1beta1 (the API version corresponding to the CAPI 1.0 release).


Technology Short Take 146

Welcome to Technology Short Take #146! Over the last couple of weeks, I’ve gathered a few technology-related links for you all. There’s some networking stuff, a few security links, and even a hardware-related article. But enough with the introduction—let’s get into the content!


Installing Cilium via a ClusterResourceSet

In this post, I’m going to walk you through how to install Cilium onto a Cluster API-managed workload cluster using a ClusterResourceSet. It’s reasonable to consider this post a follow-up to my earlier post that walked you through using a ClusterResourceSet to install Calico. There’s no need to read the earlier post, though, as this post includes all the information (or links to the information) you need. Ready? Let’s jump in!


Technology Short Take 145

Welcome to Technology Short Take #145! What will you find in this Tech Short Take? Well, let’s see…stuff on Envoy, network automation, network designs, M1 chips (and potential open source variants!), a bevy of security articles (including a couple on very severe vulnerabilities), Kubernetes, AWS IAM, and so much more! I hope that you find something useful here. Enjoy!


Technology Short Take 144

Welcome to Technology Short Take #144! I have a fairly diverse set of links for readers this time around, covering topics from microchips to improving your writing, with stops along the way in topics like Kubernetes, virtualization, Linux, and the popular JSON-parsing tool jq. I hope you find something useful!


Establishing VPC Peering with Pulumi and Go

I use Pulumi to manage my lab infrastructure on AWS (I shared some of the details in this April 2020 blog post published on the Pulumi site). Originally I started with TypeScript, but later switched to Go. Recently I had a need to add some VPC peering relationships to my lab configuration. I was concerned that this may pose some problems—due entirely to the way I structure my Pulumi projects and stacks—but as it turned out it was more straightforward than I expected. In this post, I’ll share some example code and explain what I learned in the process of writing it.


Using the AWS CLI to Tag Groups of AWS Resources

To conduct some testing, I recently needed to spin up a group of Kubernetes clusters on AWS. Generally speaking, my “weapon of choice” for something like this is Cluster API (CAPI) with the AWS provider. Normally this would be enormously simple. In this particular case—for reasons that I won’t bother going into here—I needed to spin up all these clusters in a single VPC. This presents a problem for the Cluster API Provider for AWS (CAPA), as it currently doesn’t add some required tags to existing AWS infrastructure (see this issue). The fix is to add the tags manually, so in this post I’ll share how I used the AWS CLI to add the necessary tags.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!