Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Technology Short Take 175

Welcome to Technology Short Take #175! Here’s your weekend reading—a collection of links and articles from around the internet on a variety of data center- and cloud-related topics. I hope you find something useful here!



  • I attended a local meetup here in the Denver metro area a short while ago and was introduced to sops.
  • AMD processors have been discovered to have multiple security flaws; more details available here.
  • The Linux kernel project has become a CVE Numbering Authority (CNA); Greg Kroah-Hartman wrote a blog post that discusses this in more depth.

Cloud Computing/Cloud Management

  • Josh Biggley shows how to deploy Tetragon with Cribl Edge. The blog post is a bit heavy on the Cribl marketing, but I suppose that is to be expected (it’s extremely common with most vendor blogs).
  • Jack Lindamood’s list of infrastructure decisions he endorses or regrets provides some valuable insight into his personal experience with a variety of technologies and processes. Well worth reading, in my opinion. (Hat tip to Simon Wardley for sharing this on Twitter.)
  • Ivan Yurochko of PerfectScale discusses how to manage S3 throttling.
  • This post is an interesting look “inside” the CNCF Technical Oversight Committee (TOC), with a view on some of the challenges facing the CNCF and its related projects.
  • Tyler Treat argues that it’s possible—preferable, perhaps—to do cloud without Kubernetes.
  • Rory McCune reviews his final Kubernetes census.
  • The Open Constructs Foundation recently launched a “community-driven CDK construct library initiative,” which seeks to provide a way for the CDK community to build and share CDK constructs.
  • Michael Levan insists that cloud-native is in shambles. I think the article title is a bit click-baity, but the key point in the article—focusing on the expected outcome—is spot on.
  • Tony Norlin discusses running Kubernetes with Cilium on FreeBSD.
  • This is an older post (but still useful, I think, given the review of the code that implements the functionality) on Kubernetes leader election.

Operating Systems/Applications


  • Although it gets a bit deep into Rego, this article by Jasper Van der Jeugt of Snyk explains how automatic source code location for violations—pinpointing the file, line, and column where policy violations occur.
  • Josh Collinsworth weighs in regarding LLMs and generative AI in his essay regarding GitHub Copilot. The experiences Josh describes with Copilot are not unique to Copilot; I’ve experienced the same with other LLM-based generative AI tools. The key takeaway (for me) is that generative AI doesn’t make things more accessible; it’s actually the opposite, because you need to know enough to know whether or not the generative AI tool is actually accurate or not.


  • While certainly not unique to virtualization, I think it’s fair to say that virtualization has had a pretty significant impact on home labs. Sean Massey takes a moment to provide an update on his latest home lab update.

That’s all I have for you this time around. I love to hear from readers, so if you have feedback on this post (or any post!) on my site, please feel free to reach out. You can find me on Twitter, on the Fediverse, or in a number of different Slack communities. My e-mail address is also on this site and isn’t too hard to find…feel free to drop me a line!

Technology Short Take 174

Welcome to Technology Short Take #174! For your reading pleasure, I’ve collected links on topics ranging from Kubernetes Gateway API to recent AWS attack techniques to some geeky Linux and Git topics. There’s something here for most everyone, I’d say! But enough of my rambling, let’s get on to the good stuff. Enjoy!


  • I want to be Ivan Pepelnjak when I grow up. Why? Read this article on his response to someone wanting to use NSX to create availability zones.
  • Nico Vibert has a tutorial that takes readers through using Cilium’s Gateway API functionality to do L7 traffic management (HTTP redirects, HTTP rewrites, and HTTP mirroring).


Cloud Computing/Cloud Management

Operating Systems/Applications


  • This blog post announces the release of Pkl (pronounced “pickle”), described as a new “programming language for producing configuration”. Unless I’m reading this wrong, this sounds like quite an overlap with CUE. Or am I way wrong here?
  • Milas Bowman examines some new features in Docker Compose specifically targeted at improving the development experience.

Career/Soft Skills

That’s all I have for you this time around, but check back in 2-3 weeks for the next Technology Short Take. Until then, feel free to share this article on your favorite social media platform, and I invite you to contact me if you have any feedback about this or any article on my site. You can find me on the Fediverse, on Twitter, or in a number of different Slack communities. Heck, if you try hard enough you can find my e-mail address on this site and drop me a message that way!

Using NAT Instances on AWS with Pulumi

For folks using AWS in their day-to-day jobs, it comes as no secret that AWS’ Managed NAT Gateway—responsible for providing outbound Internet connectivity to otherwise private subnets—is an expensive proposition. While the primary concern for large organizations is the data processing fee, the concern for smaller organizations or folks like me who run a cloud-based lab instead of a hardware-based home lab is the per-hour cost. In this post, I’ll show you how to use Pulumi to use a NAT instance for outbound Internet connectivity instead of a Managed NAT Gateway.

For a bit more about why Managed NAT Gateways aren’t ideal for larger organizations, I’d recommend this article by Corey Quinn. For smaller organizations or cloud-based labs, data processing fees probably aren’t the main concern (although I could be wrong); it would be the ~$32/mo per Managed NAT Gateway. Since many tools configure a Managed NAT Gateway per availability zone, now you’re talking more like $96/mo—and you haven’t even spun up any real workloads yet! Running your own NAT instance can dramatically reduce but not eliminate this expense.

Now that I’ve established why running a NAT instance can be beneficial, let’s review what you’ll need to have installed in order to follow along with (or use) what I’ll show you in this post:

  1. I’m automating the entire process with Pulumi, so you’ll want to have the Pulumi CLI installed. (Installation instructions are here.)
  2. I write my Pulumi using Go, so you’d need Go installed. (Installation instructions are here.)
  3. A typical EC2 AMI isn’t pre-configured for NAT, so you’ll need either a configuration mechanism for setting that up (like Ansible and an associated playbook) or a preconfigured AMI. I chose to go the latter route and am using the excellent fck-nat AMI (check out the website and the associated GitHub repository).

I’ll walk through select pieces of the code below to explain what’s being provisioned or configured. For your reference, the full code is found in my GitHub “learning-tools” repository, in the aws/nat-instance-pulumi folder.

Setting up the VPC and Subnets

All the Pulumi code for setting up the VPC and subnets is separated into a file named vpc.go, and is invoked from main.go through a function named buildInfrastructure. At a high-level, the buildInfrastructure function does the following things:

  • It gets the number of availability zones (AZs) and the names of the zones, and stores that information for later use.
  • It builds a VPC with a preconfigured CIDR block. (In most of my Pulumi programs I make this a configuration value, but in this particular case it’s hard-coded. There’s no reason for that other than my own lack of time.)
  • It creates a public subnet in each of the AZs.
  • It handles the routing configuration for the public subnets (creates an Internet Gateway, creates a route table, creates an outbound route via the gateway, and links the public subnets to the route table).
  • It creates a private subnet in each of the AZs.
  • It creates a route table for the private subnets and links the private subnets to the route table, but does not create a route.

All said, that’s about 150 lines of code. You might wonder why I didn’t use Pulumi’s AWSX (Crosswalk for AWS) component for a VPC, which allows users to do almost the same thing in about 10 lines of code. That would be an excellent question! Currently, the AWSX VPC component doesn’t currently expose the route table IDs, which are needed so that I can add a route of my own creation. The AWSX VPC component is outstanding otherwise; if you can use it for your use case, I generally recommend it.

Setting up the NAT Infrastructure

Now the program moves on to creating the necessary NAT infrastructure. This code is split into a separate file named nat.go and invoked from main.go via the buildNat function.

This code is reasonably straightforward:

  • It creates a security group to allow traffic to move through the NAT instance.
  • It dynamically looks up the AMI ID for the fck-nat instance.
  • It launches an EC2 instance (a “tg4.nano” is sufficient to handle Gbps-level traffic) using the fck-nat AMI.
  • Once the EC2 instance is launched, it adds a route to the private subnet route table that directs outbound traffic for the private subnets through the EC2 instance. (We couldn’t do that earlier because we needed the interface ID associated with the EC2 instance.)

Finishing the Final Touches

For your own architecture implementation, you could stop there, but my code continues on so that there’s a way to test that the NAT instance is working as expected. All of this code is found in main.go.

Before main.go invokes the buildInfrastructure and buildNat functions, it first creates an SSH key and an associated AWS key pair. It passes the key pair name to the buildNat function so that the fck-nat instance is configured with the SSH key. This allows you to SSH into the NAT instance with the user “ec2-user” and the associated private key (which you can get from Pulumi using pulumi stack output).

After invoking buildInfrastructure and buildNat, the Pulumi program goes on to create an EC2 instance (based on a dynamically-obtained AMI ID) in one of the private subnets and a security group to allow SSH traffic to that instance. This allows you to test that the fck-nat instance is both a) working properly as an SSH bastion host, and b) working properly as a NAT instance.

Congratulations! You have now reduced your NAT costs to about 1/10th the cost of a Managed NAT Gateway.


This code isn’t necessarily intended for commercial production use, as there are a number of caveats with the architecture it creates:

  • There is only a single NAT instance for all AZs. If that AZ fails, then outbound traffic from all private subnets in other AZs is also down.
  • There is only a single NAT instance. If the NAT instance fails, then…well, you get the idea.

The fck-nat AMI has some functionality to help address some of these caveats, so I encourage you to review the website for more information. I’ll leave updating this code to support these features as an exercise for the reader. (Feel free to submit one or more PRs if you are so inclined.)

Additional Resources

To get access to the full Pulumi program, see my GitHub “learning-tools” repository in the aws/nat-instance-pulumi folder. If you have questions about the code or about Pulumi, feel free to join the Pulumi Community Slack, where I and other Pulumi enthusiasts and experts hang out. You’re also welcome to find me online; I am available on Twitter, on the Fediverse, and in various other Slack communities. I’d be more than happy to hear from readers with questions or feedback on this or any article on my site. Thanks for reading!

Using SSH with the Pulumi Docker Provider

In August 2023, Pulumi released a version of the Docker provider that supported SSH-based connections to a Docker daemon. I’ve written about using SSH with Docker before (see here), and I sometimes use AWS-based “Docker build hosts” with my M-series Macs to make it easier/simpler (and sometimes faster) to build x86_64-based Docker images. Naturally, I’m using an SSH connection in those cases. Until this past weekend, however, I hadn’t really made the time to look deeper into how to use SSH with the Pulumi Docker provider. In this post, I’ll share some details that (unfortunately) haven’t yet made it into the documentation about using SSH with the Pulumi Docker provider.

First, let’s talk about some prerequisites to making this work.

  1. You’ll need Docker installed locally. I fairly certain this is only the docker CLI (much in the same way the Pulumi Kubernetes provider requires kubectl to be installed locally), but I haven’t verified this for certain yet. I tested this from a Linux system running Docker 24.0.7; I think the earliest version that is supported is 18.09.
  2. You’ll need Docker installed on the remote SSH host (obviously). I used Flatcar Container Linux (stable channel) on AWS.
  3. You’ll need Pulumi installed locally. I tested with a pretty recent version of the pulumi CLI (v3.101.1).
  4. I tested this with the latest version of the Docker provider as of this writing (v4.5.1), using Go 1.21 as the programming language.

You may already be aware that there are a couple of ways to use Pulumi providers when writing Pulumi infrastructure as code programs:

  • There’s the default provider. The default provider uses what I would call “ambient” configuration—for example, the default AWS provider uses whatever AWS credentials/profile are available (or are specified in the stack configuration), and the default Docker provider uses whatever is specified by the DOCKER_HOST environment variable.
  • There’s also explicit providers. Explicit providers are declared programmatically in your Pulumi program, and you can pass configuration details to the provider when it’s declared. You could, for example, declare a couple of explicit AWS providers so that you could provision resources in different accounts or in different regions (from within the same program).

More details on providers can be found here.

With regard to the Pulumi Docker provider, this means the following:

  • If you want to use the Docker provider against a Docker daemon that is preexisting, then you can use the default provider and supply configuration either through the DOCKER_HOST environment variable or via stack configuration (pulumi config set docker:host <ssh-url>). (Note that, as of the time of this writing, the Docker provider does not support Docker contexts.)
  • If you want to use the Docker provider with a resource being provisioned in the same stack or if you—for whatever reason—need to programmatically assign the Docker daemon endpoint in your program, then you need to use an explicit provider, and configure that explicit provider to use SSH.

Using and configuring the default provider is reasonably straightforward, so in this article I’ll focus on the explicit provider; specifically, on the use of an explicit provider to make SSH-based connections to a Docker daemon.

Declaring a basic explicit provider is not terribly complex:

remoteDocker, err := docker.NewProvider(ctx, "remote-docker", &docker.ProviderArgs{})

To make the explicit provider actually work in this use case (i.e., connect over SSH to a remote Docker daemon), the configuration is a bit more complex:

remoteDocker, err := docker.NewProvider(ctx, "remote-docker", &docker.ProviderArgs{
    Host: pulumi.Sprintf("ssh://<username>@%s", <ip-address>),
    SshOpts: pulumi.StringArray{
        pulumi.String("-i"), pulumi.String("/path/to/private/key"),
        pulumi.String("-o"), pulumi.String("StrictHostKeyChecking=no"),
        pulumi.String("-o"), pulumi.String("UserKnownHostsFile=/dev/null"),

You’d need to substitute appropriate values for username (on Flatcar you’d likely use “core”), ip-address, and /path/to/private/key. Since I’m discussing using the Docker provider with a resource provisioned in the same stack, ip-address is most likely going to be a reference to the public IP address of an EC2 instance—such as flatcarInstance.publicIp. That’s also why the code above uses pulumi.Sprintf, which is capable of dealing with Outputs in Pulumi code.

The syntax of the SshOpts section isn’t currently defined in the docs; fortunately, I found a clue here that led to the Go code you see above. Given that this is using a resource that was provisioned in the same stack, the only way to make it work is to disable strict host key checking.

There’s one final complication. EC2 instances—or their equivalents on Azure or Google Cloud—take a small amount of time to boot up and become ready. The Docker provider needs to check the connection, and if it attempts that before the remote host is ready it will throw an error.

“No problem!” you say. “Just throw a sleep in there.”

Well…Pulumi doesn’t necessarily execute your Go code in the way you might normally expect, so this won’t work. What we need to do is create a resource that the Pulumi engine can add to the dependency graph that will insert a delay before creating the Docker provider. Fortunately, there is a Time provider that provides a Sleep resource to accomplish exactly what we need. To create the necessary dependencies and insert the delay in the right place, we make the Sleep resource dependent on the EC2 instance and the Docker provider dependent on the Sleep resource.

Including the EC2 instance, the Sleep resource, and the Docker provider, the code now looks like this (I’ve omitted error checking code for simplicity):

// Launch an instance using Flatcar Linux AMI
flatcarInstance, err := ec2.NewInstance(ctx, "flatcar-instance", &ec2.InstanceArgs{
    Ami:                      pulumi.String(flatcarAmi.Id),
    InstanceType:             pulumi.String(instanceType),
    AssociatePublicIpAddress: pulumi.Bool(true),
    KeyName:                  pulumi.StringPtr(userSuppliedKeyPair),
    SubnetId:                 dockerVpc.PublicSubnetIds.Index(pulumi.Int(0)),
    VpcSecurityGroupIds:      pulumi.StringArray{dockerSg.ID()},
    Tags: pulumi.StringMap{
        "Name": pulumi.String("flatcar-instance"),

// Sleep for 20 seconds to allow instance to boot
instanceBootDelay, err := time.NewSleep(ctx, "instance-boot-delay", &time.SleepArgs{
    CreateDuration: pulumi.String("20s"),
}, pulumi.DependsOn([]pulumi.Resource{flatcarInstance}))

// Create a new Docker provider
remoteDocker, err := docker.NewProvider(ctx, "remote-docker", &docker.ProviderArgs{
    Host: pulumi.Sprintf("ssh://core@%s", flatcarInstance.PublicIp),
    SshOpts: pulumi.StringArray{
        pulumi.String("-i"), pulumi.String(userSuppliedPrivateKeyFile),
        pulumi.String("-o"), pulumi.String("StrictHostKeyChecking=no"),
        pulumi.String("-o"), pulumi.String("UserKnownHostsFile=/dev/null"),
}, pulumi.DependsOn([]pulumi.Resource{instanceBootDelay}))

Following this code, you could then have the remote Docker daemon pull an image and deploy a container from that image:

// Pull down a container image on the remote host
nginxImage, err := docker.NewRemoteImage(ctx, "nginx-image", &docker.RemoteImageArgs{
    Name: pulumi.String("nginx:1.17.4-alpine"),
}, pulumi.Provider(remoteDocker))

// Launch a container on the remote host
_, err = docker.NewContainer(ctx, "nginx-container", &docker.ContainerArgs{
    Image: nginxImage.ImageId,
}, pulumi.Provider(remoteDocker))

Neat, right? (And useful!) In one Pulumi stack, you can provision the EC2 instance, create a Docker provider to communicate with that instance, and deploy containers on that instance—all in the programming language of your choice.

If you’d like to see the full Pulumi program, you can find it in the docker/docker-ssh-pulumi folder of my GitHub “learning-tools” repository. The code is useful in that it illustrates the correct syntax for a number of useful constructs when using Pulumi with Go:

  • Creating a dependency between resources
  • Referencing an explicit provider for a resource
  • Configuring the SSH options for the Docker provider
  • Looking up the AMI for an instance (not shown above, but it is in the full code on GitHub)

I hope this proves useful to someone. If you have questions, you are welcome to open an issue on my “learning-tools” GitHub repository, or you can reach out to me directly. You can contact me on Twitter, on the Fediverse, or via Slack (I’m active in a number of different Slack communities). I’d love to hear from you, hear any feedback you might have, or try to answer questions about this article or any of my articles. Thanks for reading!

Technology Short Take 173

Welcome to Technology Short Take #173! After a lull in links to share last time around, it looks like things have rebounded and folks are in full swing writing new content for me to share with you. I think I have a decent round-up of links for you; hopefully you can find something useful here. Enjoy!




Cloud Computing/Cloud Management

  • Dean has published information on migrating your Red Hat OpenShift clusters to Cilium (from one of the “default” networking solutions).
  • I think I’ve linked to Ricardo Sueiras’ “AWS open source newsletter” before; it’s such a useful resource. In edition 184, Ricardo shares some links to some useful posts on EKS; the one on using Istio with EKS to improve the user experience caught my eye. (Check out the newsletter to get the link to the Istio article.)
  • Ian McKay digs into the details of the recently-announced support for HTTPS Endpoints in AWS Step Functions.
  • Matt Gowie of MasterPoint explains terraform-null-label and its use in providing more consistent naming and tagging of cloud resources.

Operating Systems/Applications


  • Jamie Tanna explains how to represent a JSON field in Go that could be absent, null, or have a value.


Career/Soft Skills

It’s time to wrap up now; as always, I’d love to hear from readers about what you find useful (or not useful!) about the Technology Short Takes—or any of the posts on my site. Feel free to reach out to me via social media; you can find me on Twitter as well as on the Fediverse. I also tend to frequent a few different Slack communities, so you’re welcome to DM me there. Finally, if you’d like to drop me an e-mail, my address isn’t too hard to find. Thanks for reading!

Recent Posts

Technology Short Take 172

Welcome to Technology Short Take #172, the first Technology Short Take of 2024! This one is really short, which I’m assuming reflects a lack of blogging activity over the 2023 holiday season. Nevertheless, I have managed to scrape together a few links to share with readers. As usual, I hope you find something useful. Enjoy!


Selectively Replacing Resources with Pulumi

Because Pulumi operates declaratively, you can write a Pulumi program that you can safely run (via pulumi up) multiple times. If no changes are needed—meaning that the current state of the infrastructure matches what you’ve defined in your Pulumi program—then nothing happens. If only one resource needs to be updated, then it will update only that one resource (and any dependencies, if there are any). There may be times, however, when you want to force the replacement of specific resources. In this post, I’ll show you how to target specific resources for replacement when using Pulumi.


Dynamically Enabling the Azure CLI with Direnv

I’m a big fan of direnv, the tool that lets you load and unload environment variables depending on the current directory. It’s so very useful! Not too terribly long ago, I wanted to find a way to “dynamically activate” the Azure CLI using direnv. Basically, I wanted to be able to have the Azure CLI disabled (no configuration information) unless I was in a directory where I needed or wanted it to be active, and be able to make it active using direnv. I finally found a way to make it work, and in this blog post I’ll share how you can do this, too.


Conditional Git Configuration

Building on the earlier article on automatically transforming Git URLs, I’m back with another article on a (potentially powerful) feature of Git—the ability to conditionally include Git configuration files. This means you can configure Git to be configured (and behave) differently based on certain conditions, simply by including or not including Git configuration files. Let’s look at a pretty straightforward example taken from my own workflow.


Automatically Transforming Git URLs

Git is one of those tools that lots of people use, but few people truly master. I’m still on my own journey of Git mastery, and still have so very far to go. However, I did take one small step forward recently with the discovery of the ability for Git to automatically rewrite remote URLs. In this post, I’ll show you how to configure Git to automatically transform the URLs of Git remotes.


Technology Short Take 171

Welcome to Technology Short Take #171! This is the next installation in my semi-regular series that shares links and articles from around the interwebs on various technology areas of interest. Let the linking begin!


Saying Goodbye to the Full Stack Journey

In January 2016, I published the first-ever episode of the Full Stack Journey podcast. In October 2023, the last-ever episode of the Full Stack Journey podcast was published. After almost seven years and 83 episodes, it was time to end my quirky, eclectic, and unusual podcast that explored career journeys alongside various technologies, products, and open source projects. In this post, I wanted to share a few thoughts about saying goodbye to the Full Stack Journey.


Guest Post: Moving Secrets Where They Belong

(This is a guest post by Simen A.W. Olsen.)

Pulumi recently shipped Pulumi ESC, which adds the “Environment” tab to Pulumi Cloud. For us at Bjerk, this means we can move secrets into a secrets manager like Google Secrets Manager. Let me show you how we did it!


Assigning Tags by Default on AWS with Pulumi

Appropriately tagging resources on AWS is an important part of effectively managing infrastructure resources for many organizations. As such, an infrastructure as code (IaC) solution for AWS must have the ability to ensure that resources are always created with the appropriate tags. (Note that this is subtly different from a policy mechanism that prevents resources from being created without the appropriate tags.) In this post, I’ll show you a couple of ways to assign tags by default when creating AWS resources with Pulumi. Code examples are provided in Golang.


Technology Short Take 170

Welcome to Technology Short Take #170! I had originally intended to get this published before the long Labor Day weekend, but didn’t quite have it ready. So, here you go—here’s your latest collection of links from around the internet focused on data center and cloud-related technologies. I hope that you find something useful here.


Mac, iPad, or Both?

Both Jason Snell and John Gruber, both stalwarts in the Apple journalism world, have recently weighed in on this topic. Jason says he’s given up on the iPad-only travel dream; John says he keeps throwing his iPad in his bag when he travels, even if he never uses it. I have thoughts on this topic—as you might think, considering I decided to write about it! (Ah, but what device did I use to write?)


Technology Short Take 169

Welcome to Technology Short Take #169! Prior to the recent Spousetivities post, it had been a few months since I posted on the site; life has been busy, and it hasn’t left much time for blogging. Hopefully things will settle down soon, but until then I’ll continue to do the best I can to share useful information with folks. Hopefully something I’ve included in this Technology Short Take proves to be useful to someone. OK, let’s get on to the content!


Spousetivities Returns to VMware Explore 2023

After a lengthy hiatus—prompted by a pandemic and the suspension of in-person events as a result—Spousetivities returns to VMware Explore! VMware Explore, the event formerly known as VMworld, is happening in Las Vegas, NV, and Spousetivities will be there offering organized activities for spouses, partners, significant others, family, or friends traveling with conference attendees. Registration is already open!


Technology Short Take 168

Welcome to Technology Short Take #168! Although this weekend is (in the US, at least) celebrated as Mother’s Day weekend—don’t forget to call or visit your mom!—I thought you all might want some light weekend reading. I’m here to help, after all. To that end, here’s the latest Technology Short Take, with links to a variety of articles in various disciplines. Enjoy!


Technology Short Take 167

Welcome to Technology Short Take #167! This Technology Short Take is a tad shorter than the typical one; I’ve been busy recently and my intake volume of content has gone down, thus resulting in fewer links to share with all of you! I opted to go ahead and publish a shorter Technology Short Take instead of making everyone wait around for a longer one. In any case, here’s hoping that I’ve included something useful for you!


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!