Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Creating an AWS VPC Endpoint with Pulumi

In this post, I’d like to show readers how to use Pulumi to create a VPC endpoint on AWS. Until recently, I’d heard of VPC endpoints but hadn’t really taken the time to fully understand what they were or how they might be used. That changed when I was presented with a requirement for the AWS EC2 APIs to be available within a VPC that did not have Internet access. As it turns out—and as many readers are probably already aware—this is one of the key use cases for a VPC endpoint (see the VPC endpoint docs). The sample code I’ll share below shows how to programmatically create a VPC endpoint for use in infrastructure-as-code use cases.

For those that aren’t familiar, Pulumi allows users to use one of a number of different general-purpose programming languages and apply them to infrastructure-as-code scenarios. In this example, I’ll be using TypeScript, but Pulumi also supports JavaScript and Python (and Go is in the works). (Side note: I intend to start working with the Go support in Pulumi when it becomes generally available as a means of helping accelerate my own Go learning.)

Here’s a snippet of TypeScript code that will create a VPC endpoint (I’ve randomized the resource IDs of the AWS resources below):

const ec2ApiEndpoint = new aws.ec2.VpcEndpoint("test-vpc-endpoint", {
    privateDnsEnabled: true,
    serviceName: "",
    securityGroupIds: [ "sg-1db01e1f9be081d76" ],
    subnetIds: [ "subnet-0db757befe2c48624" ],
    vpcEndpointType: "Interface",
    vpcId: "vpc-0e2c348614e845763",
    tags: {
        Name: "test-vpc-endpoint"

Let’s walk through this a bit (the full API reference is here):

  • The privateDnsEnabled: true line enables us to use the “standard” EC2 API URL endpoint within the VPC and have it resolve to a private IP address out of the VPC/subnets. This relies on Route 53, and is necessary for use cases where you (the user) don’t have any control over how the API endpoint is being called from within the VPC (it may be a bit of commercial software you can’t modify). In this case, any software trying to reach “” will resolve to a private IP address instead of the public IP address.
  • The serviceName specifies which service is available via this endpoint (naturally, this is for EC2).
  • The vpcEndpointType specifies “Interface”, which means that an ENI is created to allow traffic to reach the service represented by this endpoint. The other option is “Gateway”, which would require you to modify route tables, and is only supported for some services (like S3). This article provides an example of a gateway endpoint.

The other settings are all pretty self-explanatory. This being TypeScript, you could—of course—use references to variables or outputs from other portions of your Pulumi code instead of hard-coding values as I have in this example (and that would be highly recommended). This could be easily incorporated into a large portion of code that created a new VPC, new subnets, etc.

And that’s it! I know this is a fairly simple example, but I hadn’t found very many articles showing how this is done, and thought that sharing this may prove helpful to others. Feel free to contact me on Twitter if you have corrections (in the event I made a mistake), suggestions for improvement, or other feedback.

Manually Loading Container Images with containerD

I recently had a need to manually load some container images into a Linux system running containerd (instead of Docker) as the container runtime. I say “manually load some images” because this system was isolated from the Internet, and so simply running a container and having containerd automatically pull the image from an image registry wasn’t going to work. The process for working around the lack of Internet access isn’t difficult, but didn’t seem to be documented anywhere that I could readily find using a general web search. I thought publishing it here may help individuals seeking this information in the future.

For an administrator/operations-minded user, the primary means of interacting with containerd is via the ctr command-line tool. This tool uses a command syntax very similar to Docker, so users familiar with Docker should be able to be productive with ctr pretty easily.

In my specific example, I had a bastion host with Internet access, and a couple of hosts behind the bastion that did not have Internet access. It was the hosts behind the bastion that needed the container images preloaded. So, I used the ctr tool to fetch and prepare the images on the bastion, then transferred the images to the isolated systems and loaded them. Here’s the process I followed:

  1. On the bastion host, first I downloaded (pulled) the image from a public registry using ctr image pull (the example I’ll use here is for the Calico node container image, used by the Calico CNI in Kubernetes clusters):

    ctr image pull

    (Note that sudo may be needed for all these ctr commands; that will depend on your system configuration.)

    I repeated this process for all the other images the isolated systems also needed.

  2. Still on the bastion host, I exported the pulled images to standalone archives:

    ctr image export calico-node-v3.11.2.tar \

    The general format for this command looks like this:

    ctr image export <output-filename> <image-name>

    If you don’t know what the image name (according to containerd) is, use ctr image ls.

  3. After transferring the standalone archives to the other systems (using whatever means you prefer; I used scp), then load (or import) the images into containerd with this command:

    ctr image import <filename-from-previous-step>

    Repeat as needed for additional images. It appears, by the way, that using wildcards in the ctr image import command won’t work; I had to manually specify each individual file for import.

    If you need these images to be available to Kubernetes, you must be sure to add the flag to the ctr image import command, like this:

    ctr images import <filename-from-previous-step>
  4. Verify that the image(s) are present and recognized by containerd using ctr image ls.

    If you specified the namespace when importing the images in the previous step—so as to make the images available to Kubernetes—then you can verify that CRI (Container Runtime Interface, the means by which Kubernetes talks to containerd) sees these images by running crictl images (again, sudo may be required, based on your configuration).

That should do it for you!

How I Tested

To test this procedure, I used multiple Ubuntu 18.04 instances on AWS. One instance had Internet access (the bastion host); the others did not. The instances had been built from AMIs prepared using Packer and Ansible with playbooks from the Kubernetes “image-builder” repository (specifically, the playbooks used to build the Kubernetes Cluster API images), so containerd and all the other CLI tools mentioned in this article were already installed and available for use.

Additional Resources

I did find a few sites with some containerd information while searching, and wanted to include them here for credit and additional context:

Getting started with Containerd

crictl User Guide

I hope the information in this post is helpful. If you have questions, comments, suggestions, or corrections, you are welcome to contact me on Twitter, or drop me an e-mail (my e-mail address isn’t too hard to find).

Thinking and Learning About API Design

In July of 2018 I talked about Polyglot, a very simple project I’d launched whose only purpose was simply to bolster my software development skills. Work on Polyglot has been sporadic at best, coming in fits and spurts, and thus far focused on building a model for the APIs that would be found in the project. Since I am not a software engineer by training (I have no formal training in software development), all of this is new to me, and I’ve found myself encountering lots of questions about API design along the way. In the interest of helping others who may be in a similar situation, I thought I’d share a bit here.

I initially approached the API in terms of how I would encode (serialize?) data on the wire using JSON (I’d decided on using a RESTful API with JSON over HTTP). Starting with how I anticipated storing the data in the back-end database, I created a representation of how a customer’s information would be encoded (serialized) in JSON:

    "customers": [
            "customerID": "5678",
            "streetAddress": "123 Main Street",
            "unitNumber": "Suite 123",
            "city": "Anywhere",
            "state": "CO",
            "postalCode": "80108",
            "telephone": "3035551212",
            "primaryContactFirstName": "Scott",
            "primaryContactLastName": "Lowe"

It’s pretty obvious, though, that this representation is closely tied to how I anticipate I’ll store customer data in the back-end database, right down to the field names. One of the books I’ve been reading is Building Microservices, an O’Reilly book by Sam Newman (see here; I’ll post a review of the book once I complete it). One of the things Newman cautions readers against in his book is exposing implementation details internal to a service to other services—did this qualify? Newman also discussed hypermedia as the engine of application state (HATEOAS, what an acronym!), but I’ll be honest in admitting that it didn’t click with me at first.

While still working through Newman’s book, I added REST API Design Rulebook by Mark Masse to the mix (see my review of the book). Masse talked extensively about HATEOAS, but—as you can see in my review of the book—it still didn’t click for me.

Next up, I added RESTful Web APIs by Leonard Richardson, Sam Ruby, and Mike Amundsen to my reading list (here’s the O’Reilly page for the book; I’m still reading the book so no review yet). Whether by repetition or by seeing the information presented in a different context, the discussion of HATEOAS and hypermedia by Richardson et al. finally made sense. It also unlocked some of the ideas presented by Masse and Newman earlier, so that I was able to really understand that I’d made several critical mistakes in my approach thus far:

  • I chose a media type (JSON) before fully understanding the particulars of my application without realizing the constraints that media type creates.
  • The client representation was too closely tied to the back-end/server-side implementation, making things far too brittle (what would happen if I needed to change the database schema?).
  • I invented new field names instead of re-using properties and fields that are already well-known and well-accepted (like re-using information from an appropriate microformat).
  • The client representation doesn’t provide any information on state transitions (How is a customer record updated? How is a customer record deleted?) or relationships between resources (What’s the relationship between an order record and a customer record? Or vice versa?).

It’s clear that something more than just encoding data in JSON is required.

The question, for me, is how much more? And what to include, exactly? Richardson et al. seem to advocate the Collection+JSON format, but it’s only a personal standard (see his book for definitions of various standards) and not necessarily widely accepted/used. However, even within Collection+JSON, the specific links provided as hypermedia are left as “implementation details.” So what sort of information makes sense?

Given that Polyglot is intended to be a simple order entry/lookup system, a few potential ideas come to mind:

  • All discussions of HATEOAS and hypermedia agree that a “self” link is needed that provides the URL to the resource (customer record or order record) being returned.
  • It seems reasonable to return “related” information. So, if viewing an order record, provide a link to the associated customer record (for example), the previous order, the next order, etc.

Clearly I have lots of work to do!

Now comes the task of shaping the limited work I’ve done so far into something more usable. I also need to finish reading RESTful Web APIs; although the content was difficult for me to grok at first, recently I’ve found that I’m beginning to more fully understand the concepts the authors are presenting. I still have lots of questions—how do things like OpenAPI fit in here?—so if you have answers, feel free to hit me on Twitter. This is definitely a learning exercise for me, so I welcome any other resources I should be using to help guide my way.

Technology Short Take 123

Welcome to Technology Short Take #123, the first of 2020! I hope that everyone had a wonderful holiday season, but now it’s time to jump back into the fray with a collection of technical articles from around the Internet. Here’s hoping that I found something useful for you!


  • Eric Sloof mentions the NSX-T load balancing encyclopedia (found here), which intends to be an authoritative resource to NSX-T load balancing configuration and management.
  • David Gee has an interesting set of articles exploring service function chaining in service mesh environments (part 1, part 2, part 3, and part 4).



  • On January 13, Brian Krebs discussed the critical flaw (a vulnerability in crypt32.dll, a core Windows cryptographic component) that was rumored to be fixed the next day (January 14) on the first “Patch Tuesday” of 2020. The next day, the Microsoft Security Response Center confirmed the vulnerability. Time to get patching, folks!
  • It’s good to see 1Password add support for U2F security keys for logging into your account via a web browser. Now I really want to see hardware security key support in the desktop and mobile apps!

Cloud Computing/Cloud Management

Operating Systems/Applications


Nothing this time around; I’ll try to find some content for next time!


Career/Soft Skills

  • Scott Driver talks about “the vCommunity” and some of the benefits that can be had, career-wise, from actively participating. Although Scott’s article focuses on the VMware-centric community, this is true (in my opinion) of many different communities, including various open source communities—and, to be frank, the physical communities in which we live and work.
  • Now that I’ve moved back onto macOS (see here for the explanation), I’m also taking OmniFocus back up, and so I found this article on using OmniFocus’ new Tags functionality helpful.
  • Thomas Maurer has some great tips for creating technical demos and presentations, although (I guess somewhat understandably given he works for Microsoft) the tips are a tad Windows-centric.

I guess that’s all for now! Thanks for reading, and again I hope that you found something useful here. If you have feedback, suggestions for improvement, or just want to say hi, feel free to hit me up on Twitter. Thanks!

Removing Unnecessary Complexity

Recently, I’ve been working to remove unnecessary complexity from my work environment. I wouldn’t say that I’m going full-on minimalist (not that there’s anything wrong with that), but I was beginning to feel like maintaining this complexity was taking focus, willpower, and mental capacity away from other, more valuable, efforts. Additionally, given the challenges I know lie ahead of me this year (see here for more information), I suspect I’ll need all the focus, willpower, and mental capacity I can get!

When I say “unnecessary complexity,” by the way, I’m referring to added complexity that doesn’t bring any real or significant benefit. Sometimes there’s no getting around the complexity, but when that complexity doesn’t create any value, it’s unnecessary in my definition.

Primarily, this “reduction in complexity” shows up in three areas:

  1. My computing environment
  2. My home office setup
  3. My lab resources

My Computing Environment

Readers who have followed me for more than a couple years know that I migrated away from macOS for about 9 months in 2017 (see here for a wrap-up of that effort), then again in 2018 when I joined Heptio (some details are available in this update). Since switching to Fedora on a Lenovo X1 Carbon when I joined Heptio, I was using Linux on the desktop full-time. (This post contains links to all the posts I wrote about my Linux migration.)

About a month ago, though, I switched back to macOS (macOS Mojave, specifically). Why? Although I was making Linux work for me, there was just enough “extra friction” in daily tasks to make it more of a chore than I felt it needed to be:

  • E-mail connectivity to Office 365 was a challenge (my employer shut down IMAP/SMTP access to e-mail).
  • Calendaring was difficult, at best, and the shared calendar for my team totally confused most solutions I tried.
  • Interoperability between LibreOffice and Office 365 remained subpar (at best) for all but the most basic documents, unfortunately.

Contrast that with macOS, where in a matter of minutes I had full connectivity to e-mail, full calendar access (including the shared team calendar), and a working version of Office 365. I also had multi-lingual support working with relative ease, something with which I suspect Linux desktop environments would’ve struggled.

Although I had workable solutions (use a Windows 10 desktop on VMware View for some things, use the Office web applications for other aspects), these solutions still introduced what I felt was unnecessary complexity. So, to remove this unnecessary complexity, I switched back to macOS full-time.

I also switched back to iOS (I bought an iPhone 11 Pro) from Android (I’d been using a Google Pixel). Given that I was moving back into the Apple ecosystem, and given that using an Android-powered phone had made it a bit more difficult to stay in touch with my wife when I was traveling, it didn’t seem to make sense to continue down that path. There was no significant benefit in exchange for the added friction.

My Home Office Setup

In addition to simplifying my computing environment, I also set out to simplify my home office setup. For years, I’d used both a 2012-era Mac Pro (one of the classic aluminum tower systems, which I’d purchased to help with video rendering) and a laptop, and used tools like Synergy (to have a single shared keyboard and mouse across both systems) and Unison (to keep files synchronized across both systems). In reality, though, I didn’t need the Mac Pro as a desktop system; I really needed it more like a server system to which I could offload rendering tasks.

Around the same time I switched back to macOS, I also switched back to a single-computer setup. I no longer use Synergy to share a keyboard and mouse across multiple systems; instead, I have only my 2017 13” MacBook Pro. It uses a USB-C port replicator to connect to an external monitor (the same 34” 21:9 ultra-wide monitor I’d been using previously), external keyboard, external microphone and webcam, and USB-based headset for Zoom meetings. I use a small application named ControlPlane to automatically adjust settings based on whether the laptop is “docked” or not. Plug the laptop in, and it reconfigures itself; unplug it, and it automatically adapts.

The Mac Pro sits off to the side; I’m still working out details on exactly how I’ll utilize it. It’s still running Linux (Fedora, specifically), and will most likely continue to do so; since it isn’t being used as a desktop system I see little value in moving back to macOS on that hardware. It’ll probably take over tasks from an aging Mac mini as well as helping with video rendering tasks, once I figure out the specific details on the best way to make that work.

My Lab Resources

Making mention of the aging Mac mini naturally leads me into the last area, my lab resources. This effort actually started a few years ago when I moved into my current home. Prior to that, I ran a full-fledged home lab: eight dual-socket Dell servers as hypervisors (a mix of KVM and ESXi), 10Gb and 40Gb Ethernet with multiple VLANs, multiple VMs running for a variety of purposes. When I moved into the current house, though, I did away with all of it except the Mac mini and an Apple Time Capsule. Moving “up the stack” to focus on things like Linux containers and Kubernetes meant there wasn’t as much value in running all this hardware, and why spend time managing all that complexity if I didn’t really need to?

Instead, I started leveraging ephemeral instances in AWS for everything I needed when it came to “lab resources.” Over the last few years, I’ve used that approach to strengthen and expand my skills around infrastructure-as-code. My current approach leverages Pulumi to manage a base set of AWS resources in multiple regions. Additional capacity is spun up on-demand, and shut down when it’s no longer actively needed. Since it’s almost exclusively Kubernetes I’m using, I leverage kubeadm and Cluster API to create and destroy Kubernetes clusters as needed. Yes, there are some limitations, but for the most part this arrangement has worked reasonably well.

Anyone else undertaken a similar journey? I’d be interested to hear from you, and perhaps learn of some additional ways I may be able to simplify and eliminate unnecessary complexity. Feel free to reach out to me on Twitter, or drop me an e-mail (my e-mail address isn’t hard to find, to be honest). Thanks!

Recent Posts

Looking Back: 2019 Project Report Card

As has been my custom over the last five years or so, in the early part of the year I like to share with my readers a list of personal projects for the upcoming year (here’s the 2019 list). Then, near the end of that same year or very early in the following year, I evaluate how I performed against that list of personal projects (for example, here’s my project report card for 2018). In this post, I’ll continue that pattern with an evaluation of my progress against my 2019 project list.


New Year, New Adventure

I’ll skip the build-up and jump straight to the whole point of this post: a once-in-a-lifetime opportunity has come up and I’m embarking on a new adventure starting in early 2020. No, I’m not changing jobs…but I am changing time zones.


Technology Short Take 122

Welcome to Technology Short Take #122! Luckily I did manage to get another Tech Short Take squeezed in for 2019, just so all my readers could have some reading materials for the holidays. I’m kidding! No, I mean I really am kidding—don’t read stuff over the holidays. Spend time with your family instead. The investment in your family will pay off in later years, trust me.


Technology Short Take 121

Welcome to Technology Short Take #121! This may possibly be the last Tech Short Take of 2019 (not sure if I’ll be able to squeeze in another one), so here’s hoping that you find something useful, helpful, or informative in the links that I’ve collected. Enjoy some light reading over your festive holiday season!


Technology Short Take 120

Welcome to Technology Short Take #120! Wow…hard to believe it’s been almost two months since the last Tech Short Take. Sorry about that! Hopefully something I share here in this Tech Short Take is useful or helpful to readers. On to the content!


KubeCon 2019 Day 3 and Event Summary

Keynotes Bryan Liles kicked off the day 3 morning keynotes with a discussion of “finding Kubernetes’ Rails moment”—basically focusing on how Kubernetes enables folks to work on/solve higher-level problems. Key phrase from Bryan’s discussion (which, as usual, incorporated the humor I love to see from Bryan): “Kubernetes isn’t the destination. Kubernetes is the vehicle that takes us to the destination.” Ian Coldwater delivered a talk on looking at Kubernetes from the attacker’s point of view, and using that perspective to secure and harden Kubernetes.Read more...

KubeCon 2019 Day 2 Summary

Keynotes This morning’s keynotes were, in my opinion, better than yesterday’s morning keynotes. (I missed the closing keynotes yesterday due to customer meetings and calls.) Only a couple of keynotes really stuck out. Vicki Cheung provided some useful suggestions for tools that are helping to “close the gap” on user experience, and there was an interesting (but a bit overly long) session with a live demo on running a 5G mobile core on Kubernetes.Read more...

KubeCon 2019 Day 1 Summary

This week I’m in San Diego for KubeCon + CloudNativeCon. Instead of liveblogging each session individually, I thought I might instead attempt a “daily summary” post that captures highlights from all the sessions each day. Here’s my recap of day 1 at KubeCon + CloudNativeCon.


Using Kustomize with Cluster API Manifests

A topic that’s been in the back of my mind since writing the Cluster API introduction post is how someone could use kustomize to modify the Cluster API manifests. Fortunately, this is reasonably straightforward. It doesn’t require any “hacks” like those needed to use kustomize with kubeadm configuration files, but similar to modifying kubeadm configuration files you’ll generally need to use the patching functionality of kustomize when working with Cluster API manifests. In this post, I’d like to take a fairly detailed look at how someone might go about using kustomize with Cluster API.


Programmatically Creating Kubernetes Manifests

A while ago I came across a utility named jk, which purported to be able to create structured text files—in JSON, YAML, or HCL—using JavaScript (or TypeScript that has been transpiled into JavaScript). One of the use cases was creating Kubernetes manifests. The GitHub repository for jk describes it as “a data templating tool”, and that’s accurate for simple use cases. In more complex use cases, the use of a general-purpose programming language like JavaScript in jk reveals that the tool has the potential to be much more than just a data templating tool—if you have the JavaScript expertise to unlock that potential.


Spousetivities in Barcelona at VMworld EMEA 2019

Barcelona is probably my favorite city in Europe—which works out well, since VMware seems to have settled on Barcelona at the destination for VMworld EMEA. VMworld is back in Barcelona again this year, and I’m fortunate enough to be able to attend. VMworld in Barcelona wouldn’t be the same without Spousetivities, though, and I’m happy to report that Spousetivities will be in Barcelona. In fact, registration is already open!


Using Kustomize with Kubeadm Configuration Files

Last week I had a crazy idea: if kustomize can be used to modify YAML files like Kubernetes manifests, then could one use kustomize to modify a kubeadm configuration file, which is also a YAML manifest? So I asked about it in one of the Kubernetes-related channels in Slack at work, and as it turns out it’s not such a crazy idea after all! So, in this post, I’ll show you how to use kustomize to modify kubeadm configuration files.


Technology Short Take 119

Welcome to Technology Short Take #119! As usual, I’ve collected some articles and links from around the Internet pertaining to various data center- and cloud-related topics. This installation in the Tech Short Takes series is much shorter than usual, but hopefully I’ve managed to find something that proves to be helpful or informative! Now, on to the content!


Exploring Cluster API v1alpha2 Manifests

The Kubernetes community recently released v1alpha2 of Cluster API (a monumental effort, congrats to everyone involved!), and with it comes a number of fairly significant changes. Aside from the new Quick Start, there isn’t (yet) a great deal of documentation on Cluster API (hereafter just called CAPI) v1alpha2, so in this post I’d like to explore the structure of the CAPI v1alpha2 YAML manifests, along with links back to the files that define the fields for the manifests. I’ll focus on the CAPI provider for AWS (affectionately known as CAPA).


An Introduction to Kustomize

kustomize is a tool designed to let users “customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is” (wording taken directly from the kustomize GitHub repository). Users can run kustomize directly, or—starting with Kubernetes 1.14—use kubectl -k to access the functionality (although the standalone binary is newer than the functionality built into kubectl as of the Kubernetes 1.15 release). In this post, I’d like to provide an introduction to kustomize.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!