Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Technology Short Take 112

Welcome to Technology Short Take #112! It’s been quite a while since the last one, as life and work have been keeping me busy. I have, however, finally managed to pull together this list of links and articles from around the Internet, and I hope that something I’ve included here proves useful to readers.



Nothing this time around! I’ll stay alert for content I can include next time.


  • Tim Hinrichs discusses securing the Kubernetes API with Open Policy Agent.
  • Pod Security Policies (PSPs) are an important security feature in Kubernetes. Sysdig explains PSPs, and talks about kube-psp-advisor, a tool to help simplify deploying PSPs.
  • ClusterScope is a handy tool for finding outdated images in your Kubernetes cluster.
  • This article discusses four open source secrets management tools.
  • Many organizations prefer to use two-factor authentication (2FA) to help protect their systems. While this article on how to configure 2FA for SSH on Fedora probably won’t work in many corporate environments (few use Fedora), it may provide enough information to figure out what it would look like in your environment.

Cloud Computing/Cloud Management

  • Bahubali (Bill) Shetti walks through analyzing the cost of a self-managed Kubernetes cluster on AWS using VMware CloudHealth.
  • Ahmet Alp Balkan does a deep dive on the KUBECONFIG file.
  • Lee Briggs writes about his experience with Fargate. I think the key takeaway here is that prior experience always affects our perceptions and how we go about learning new technologies/acquiring new skills. My prior experience with hypervisors (vSphere, then KVM) affected how I learned Docker and containers. Lee’s prior experience with Kubernetes affected how he learned Fargate. Someone who’d worked quite a bit with Fargate would probably have a hard time switching to Kubernetes. An individual’s learning curve is strongly dictated by previous experience and knowledge.
  • Ernese Norelus has an introductory piece on using Terraform and Ansible to enable repeatable infrastructure builds on AWS.
  • Fernand Galiana introduces Popeye, a tool for finding and identifying misconfigurations in your Kubernetes cluster. I haven’t had the chance to give it a try yet, but it looks pretty interesting.
  • Aeva talks a bit about what happened to OpenStack. Key excerpt (for me) from this article was this statement: “…creating a viable, open source, hyperscale cloud software solution was against the best interest of the companies most heavily investing in OpenStack’s development.”

Operating Systems/Applications


Nothing this time. Have something you think I should share here? Let me know on Twitter.


Career/Soft Skills

  • This blog post from XMind has some nice tips on staying focused in the workspace.
  • I really enjoyed this discussion on deep work and real-time collaboration. Cal’s book is in the “To Read” pile on my desk; guess I need to hurry up and get to it!

That’s all for now—stay tuned for future Tech Short Takes, as I’m striving to be more regular with publishing them. In the meantime, feel free to contact me on Twitter with any comments, suggestions, corrections, or other feedback.

Using Kubeadm to Add New Control Plane Nodes with AWS Integration

In my recent post on using kubeadm to set up a Kubernetes 1.13 cluster with AWS integration, I mentioned that I was still working out the details on enabling AWS integration (via the AWS cloud provider) while also using new functionality in kubeadm (specifically, the --experimental-control-plane flag) to make it easier to join new control plane nodes to the cluster. In this post, I’ll share with you what I’ve found to make this work.

The challenge here, by the way, is that you can’t use the --config <filename>.yaml flag and the --experimental-control-plane flag at the same time. I did try this, and the results of my testing led me to believe that although kubeadm doesn’t report an error, it does ignore the --experimental-control-plane flag. (Kubernetes experts/contributors, feel free to let me know if I’ve missed something here.)

After some trial-and-error—mostly my own fault because I didn’t take the time to review the v1beta1 kubeadm API docs ahead of time—I finally arrived at a working configuration that allows you to use kubeadm join --config <filename>.yaml to join a control plane node to an existing AWS-integrated Kubernetes cluster.

Credit for finding the solution goes to Rafael Fernández López, a developer based in Madrid who works for SUSE. Rafael and I had been chatting for a bit on the Kubernetes Slack team, and he pointed me to a particular value he’d been using with kubeadm that “flipped the bit,” so to speak, in telling kubeadm that the node you’re joining to the cluster is a control plane node and not a worker node. As proof, he pointed me to this snippet of code that was working for him.

So, based on Rafael’s input and my own subsequent testing, I arrived at this YAML configuration file:

kind: JoinConfiguration
    token: hjnu4i.ebza7otvg9axuqv7
    apiServerEndpoint: ""
    unsafeSkipCAVerification: true
    cloud-provider: aws

Naturally, you can’t just copy-and-paste this and use it; you’ll need to change a few values that are specific to your particular environment/situation:

  • The token field needs to specify a valid token. Bootstrap tokens generated by kubeadm init have a TTL of 24 hours, so you may need to use kubeadm token create to create a new token and specify that value here. Try to keep your tokens as short-lived as possible (use the --ttl flag to specify a short lifetime), as these are powerful authentication secrets.
  • The apiServerEndpoint will need to point to the DNS name of the load balancer that sits in front of your control plane.
  • You’ll have to specify unsafeSkipCAVerification: true unless you know the SHA256 hash of the CA certificate. If you do know the SHA256 hash, then you’d replace this line with caCertHashes: [<hash value>]. (The brackets are required.)
  • The value is one I’ve discussed multiple times; it need to match the EC2 Private DNS entry (and this is also what the OS should have configured for hostname).
  • Finally, the controlPlane.localAPIEndpoint.advertiseAddress should be the IP address of the instance you’re joining to the cluster as a new control plane node. This is the “magic sauce,” so to speak, that tells kubeadm you’re joining a control plane node to the cluster.

You’d use this configuration file by running kubeadm join --config <filename>.yaml. The node should join the cluster as a control plane node, and since the configuration file specifies the AWS cloud provider, you should also get AWS integration.

There you have it—a kubeadm configuration file that allows you to use kubeadm join to join new control plane nodes to a cluster while enabling the AWS cloud provider.

Additional Notes

I tested this using Kubernetes 1.14.0, but it should work the same way with Kubernetes 1.13.x as well. I used an external etcd cluster, but it should also work with stacked masters (etcd co-located with the control plane components). Note that I did not test the --experimental-upload-certs functionality in Kubernetes 1.14 (it’s on my list of things to do).

If you have any questions, comments, suggestions, or corrections, please contact me on Twitter. I’d love to hear from you.

My Team's Blogs

I’m thankful to have the opportunity to work with an amazing team. Many of my teammates also produce some very useful content via their own sites, and so I thought it might be useful to my readers to share a list of links to my teammates’ blogs.

Without further ado, here is a list of my teammates who have a blog; each entry is a link to the respective site (these are presented in no particular order):

I know I’ve gained valuable insight from some of their content, and I hope you do as well.

Spousetivities at Oktane 2019

It should come as no surprise to anyone that I’m a huge supporter of Spousetivities, and not just because it was my wife, Crystal Lowe, who launched this movement. What started as the gathering of a few folks at VMworld 2008 has grown over the last 11 years, and this year marks the appearance of Spousetivities at an entirely new conference: Oktane 2019!

Oktane is the conference for Okta, a well-known provider of identity services, and the event is happening in San Francisco from April 1 through April 4 (at Moscone West). This year, Okta is bringing Spousetivities in to add activities for those traveling to San Francisco with conference attendees.

What sort of activities are planned? The Oktane19 Spousetivities landing page has full details, but here’s a quick peek:

  • A wine tour in Sonoma/Napa with private transportation (lunch is included, of course!)
  • A walking food tour of San Francisco combined with a bus tour of the city and tickets to Beach Blanket Babylon
  • A whale watching tour

…and more!

If you’re attending Oktane19 and are bringing along a spouse, domestic partner, family member, or even just a friend—I’d definitely recommend signing them up for Spousetivities. What’s particularly cool about the activities at Oktane is that some activities—the wine tour and the walking tour—are available on Sunday, March 31, for folks arriving into San Francisco early. Nice!

To sign someone up for the activities, head on over to the Oktane 19 Spousetivities landing page. Enjoy!

Looking Ahead: My 2019 Projects

It’s been a little while now since I published my 2018 project report card, which assessed my progress against my 2018 project goals. I’ve been giving a fair amount of thought to the areas where I’d like to focus my professional (technical) development this coming year, and I think I’ve come up with some project goals that align both with where I am professionally right now and where I want to be technically as I grow and evolve. This is a really difficult balance to strike, and we’ll see at the end of the year how well I did.

Without further ado, here’s my list of 2019 project goals, along with an optional stretch goal (where it makes sense).

  1. Make at least one code contribution to an open source project. For the last few years, I’ve listed various programming- and development-related project goals. In all such cases, I haven’t done well with those goals because they were too vague, and—as I pointed out in previous project report cards—these less-than-ideal results are probably due to the way programming skills tend to be learned (by solving a problem/challenge instead of just learning language semantics and syntax). So, in an effort to align my desire to increase open source contributions along with a desire to improve my programming/development skills, I’m setting a goal to make at least one code contribution to an open source project this year. For the purposes of this goal, I will count “infrastructure-as-code” contributions (Ansible, Terraform, etc.) as one-fourth of a code contribution. Contributions/commits to my own Polyglot project do not count. (Stretch goal: Make three code contributions to open source projects.)

  2. Add at least three new technology areas to my “learning-tools” repository. Established a few years ago, my “learning-tools” repository contains tools and tutorials for learning new technologies. It’s gotten a bit stale over the last couple of years, so this year I want to add at least three new technology areas to this repository. I have a few ideas about some of the technology areas I’d like to add, but I’m going to leave this open so as to account for directional changes over the course of the year. These contributions/commits do not count against my previous project goal. (Stretch goal: Add five new technology areas to the “learning-tools” repository.)

  3. Become more familiar with CI/CD solutions and patterns. In 2018 I focused the majority of my energy on becoming more fluent in Kubernetes (and I did reasonably well, though there is still plenty to learn). In 2019, I need to “move up the stack” a bit and increase my knowledge and experience with CI/CD solutions and usage patterns, particularly in containerized environments. I know that this goal is rather vague, but at this point I’m not really sure how I can make it more specific, measurable, and concrete.

  4. Create at least three non-written content pieces. I’ve been blogging for a long time (14 years as of May 2019), and previous attempts at other forms of content creation have not been quite as successful. This year, I’m going to try again, but without specifying what type of content (only that it is non-written content). It could be a presentation published via Slideshare or SpeakerDeck, a video tutorial published on YouTube, or a graphic/diagram posted somewhere. Audio content created for the Full Stack Journey podcast will not count against this project goal. (Stretch goal: Create five pieces of non-written content.)

  5. Complete a “wildcard project” (if applicable). As I’ve done in previous years, I’m going to allow room for a “wildcard project.” It’s difficult, if not impossible, to completely chart where career or projects will take me, so I use the “wildcard project” as a means of addressing that variability in the future. I won’t grade myself negatively if I don’t complete one.

So there’s my list of 2019 project goals. I’ve tried to take the lessons learned from previous years to make this year’s goals as specific and measurable as possible (where possible), and to align these goals with each other and with the larger trends in my career and the industry. Time will tell how effective I was with that alignment.

Feel free to hit me up on Twitter if you have questions or comments about these project goals. I’d certainly love to hear your feedback!

Recent Posts

Split Tunneling with vpnc

vpnc is a fairly well-known VPN connectivity package available for most Linux distributions. Although the vpnc web site describes it as a client for the Cisco VPN Concentrator, it works with a wide variety of IPSec VPN solutions. I’m using it to connect to a Palo Alto Networks-based solution, for example. In this post, I’d like to share how to set up split tunneling for vpnc.


Advanced AMI Filtering with JMESPath

I recently had a need to do some “advanced” filtering of AMIs returned by the AWS CLI. I’d already mastered the use of the --filters parameter, which let me greatly reduce the number of AMIs returned by aws ec2 describe-images. In many cases, using filters alone got me what I needed. In one case, however, I needed to be even more selective in returning results, and this lead me to some (slightly more) complex JMESPath queries than I’d used before. I wanted to share them here for the benefit of my readers.


Technology Short Take 111

Welcome to Technology Short Take #111! I’m a couple weeks late on this one; wanted to publish it earlier but work has been keeping me busy (lots and lots of interest in Kubernetes and cloud-native technologies out there!). In any event, here you are—I hope you find something useful for you!


Thoughts on VPNs for Road Warriors

A few days ago I was talking with a few folks on Twitter and the topic of using VPNs while traveling came up. For those that travel regularly, using a VPN to bypass traffic restrictions is not uncommon. Prompted by my former manager Martin Casado, I thought I might share a few thoughts on VPN options for road warriors. This is by no means a comprehensive list, but hopefully something I share here will be helpful.


Kubernetes, Kubeadm, and the AWS Cloud Provider

Over the last few weeks, I’ve noticed quite a few questions appearing in the Kubernetes Slack channels about how to use kubeadm to configure Kubernetes with the AWS cloud provider. You may recall that I wrote a post about setting up Kubernetes with the AWS cloud provider last September, and that post included a few snippets of YAML for kubeadm config files. Since I wrote that post, the kubeadm API has gone from v1alpha2 (Kubernetes 1.11) to v1alpha3 (Kubernetes 1.12) and now v1beta1 (Kubernetes 1.13). The changes in the kubeadm API result in changes in the configuration files, and so I wanted to write this post to explain how to use kubeadm 1.13 to set up a Kubernetes cluster with the AWS cloud provider.


Scraping Envoy Metrics Using the Prometheus Operator

On a recent customer project, I recommended the use of Heptio Contour for ingress on their Kubernetes cluster. For this particular customer, Contour’s support of the IngressRoute CRD and the ability to delegate paths via IngressRoutes made a lot of sense. Of course, the customer wanted to be able to scrape metrics using Prometheus, which meant I not only needed to scrape metrics from Contour but also from Envoy (which provides the data plane for Contour). In this post, I’ll show you how to scrape metrics from Envoy using the Prometheus Operator.


Technology Short Take 110

Welcome to Technology Short Take #110! Here’s a look at a few of the articles and posts that have caught my attention over the last few weeks. I hope something I’ve included here is useful for you also!


Technology Short Take 109

Welcome to Technology Short Take #109! This is the first Technology Short Take of 2019. It may be confirmation bias, but I’ve noticed of number of sites adding “Short Take”-type posts to their content lineup. I’ll take that as flattery, even if it wasn’t necessarily intended that way. Enjoy!


On Thinking About Infrastructure as Code

I just finished reading Cindy Sridharan’s excellent post titled “Effective Mental Models for Code and Systems,” and some of the points Sridharan makes immediately jumped out to me—not for “traditional” code development, but for the development of infrastructure as code. Take a few minutes to go read the post—seriously, it’s really good. Done reading it? Good, now we can proceed.


The Linux Migration: December 2018 Progress Report

In December 2016, I kicked off a migration from macOS to Linux as my primary laptop OS. Throughout 2017, I chronicled my progress and challenges along the way; links to all those posts are found here. Although I stopped the migration in August 2017, I restarted it in April 2018 when I left VMware to join Heptio. In this post, I’d like to recap where things stand as of December 2018, after 8 months of full-time use of Linux as my primary laptop OS.


Looking Back: 2018 Project Report Card

Over the last five years or so, I’ve shared with my readers an annual list of projects along with—at the year’s end—a “project report card” on how I fared against the projects I’d set for myself. (For example, here’s my project report card for 2017.) Following that same pattern, then, here is my project report card for 2018.


The Linux Migration Series

In early 2017 I kicked off an effort to start using Linux as my primary desktop OS, and I blogged about the journey. That particular effort ended in late October 2017. I restarted the migration in April 2018 (when I left VMware to join Heptio), and since that time I’ve been using Linux (Fedora, specifically) full-time. However, I thought it might be helpful to collect the articles I wrote about the experience together for easy reference. Without further ado, here they are.


Technology Short Take 108

Welcome to Technology Short Take #108! This will be the last Technology Short Take of 2018, so here’s hoping I can provide something useful for you. Enjoy!


Running Fedora on my Mac Pro

I’ve been working on migrating off macOS for a couple of years (10+ years on a single OS isn’t undone quickly or easily). I won’t go into all the gory details here; see this post for some background and then see this update from last October that summarized my previous efforts to migrate to Linux (Fedora, specifically) as my primary desktop operating system. (What I haven’t blogged about is the success I had switching to Fedora full-time when I joined Heptio.) I took another big step forward in my efforts this past week, when I rebuilt my 2011-era Mac Pro workstation to run Fedora.


KubeCon 2018 Day 2 Keynote

This is a liveblog of the day 2 (Wednesday) keynotes at KubeCon/CloudNativeCon 2018 in Seattle, WA. For additional KubeCon 2018 coverage, check out other articles tagged KubeCon2018.


Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!