Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Fine-Tuning Control Plane Access with Cluster API

When Cluster API creates a workload cluster, it also creates a load balancing solution to handle traffic to the workload cluster’s control plane. This is necessary so that the control plane endpoint is decoupled from the underlying control plane nodes (which facilitates scaling the control plane, among other things). On AWS, this mean creating an ELB and a set of security groups. For flexibility, Cluster API provides a limited ability to customize this control plane load balancer. In this post, I’ll show you how to use this functionality to fine-tune access to a workload cluster’s control plane when using Cluster API with AWS.

If you’re not familiar with Cluster API (hereafter just referred to as “CAPI”), then my introduction to CAPI article may be useful. Keep in mind that article was written in 2019, while the project was still in its early stages. The high-level concepts are correct, but some of the details may have shifted slightly over the last three years as the project progressed from v1alpha1 APIs to the now-current v1beta1 APIs.

The key here is the controlPlaneLoadBalancer object, which is part of the AWSCluster object (see details here in the code or here via pkg.go.dev). With regard to access to a workload cluster’s control plane via its control plane load balancer, two fields are of particular interest:

  1. The additionalSecurityGroups field allows you to list any additional security groups—referenced by security group ID—that the control plane load balancer should use (this is in addition to the standard set of security groups created by CAPI). This is an optional field.
  2. The scheme field allows you to specify whether the load balancer should be externally-accessible from the Internet (specified with the value “internet-facing”, which is the default value) or not (specified with the value “internal”). If you specify “internal” as the scheme, then the load balancer won’t get a public IP address and won’t be reachable via the Internet. This would be a great way to restrict access to only those connections coming via a VPN (or similar) into the VPC.

Personally, I would at least switch the scheme on the control plane load balancer; otherwise, the API server of your newly-created workload cluster is publicly accessible via the Internet. You can then leverage any additional security groups as needed.

One note to keep in mind: the default security group created by the AWS provider for CAPI (referred to as CAPA) allows access from 0.0.0.0/0 to the control plane load balancer. If you need to restrict access to something more fine-grained than that, using the additionalSecurityGroups field is not the way. As outlined here, you’ll instead want to use the securityGroupOverrides field (which is found at spec.network.securityGroupOverrides).

I hope this information is useful. Feel free to reach out to me if you have any questions (or corrections, in the event I explained/presented something incorrectly). You can find me on the Kubernetes Slack (as well as a number of other Slack communities) as well as on Twitter.

Technology Short Take 154

Welcome to Technology Short Take #154! My link of links and articles from around the Internet is a bit light on networking and virtualization this time around, but heftier in the security, cloud, and OS/application sections. I hope that I’ve managed to include something that you’ll find useful. Enjoy the content!

Networking

  • Lucas Pardue and Christopher Wood share a primer on proxies. (Hat tip to Matt Oswalt for putting this on my Twitter timeline.)
  • Ivan Pepelnjak talks about the 1.2.1 release of netsim-tools in this blog post.

Servers/Hardware

  • It seems like silicon photonics is something the industry has been talking about for years, but something that never seems to come to fruition. The Next Platform recently chatted with Andy Bechtolsheim about it; you can read the results of that discussion here.

Security

Cloud Computing/Cloud Management

Operating Systems/Applications

Storage

Virtualization

Career/Soft Skills

  • Robin Moffatt, a developer advocate at Confluent, recently wrote about hanging up his boarding passes and jetlag (for now). I have a lot of respect for Robin taking the time to do some serious introspection, and deciding to put family first. Kudos.
  • I love this motivational post by Ivan Pepelnjak on why you (yes, you!) should keep blogging. And thanks for the shout-out at the end of that post, Ivan!

That’s all for now! I’ll be back in a couple weeks with another Technology Short Take, but until then feel free to contact me if you have links to share, interesting articles you’ve found, or if you just want to say hello! You can find me on any number of Slack communities, or you can contact me on Twitter. Thanks for reading!

Technology Short Take 153

Welcome to Technology Short Take #153! My personal and professional life has kept me busy over the last couple of months, so things have been quiet here on the blog. I’ve still been collecting links to share with you, though, and here’s the latest collection. I hope you’re able to find something useful here!

Networking

  • This article contains some good information on IPv6 for those who are just starting to get more familiar with it, although toward the end it turns into a bit of an advertisement.
  • Want to understand kube-proxy, a key part of Kubernetes networking, a bit better? Start here. Arthur Chiao’s post on cracking kube-proxy is also an excellent resource—in fact, there’s so much information packed in there you may need to read it more than once.
  • Xavier Avrillier walks readers through using Antrea (a Kubernetes CNI built on top of Open vSwitch—a topic I’ve touched on a time or two) to provide on-premise load balancing in Kubernetes.

Servers/Hardware

  • Cabling is hardware, right? What happens to submarine cables when there are massive events, like a volcanic eruption? Ulrich Speidel shares some of the findings after the volcanic eruption in Tonga.

Security

  • Although Linux is often considered to be superior to Windows and macOS with regard to security, it is not without its own security flaws. I, personally, would posit that Linux can be more secure than other platforms, but it requires proper configuration and can still be undone by a loose nut behind the keyboard.
  • Ouch—another Azure vulnerability that allows cross-account access.

Cloud Computing/Cloud Management

Operating Systems/Applications

  • Steven Bright shows how to deploy Salt minions automatically using VMware Tools. (Note that this page doesn’t seem to render properly in Firefox, although Chrome and Safari seem fine.)
  • I found this article to be a good overview of WebAssembly.
  • Matt Oswalt takes readers though a fairly in-depth look at sockets and address binds in Linux.
  • If you are an absolute beginner with Envoy Proxy, this tutorial will help you get started. However, it quickly runs out of steam, and you’ll need to progress to more advanced examples and documentation to continue your learning journey.
  • I asked a question about Git, Git tags, and releasing versions of a project on Twitter the other day (here’s the tweet), and this article on Git branching was shared with me. Perhaps you’ll find it useful as well!
  • I’ve been digging into OIDC/OAuth 2.0 lately, and learning about JWTs (JSON Web Tokens, see here for an introduction). I found this article telling folks to stop overloading JWTs with permissions claims a while ago, but at the time I didn’t fully understand all the concepts or the implications. Now that I’ve had some time and some exposure, this makes a lot more sense to me—and I agree with the author!
  • Speaking of JWTs: they don’t score so well in Thomas Ptacek’s survey of API tokens. There’s a lot to unpack in Ptacek’s article; I suspect I’ll be coming back to read it again later as my knowledge in this area grows. Some of it is still too advanced for me right now.

Storage

Nothing here this time around—maybe go check J Metz’ blog instead?

Virtualization

Career/Soft Skills

  • Have you ever had one of those days where you just aren’t feeling like working? I think we all have. Here is an article from the Harvard Business Review on some strategies for motivating yourself when that happens.
  • Although written from the perspective of a developer seeking to improve their knowledge in a particular programming language, I think the ideas and principles laid out in this article on how developers remember things are applicable to folks learning new things in other IT disciplines as well.

That’s it for this time around. I’ll be back soon with more original content as well as more Short Takes! In the meantime, feel free to reach out to me—my e-mail address isn’t hard to find, I’m present in a number of Slack communities, and you can always hit me on Twitter (DMs are open). Thanks for reading!

Technology Short Take 152

Welcome to Technology Short Take #152! Normally I’d publish a Technology Short Take in the morning on a Friday, but I really wanted to get this one out so I’m making it live late in the day on a Monday. Here’s hoping I’ve included some content below that you find useful!

Networking

  • I was (and am) familiar with RFC 1918 and the concept of non-routable address spaces. However, I was not familiar with the term “bogons” to refer to such prefixes that should not be publicly routed. Thanks to this article, that oversight is now corrected. Oh, and the article shares a handy Python script to help implement bogon filtering in NSX-T.
  • Koyeb describes, at a high level, the global networking stack for their serverless platform. Components involved include the open source Kuma service mesh (in turn leveraging Envoy), anycast BGP, and mutual TLS (mTLS).
  • Ivan Pepelnjak does a great job of describing all the things you really shouldn’t do (or don’t really need to do) when trying to deal with migrating container hosts in a data center fabric. In truth, the answer is exactly as Ivan says at the top of the article: when it comes to migrating container hosts, you don’t. (Seriously.)

Servers/Hardware

Security

Cloud Computing/Cloud Management

  • Anders Eknert shows how to do—or at least provides a framework on getting started with—dynamic policy composition with Open Policy Agent (OPA) in this blog post from April 2021.
  • Here’s an article on how to get started with OrgFormation.
  • This workshop on using Graviton2 on AWS is pretty neat.
  • Xavier Avrillier writes about how to workaround a vSphere CSI controller CrashLoopBackoff when using Cluster API for vSphere (CAPV).
  • Jorge de la Cruz shares how to install Kubernetes on Ubuntu 20.04. Most of my readers are probably already familiar with this process, but I wanted to share it here just in case.
  • Marc Brooker has a pair of articles on circuit breakers and retries that are, in my opinion, worth reading (part 1 on circuit breakers is here, part 2 on retries is here). And when I say “circuit breakers,” I’m not talking about your electrical panel, either. Read the articles if you’re unfamiliar with the concept of circuit breakers in modern distributed systems.
  • For folks who may be just getting started with Terraform, here’s a tutorial on how to deploy EC2 instances in AWS using Terraform.
  • I just came across this relatively-new EKS News site. Have a look at the archives for links to all the issues of EKS News. It’s a shame the site doesn’t appear to have an RSS/Atom feed…

Operating Systems/Applications

Storage

Programming

Virtualization

  • Normally Ivan Pepelnjak’s articles land in the “Networking” section, but this time Ivan is squarely discussing virtualization when he talks about running an Ubuntu VM on an M1-based Mac. I guess this is just yet another example of how fluid the “boundaries” between technical disciplines has become.

Career/Soft Skills

  • I really enjoyed this post on lessons learned by moving the office home. There’s several nuggets in here that I need to apply to my next office move (which will be happening in about a month or so, if I don’t run into any more delays).
  • I like this idea of thinking about an intention instead of resolutions.

That’s a wrap! I’ll be back in a couple of weeks with another Technology Short Take; until then, keep exploring, keep learning, and keep sharing! If you want to contact me—perhaps to provide some feedback, suggest a site you’ve found helpful, or just to say hello—feel free to contact me on Twitter or hit me up in any one of a number of Slack communities. I’d love to hear from you.

Using cert-manager with Kuma for mTLS

When configuring mutual TLS (mTLS) on the open source Kuma service mesh, users have a couple of different options. They can use a “builtin” certificate authority (CA), in which Kuma itself will generate a CA certificate and key for use in creating service-specific mTLS certificates. Users also have the option of using a “provided” CA, in which they must supply a CA certificate and key for Kuma to use when creating service-specific mTLS certificates. Both of these options are described on this page in the Kuma documentation. In this post, I’d like to explore the use of cert-manager as a “provided” CA for mTLS on Kuma.

Currently, Kuma lacks direct integration with cert-manager, so the process is a bit more manual than I’d prefer. If direct cert-manager integration is something you’d find useful, please consider opening an issue to that effect on the Kuma GitHub repository.

Assuming you have cert-manager installed already, the process for using cert-manager as the CA for a “provided” CA mTLS backend looks like this:

  1. Define the root CA in cert-manager.
  2. Prepare the secrets for Kuma.
  3. Configure the Kuma mesh object for mTLS.

I know these steps are really too high level to be useful on their own, so I’ll step through them in a bit more detail in the sections below.

Define the mTLS Root CA

If you’ve browsed the cert-manager documentation, you’ve probably seen that there are a variety of ways to set up an Issuer (a means of issuing Certificate objects managed by cert-manager). In this example, I’ll just use a SelfSigned issuer, although you could use a different method. (To be honest, though, if you were going to go down the CA route you might as well just use the CA certificate and key with Kuma directly.)

To create a SelfSigned issuer, this piece of YAML would work:

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

Then use that SelfSigned issuer to issue a root CA for Kuma to use:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kuma-mtls-root-ca
  namespace: kuma-system
spec:
  isCA: true
  commonName: kuma-mtls-root-ca
  secretName: kuma-mtls-root-ca
  duration: 43800h # 5 years
  renewBefore: 720h # 30d
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - digital signature
    - key encipherment
    - cert sign
  issuerRef:
    name: selfsigned-issuer # References self-signed ClusterIssuer
    kind: ClusterIssuer
    group: cert-manager.io

Once this is applied to the cluster where cert-manager is running (which doesn’t necessarily need to be the cluster where Kuma will be installed, as you’ll see in a moment), then cert-manager will create not only the Certificate resource but also a corresponding Secret that contains the certificate and the private key (the name of that Secret is controlled by the spec.secretName field above). You’ll use that secret for the next step, which is preparing the secrets for Kuma.

Preparing Secrets for Kuma

It would be nice if Kuma could just use/read the Secret created by cert-manager directly, but that functionality doesn’t currently exist. As a result, we’ll need to take the Kubernetes Secret created by cert-manager and turn it into something that Kuma can leverage.

To do this, you’ll need to make two mesh-scoped Kuma secrets (see the Kuma secrets documentation). On Kubernetes, these will be Kubernetes Secrets, but labeled for a specific mesh and tagged as a specific type.

This example template shows what a mesh-scoped Kuma secret would look like on Kubernetes:

---
apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
data:
  value: blah
type: system.kuma.io/secret

The label (kuma.io/mesh: default), the type (system.kuma.io/secret), and the namespace (kuma-system) mark this as a mesh-scoped Kuma secret. How do you take the Kubernetes Secret generated by cert-manager and turn it into one of these?

Well, you could export the contents of the Kubernetes Secret created by cert-manager (using kubectl get secret <name> -jsonpath='{.data.tls\.crt}', piping that through base64 to decode it, and then redirecting to a file), and then use the file to create a new mesh-scoped Kuma secret using the template above, making sure to re-encode the file contents with base64 alone the way. However, there’s no real need to have the CA certificate and key in a file on the disk, and there’s no real sense in decoding and then re-encoding the Secret’s contents. Is there a better way?

In fact, there is. Using yq (here’s the GitHub repository for it), you can do what you need with a single command (I’ll break this command down in a moment):

yq e ".data.value = \
\"$(kubectl -n kuma-system get secret kuma-mtls-root-ca \
-jsonpath='{.data.tls\.crt}')\" secret-template.yaml | \
yq e '.metadata.name = "mtls-root-certificate"' -

That’s a doozy! What’s happening here? You’re using yq to modify the values of a couple of fields; there’s two iterations of yq, each modifying a different field. The first iteration of yq modifies the .data.value field, making its value equal to the output of the kubectl get secret command (which is the base64-encoded CA certificate in the Secret created by cert-manager). yq expects double quotes around the value being assigned to the field, but double quotes are needed to do the command substitution so the “inner quotes” are backslash-escaped. The second iteration of yq modifies the .metadata.name field to be equal to “mtls-root-certificate”. Since there’s no command substitution in this iteration, you’re using single quotes around the yq operation and unescaped double quotes around the value.

The end result looks something like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: mtls-root-certificate
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
data:
  value: LS0tLS1CRUdJTiBDRVJUSUZJ... # Base64-encoded contents
type: system.kuma.io/secret

Just add one final pipe (pipe it into kubectl apply -f -) to the command above and you’re good to go.

Repeat for the key:

yq e ".data.value = \
\"$(kubectl -n kuma-system get secret kuma-mtls-root-ca \
-jsonpath='{.data.tls\.key}')\" secret-template.yaml | \
yq e '.metadata.name = "mtls-root-key"' - | \
kubectl apply -f -

You’re now ready for the final step, which is configuring the Kuma mesh for mTLS.

Configuring the Mesh for mTLS

Fortunately, this step is pretty straightforward. Working from the example found in the Kuma mTLS documentation for a “provided” backend, something like this would work:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: cert-manager
    backends:
      - name: cert-manager
        type: provided
        dpCert:
          rotation:
            expiration: 1d
        conf:
          cert:
            secret: mtls-root-certificate
          key:
            secret: mtls-root-key

The backend name is entirely subjective, of course, but the names of the secrets referenced by spec.conf.cert.secret and spec.conf.key.secret must match the names of the mesh-scoped Kuma secrets you created in the previous section.

And that’s it! Once this configuration is in place, Kuma will use the CA certificate issued by cert-manager (which you then translated into a pair of mesh-scoped Kuma secrets) to automatically issue and sign the mTLS certificates used by the data plane proxies. (You could use the Envoy admin API to see more details on these certificates, but that’s a different post for a different day.)

If you have any questions about this, or about Kuma in general, feel free to hit me up. You can find me in the Kuma community Slack, or hit me on Twitter. My DMs are open and I’d be happy to help if I’m able.

Recent Posts

Follow Up: Bootstrapping Servers into Ansible

Seven years ago, I wrote a quick post on bootstrapping servers into Ansible. The basic gist of the post was that you can use variables on the Ansible command-line to specify hosts that aren’t part of your inventory or log in via a different user (useful if the host doesn’t yet have a dedicated Ansible user account because you want to use Ansible to create that account). Recently, though, I encountered a situation where this approach doesn’t work, and in this post I’ll describe the workaround.

Read more...

Technology Short Take 151

Welcome to Technology Short Take #151, the first Technology Short Take of 2022. I hope everyone had a great holiday season and that 2022 is off to a wonderful start! I have a few more links than normal this time around, although I didn’t find articles in a couple categories. Don’t worry—I’ll keep my eyes peeled and my RSS reader ready to pull in new articles in those categories for next time. And now for the content!

Read more...

Getting Certificate Details from HashiCorp Vault

It seems there are lots of tutorials on setting up a PKI (public key infrastructure) using HashiCorp Vault. What I’ve found missing from most of these tutorials, however, is how to get details on certificates issued by a Vault-driven PKI after the initial creation. For example, someone other than you issued a certificate, but now you need to get the details for said certificate. How is that done? In this post, I’ll show you a couple ways to get details on certificates issued and stored in HashiCorp Vault.

Read more...

Using Test-Driven Development for Kustomize Overlays

I am by no means a developer (not by a long shot!), but I have been learning lots of development-related things over the last several years and trying to incorporate those into my workflows. One of these is the idea of test-driven development (see Wikipedia for a definition and some additional information), in which one writes tests to validate functionality before writing the code to implement said functionality (pardon the paraphrasing). In this post, I’ll discuss how to use conftest to (loosely) implement test-driven development for Kustomize overlays.

Read more...

Technology Short Take 150

Welcome to Technology Short Take #150! This is the last Technology Short Take of 2021, so hopefully I’ll close the year out “with a bang” with this collection of links and articles on various technology areas. Bring on the content!

Read more...

Review: OWC Thunderbolt 3 Dock

About six months ago I purchased an OWC Thunderbolt 3 Dock to replace my Anker PowerElite Thunderbolt 3 Dock (see my review here). While there was nothing necessarily wrong with the Anker PowerElite, it lacked a digital audio port that I could use to send audio to a soundbar positioned under my monitor. (I’d grown accustomed to using a soundbar when my 2012 Mac Pro was my primary workstation.) In this post, I’ll provide a brief review of the OWC Thunderbolt 3 Dock.

Read more...

Technology Short Take 149

Welcome to Technology Short Take #149! I’ll have one more Technology Short Take in 2021, scheduled for three weeks from now (on the last day of the year!). For now, though, I have a small collection of articles and links for your reading pleasure—not as many as I usually include in a Technology Short Take, but better than nothing at all (I hope!). Enjoy!

Read more...

Technology Short Take 148

Welcome to Technology Short Take #148, aka the Thanksgiving Edition (at least, for US readers). I’ve been scouring RSS feeds and various social media sites, collecting as many useful links and articles as I can find: from networking hardware and networking CI/CD pipelines to Kernel TLS and tricks for improving your working memory. That’s quite the range! I hope that you find something useful here.

Read more...

Using Kustomize Components with Cluster API

I’ve been using Kustomize with Cluster API (CAPI) to manage my AWS-based Kubernetes clusters for quite a while (along with Pulumi for managing the underlying AWS infrastructure). For all the time I’ve been using this approach, I’ve also been unhappy with the overlay-based approach that had evolved as a way of managing multiple workload clusters. With the recent release of CAPI 1.0 and the v1beta1 API, I took this opportunity to see if there was a better way. I found a different way—time will tell if it is a better way. In this post, I’ll share how I’m using Kustomize components to help streamline managing multiple CAPI workload clusters.

Read more...

Technology Short Take 147

Welcome to Technology Short Take #147! The list of articles is a bit shorter than usual this time around, but I’ve still got a good collection of articles and posts covering topics in networking, hardware (mostly focused on Apple’s processors), cloud computing, and virtualization. There’s bound to be something in here for most everyone! (At least, I hope so.) Enjoy your weekend reading!

Read more...

Influencing Cluster API AMI Selection

The Kubernetes Cluster API (CAPI) project—which recently released v1.0—can, if you wish, help manage the underlying infrastructure associated with a cluster. (You’re also fully able to have CAPI use existing infrastructure as well.) Speaking specifically of AWS, this means that the Cluster API Provider for AWS is able to manage VPCs, subnets, routes and route tables, gateways, and—of course—EC2 instances. These EC2 instances are booted from a set of AMIs (Amazon Machine Images, definitely pronounced “ay-em-eye” with three syllables) that are prepared and maintained by the CAPI project. In this short and simple post, I’ll show you how to influence the AMI selection process that CAPI’s AWS provider uses.

Read more...

Creating Reusable Kuma Installation YAML

Using CLI tools—instead of a “wall of YAML”—to install things onto Kubernetes is a growing trend, it seems. Istio and Cilium, for example, each have a CLI tool for installing their respective project. I get the reasons why; you can build logic into a CLI tool that you can’t build into a YAML file. Kuma, the open source service mesh maintained largely by Kong and a CNCF Sandbox project, takes a similar approach with its kumactl tool. In this post, however, I’d like to take a look at creating reusable YAML to install Kuma, instead of using the CLI tool every time you install.

Read more...

Using the External AWS Cloud Provider for Kubernetes

In 2018, after finding a dearth of information on setting up Kubernetes with AWS integration/support, I set out to try to establish some level of documentation on this topic. That effort resulted in a few different blog posts, but ultimately culminated in this post on setting up an AWS-integrated Kubernetes cluster using kubeadm. Although originally written for Kubernetes 1.15, the process described in that post is still accurate for newer versions of Kubernetes. With the release of Kubernetes 1.22, though, the in-tree AWS cloud provider—which is what is used/described in the post linked above—has been deprecated in favor of the external cloud provider. In this post, I’ll show how to set up an AWS-integrated Kubernetes cluster using the external AWS cloud provider.

Read more...

Kustomize Transformer Configurations for Cluster API v1beta1

The topic of combining kustomize with Cluster API (CAPI) is a topic I’ve touched on several times over the last 18-24 months. I first touched on this topic in November 2019 with a post on using kustomize with CAPI manifests. A short while later, I discovered a way to change the configurations for the kustomize transformers to make it easier to use it with CAPI. That resulted in two posts on changing the kustomize transformers: one for v1alpha2 and one for v1alpha3 (since there were changes to the API between versions). In this post, I’ll revisit kustomize transformer configurations again, this time for CAPI v1beta1 (the API version corresponding to the CAPI 1.0 release).

Read more...

Technology Short Take 146

Welcome to Technology Short Take #146! Over the last couple of weeks, I’ve gathered a few technology-related links for you all. There’s some networking stuff, a few security links, and even a hardware-related article. But enough with the introduction—let’s get into the content!

Read more...

Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!