Scott's Weblog The weblog of an IT pro focusing on cloud computing, Kubernetes, Linux, containers, and networking

Pausing Cluster API Reconciliation

Cluster API is a topic I’ve discussed here in a number of posts. If you’re not already familiar with Cluster API (also known as CAPI), I’d encourage you to check out my introductory post on Cluster API first; you can also visit the official Cluster API site for more details. In this short post, I’m going to show you how to pause the reconciliation of Cluster API cluster objects, a task that may be necessary for a variety of reasons (including backing up the Cluster API objects in your management cluster).

Since CAPI leverages Kubernetes-style APIs to manage Kubernetes cluster lifecycle, the idea of reconciliation is very important—it’s a core Kubernetes concept that isn’t at all specific to CAPI. This article on level triggering and reconciliation in Kubernetes is a great article that helps explain reconciliation, as well as a lot of other key concepts about how Kubernetes works.

When reconciliation is active, the controllers involved in CAPI are constantly evaluating desired state and actual state, and then reconciling differences between the two. There may be times when you need to pause this reconciliation loop. Fortunately, CAPI makes this pretty easy: there is a paused field that allows users to pause the reconciliation loop (see here in the v1alpha3 CAPI reference).

This field is optional, which you can observe using kubectl with an existing workload cluster. For example, here is the relevant output from kubectl get cluster workload1 against a CAPI management cluster; note the paused field is not present (i.e., reconciliation is currently active):

spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneEndpoint:
    host: workload1-apiserver-1234567890.us-east-2.elb.amazonaws.com
    port: 6443
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: KubeadmControlPlane
    name: workload1-control-plane
    namespace: workload1
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSCluster
    name: workload1
    namespace: workload1

Reconcilation is active when the field isn’t present; setting the field to false is essentially the same as removing the field. To pause reconciliation, set it to true. One way of doing this is using kubectl patch, like this:

kubectl path cluster workload1 --type merge \
-p '{"spec":{"paused": true}}'

Naturally, you’d want to change workload1 to the name of the workload cluster for which you want reconciliation paused. After running the command, you can then run kubectl get cluster <cluster-name> -o yaml and verify the paused field is present:

spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneEndpoint:
    host: workload1-apiserver-1234567890.us-east-2.elb.amazonaws.com
    port: 6443
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: KubeadmControlPlane
    name: workload1-control-plane
    namespace: workload1
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSCluster
    name: workload1
    namespace: workload1
  paused: true

If you check the logs for the CAPI controllers, you’ll see messages like this:

I1125 17:31:40.287717       1 machine_controller.go:170] controllers/Machine "msg"="Reconciliation is paused for this object" "machine"="workload1-md-0-58c6df55bc-cqt86" "namespace"="workload1"

When reconciliation is paused, the CAPI controllers will not make changes to the workload cluster in order to reconcile differences between actual state and desired state.

When you’re ready to resume reconciliation again, just change the field to false:

kubectl path cluster workload1 --type merge \
-p '{"spec":{"paused": false}}'

This will remove the field from the spec, allowing the CAPI controllers to resume reconciliation of the CAPI-related objects (like Clusters, Machines, and MachineDeployments).

As I mentioned at the start of this post, pausing CAPI reconciliation may be necessary in a few different situations. (Backing up Cluster API is one situation.)

If anyone has any questions, or if you spot an error in this post, please let me know. Hit me on Twitter; I’d love to hear from you!

Technology Short Take 134

Welcome to Technology Short Take #134! I’m publishing a bit early this time due to the Thanksgiving holiday in the US. So, for all my US readers, here’s some content to peruse while enjoying some turkey (or whatever you’re having this year). For my international readers, here’s some content to peruse while enjoying dramatically lower volumes of e-mail because the US is on holiday. See, something for everyone!

Networking

Security

  • I’m glad to see this. Open source has become so critical to so many aspects of our computing infrastructure.
  • OpenCSPM looks like it could be quite a useful tool. I haven’t yet had time to dig in and get familiar with the details, but what I have seen so far looks good.
  • Uh oh…more hardware exploits.
  • The macOS OCSP fiasco generated quite a bit of attention and analysis (see here and here).

Cloud Computing/Cloud Management

Operating Systems/Applications

  • One of the things I love about Matt Oswalt is that he exemplifies the idea of a perpetual learner. The latest example is Matt’s post on the anatomy of a binary executable, in which he dives deep into what exactly it means to be a binary executable file. Good stuff!
  • You may have heard of eBPF, the Linux technology that is reshaping Linux applications (and in some ways reshaping Linux itself). Brendan Gregg discusses the future of BPF binaries, made possible through BTF and CO-RE. The idea of creating ELF binaries (don’t know what that is? See the previous bullet!) for BPF is pretty cool, in my opinion, and has the potential to unlock a lot of innovation in this space.
  • Something about Linux, and Fedora in particular, just keeps drawing me back. If you’re in a similar boat, and you’re looking for information on how to get Firefox on Fedora to play H.264 videos, Leo Chavez has some information that should help.
  • Here’s a bit of history on macOS, for those of us interested in such things.
  • This looks horribly confusing. It almost feels like we are well into Windows Registry territory here.
  • What a time to be alive: Microsoft has its own Linux distribution.
  • Whether it be dissatisfaction with macOS 11 “Big Sur,” or unhappiness at the direction of their hardware (there’s some discussion that the new M1 chips don’t support eGPUs), or concerns over privacy given the recent issues with OCSP and macOS “dialing home,” I’m seeing folks leaving macOS for other platforms (mostly Linux). Preslav Rachev shares his story here, and Juan Diego Caballero shares his story here.

Storage

Virtualization

That’s all this time, but hopefully it’s enough! If you have suggestions for content to include in a future Technology Short Take, or if you’d just like to catch up and say hello, feel free to contact me on Twitter. Enjoy the rest of your week!

Review: CPLAY2air Wireless CarPlay Adapter

In late September, I was given a CPLAY2air wireless CarPlay adapter as a gift. Neither of my vehicles support wireless CarPlay, and so I was looking forward to using the CPLAY2air device to enable the use of CarPlay without having to have my phone plugged into a cable. Here’s my feedback on the CPLAY2air device after about six weeks of use.

In general, the device works reasonably well. Setup with the factory radio in my GMC Sierra truck was pretty straightforward, and only took a few minutes. Since then, the device connects to my phone every time I start the vehicle, and all CarPlay functions—music, maps, Siri, sending/reading messages, etc.—all work as expected. I can leave my phone in my pocket and still gain all the benefits of CarPlay, which is incredibly convenient.

The CPLAY2air is not without a few caveats, however. The wireless connection between the CPLAY2air and my iPhone does introduce some noticeable latency. When switching music tracks, for example, it will take between one and three seconds to stop playing the old track and start playing the new track. (By comparison, the same operation using traditional wired CarPlay is nearly instantaneous.) Similarly, when talking on the phone, there is latency in the conversation—almost like the latency one might experience on international calls. Fortunately, the latency does not adversely affect the ability to use Siri, even though the latency is still there.

The other noticeable caveat regarding the CPLAY2air is the startup time. It often takes several minutes after I start the vehicle before the CPLAY2air makes the necessary connection to my iPhone and to the factory radio and wireless CarPlay starts working. Occasionally there will be an error connecting (this is reflected with an error message on the screen) and it takes even longer to retry. In general, though, this is more of a “be aware of this”-type item as opposed to a significant limitation.

Despite these drawbacks, I remain a fan of the CPLAY2air for the added convenience it offers.

Feel free to contact me on Twitter if you have any questions, or if you’d like to share your experience with this device (or related devices). Thanks for reading!

Resizing Windows to a Specific Size on macOS

I recently had a need (OK, maybe more a desire than a need) to set my browser window(s) on macOS to a specific size, like 1920x1080. I initially started looking at one of the many macOS window managers, but after reading lots of reviews and descriptions and still being unclear if any of these products did what I wanted, I decided to step back to using AppleScript to accomplish what I was seeking. In this post, I’ll share the solution (and the articles that helped me arrive at the solution).

My first stop was this blog post by Ethan Banks. I tried replicating the AppleScript he used, but couldn’t get it to work. I’m still running macOS 10.14 “Mojave,” so perhaps his code was specific to macOS 10.15 “Catalina.” I moved on, never realizing there was another section to his post that had the information I needed (and would eventually find). Let that be a lesson to be sure to read the entire post next time.

Moving on, I arrived at this post. OK, this used a different mechanism than Ethan’s post. I tried it, and it sort of worked, but it didn’t create the window geometry I was expecting. As I’ll later learn, it was just due to an incomplete understanding on my part of how the set bounds command works in AppleScript.

Finally, I found this article that shared how to use AppleScript in conjunction with Automator to create a macOS Service to resize the current window of the active application. After trying it for a while, and not getting the results I wanted, I started digging again to see what it was that I was doing wrong.

I found the answer to what I was doing wrong here. The parameters to the set bounds command had been illustrated as x-position, y-position, width, height, but they should be more accurately described as starting-x-position, starting-y-position, ending-x-position, ending-y-position. My mistake was that I was providing the desired window size (like 1920x1080 or 1280x720) as the last two parameters, when what I needed to be providing was the desired window size plus the starting X and Y position, respectively. So, if the window was placed 300 pixels away from the left edge and I wanted to the window to be 1920 pixels wide, then the third parameter needed to be 2220 (300 + 1920 = 2220). Ah! I had seen one of the examples doing this but didn’t understand why, so I hadn’t included the portion of the code. Once I fixed my code accordingly, then it started working exactly as expected.

(This piece of missing information, by the way, is also found at the bottom of Ethan’s post—the one I started with. Go figure!)

Nothing earth-shattering here, I know, but I wanted to share it nevertheless just in case it would benefit others. Contact me on Twitter if you have any questions.

Technology Short Take 133

Welcome to Technology Short Take #133! This time around, I have a collection of links featuring the new Raspberry Pi 400, some macOS security-related articles, information on AWS Nitro Enclaves and gVisor, and a few other topics. Enjoy!

Networking

Servers/Hardware

  • The Raspberry Pi 400 is a neat offering. See this post for more details.

Security

Cloud Computing/Cloud Management

Operating Systems/Applications

Programming

  • I haven’t (yet) had the chance to walk through it, but this tutorial looks to be very promising, providing exposure to both AWS Lambda and using Go. It’s definitely on my list!

Virtualization

That’s all I have for now—hopefully you found something useful and informative! Feel free to hit me on Twitter if you have any feedback or suggestions for improvement. I’m also open to items that I should consider for inclusion in a future Tech Short Take.

Recent Posts

Technology Short Take 132

Welcome to Technology Short Take #132! My list of links and articles from around the web seems to be a bit heavy on security-related topics this time. Still, there’s a decent collection of networking, cloud computing, and virtualization articles as well as a smattering of other topics for you to peruse. I hope you find something useful!

Read more...

Considerations for using IaC with Cluster API

In other posts on this site, I’ve talked about both infrastructure-as-code (see my posts on Terraform or my posts on Pulumi) and somewhat separately I’ve talked about Cluster API (see my posts on Cluster API). And while I’ve discussed the idea of using existing AWS infrastructure with Cluster API, in this post I wanted to try to think about how these two technologies play together, and provide some considerations for using them together.

Read more...

Technology Short Take 131

Welcome to Technology Short Take #131! I’m back with another collection of articles on various data center technologies. This time around the content is a tad heavy on the security side, but I’ve still managed to pull in articles on networking, cloud computing, applications, and some programming-related content. Here’s hoping you find something useful here!

Read more...

Updating AWS Credentials in Cluster API

I’ve written a bit here and there about Cluster API (aka CAPI), mostly focusing on the Cluster API Provider for AWS (CAPA). If you’re not yet familiar with CAPI, have a look at my CAPI introduction or check the Introduction section of the CAPI site. Because CAPI interacts directly with infrastructure providers, it typically has to have some way of authenticating to those infrastructure providers. The AWS provider for Cluster API is no exception. In this post, I’ll show how to update the AWS credentials used by CAPA.

Read more...

Behavior Changes in clusterawsadm 0.5.5

Late last week I needed to test some Kubernetes functionality, so I thought I’d spin up a test cluster really quick using Cluster API (CAPI). As often happens with fast-moving projects like Kubernetes and CAPI, my existing CAPI environment had gotten a little out of date. So I updated my environment, and along the way picked up an important change in the default behavior of the clusterawsadm tool used by the Cluster API Provider for AWS (CAPA). In this post, I’ll share more information on this change in default behavior and the impacts of that change.

Read more...

Technology Short Take 130

Welcome to Technology Short Take #130! I’ve had this blog post sitting in my Drafts folder waiting to be published for almost a month, and I kept forgetting to actually make it live. Sorry! So, here it is—better late than never, right?

Read more...

Creating an AWS ELB using Pulumi and Go

In case you hadn’t noticed, I’ve been on a bit of a kick with Pulumi and Go recently. There are two reasons for this. First, I have a number of “learning projects” (things that I decide I’d like to try or test) that would benefit greatly from the use of infrastructure as code. Second, I’ve been working on getting more familiar with Go. The idea of combining both those reasons by using Pulumi with Go seemed natural. Unfortunately, examples of using Pulumi with Go seem to be more limited than examples of using Pulumi with other languages, so in this post I’d like to share how to create an AWS ELB using Pulumi and Go.

Read more...

Review: Anker PowerExpand Elite Thunderbolt 3 Dock

Over the last couple of weeks or so, I’ve been using my 2017 MacBook Pro (running macOS “Mojave” 10.14.6) more frequently as my daily driver/primary workstation. Along with it, I’ve been using the Anker PowerExpand Elite 13-in-1 Thunderbolt 3 Dock. In this post, I’d like to share my experience with this dock and provide a quick review of the Anker PowerExpand Elite.

Read more...

Technology Short Take 129

Welcome to Technology Short Take #129, where I’ve collected a bunch of links and references to technology-centric resources around the Internet. This collection is (mostly) data center- and cloud-focused, and hopefully I’ve managed to curate a list that has some useful information for readers. Sorry this got published so late; it was supposed to go live this morning!

Read more...

Working Around Docker Desktop's Outdated Kubernetes Version

As of the time that I published this blog post in early July 2020, Docker Desktop for macOS was at version 2.2.0.4 (for the “stable” channel). That version includes a relatively recent version of the Docker engine (19.03.8, compared to 19.03.12 on my Fedora 31 box), but a quite outdated version of Kubernetes (1.15.5, which isn’t supported by upstream). Now, this may not be a problem for users who only use Kubernetes via Docker Desktop. For me, however, the old version of Kubernetes—specifically the old version of kubectl—causes problems. Here’s how I worked around the old version that Docker Desktop supplies. (Also, see the update at the bottom for some additional details that emerged after this post was originally published.)

Read more...

Creating an AWS Security Group using Pulumi and Go

In this post, I’m going to share some examples of how to create an AWS security group using Pulumi and Go. I’m sharing these examples because—as of this writing—the Pulumi site does not provide any examples on how this is done using Go. There are examples for the other languages supported by Pulumi, but not for Go. The syntax is, to me at least, somewhat counterintuitive, although I freely admit this could be due to the fact that I am still pretty new to Go and its syntax.

Read more...

Adopting the Default Route Table of an AWS VPC using Pulumi and Go

Up until now, when I used Pulumi to create infrastructure on AWS, my code would create all-new infrastructure: a new VPC, new subnets, new route tables, new Internet gateway, etc. One thing bothered me, though: when I created a new VPC, that new VPC automatically came with a default route table. My code, however, would create a new route table and then explicitly associate the subnets with that new route table. This seemed less than ideal. (What can I say? I’m a stickler for details.) While building a Go-based replacement for my existing TypeScript code, I found a way to resolve this duplication of resources. In this post, I’ll show you how to “adopt” the default route table of an AWS VPC so that you can manage it in your Pulumi code.

Read more...

Getting AWS Availability Zones using Pulumi and Go

I’ve written several different articles on Pulumi (take a look at all articles tagged “Pulumi”), the infrastructure-as-code tool that allows users to define their infrastructure using a general-purpose programming language instead of a domain-specific language (DSL). Thus far, my work with Pulumi has leveraged TypeScript, but moving forward I’m going to start sharing more Pulumi code written using Go. In this post, I’ll share how to use Pulumi and Go to get a list of Availability Zones (AZs) from a particular region in AWS.

Read more...

Fixes for Some Vagrant Issues on Fedora

Yesterday I needed to perform some testing of an updated version of some software that I use. (I was conducting the testing because this upgrade contained some breaking changes, and needed to understand how to mitigate the breaking changes.) So, I broke out Vagrant (with the Libvirt provider) on my Fedora laptop—and promptly ran into a couple issues. Fortunately, these issues were relatively easy to work around, but since the workarounds were non-intuitive I wanted to share them here for the benefit of others.

Read more...

Technology Short Take 128

Welcome to Technology Short Take #128! It looks like I’m settling into a roughly monthly cadence with the Technology Short Takes. This time around, I’ve got a (hopefully) interesting collection of links. The collection seems a tad heavier than normal in the hardware and security sections, probably due to new exploits discovered in Intel’s speculative execution functionality. In any case, here’s what I’ve gathered for you. Enjoy!

Read more...

Older Posts

Find more posts by browsing the post categories, content tags, or site archives pages. Thanks for visiting!