TechOps Examples

Hey — It's Govardhana MK 👋

Welcome to another technical edition.

Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.

Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities and articles.

KodeKloud gives you 1,280+ hands-on labs where you provision Kubernetes clusters, write Terraform configs, build CI/CD pipelines, configure Linux systems, containerize apps with Docker, automate with Ansible, and manage Git workflows. 

78+ playgrounds let you experiment freely in sandbox AWS environments, Kubernetes clusters, and CI/CD systems without risk. 

190+ courses across DevOps, Cloud, and AI pair theory with hands-on labs at every step. 

KodeKloud Engineer and 100 Day Challenges provide real-world job scenarios with automated grading that confirms your solutions work.

Stuck? The 55,000+ member Discord community connects you with peers and instructors ready to help. 

Every lab runs in a live environment. You deploy, you troubleshoot, you learn. No videos without context. No simulations. The kind of practice that actually builds confidence because you've done real work, not watched someone else do it.

Start Learning For Free

If you’re not a subscriber, here’s what you missed last week.

How Kubernetes Ingress Works

How to Optimize Kubernetes Workloads

To receive all the full articles and support TechOps Examples, consider subscribing:

One-time 25% OFF on all annual plans of memberships. Closes Soon.

gitdiagram - Turn any public or private GitHub repository into an interactive architecture diagram in seconds.

Click a component to open its real file or directory on GitHub, see the explanation arrive while the graph is planned.

A new node joins a cluster, kubectl get nodes shows it, but the STATUS column reads NotReady and stays that way for minutes, sometimes indefinitely. No pods schedule onto it because the scheduler filters out anything that is not Ready. This is one of the more common operational surprises in a managed Kubernetes environment, particularly on EKS, GKE, or AKS clusters running the external cloud provider model, and the cause almost always traces back to a handoff between three components that has to complete in order before a node is usable.

When kubelet starts on a new node and registers with the API server, it does not simply announce itself as available. If the cluster runs with --cloud-provider=external, meaning cloud integration has been split out of the core kubelet and kube-controller-manager into a separate binary, kubelet knows the node is not fully provisioned yet. It has no cloud metadata: no confirmed internal or external IP address, no instance type label, no availability zone. So kubelet registers the Node object and, in the same request, applies a taint to itself.

The taint exists specifically to prevent a race condition. Without it, the scheduler could place a pod on a node before cloud-controller-manager has attached the correct zone label, which would break zone-aware scheduling, or before the node's addresses are populated, which would break anything relying on kubectl get nodes -o wide or the node's reported IP for networking decisions. The taint is the mechanism that holds scheduling back until that data exists.

Node readiness also depends on a second signal that has nothing to do with cloud-controller-manager: kubelet's own health checks. Kubelet reports the Ready condition on the Node object based on whether its internal runtime checks pass, PLEG (the pod lifecycle event generator) is healthy, disk and memory pressure are within thresholds, and a CNI plugin has successfully initialized. A node can have its cloud taint removed and still sit at NotReady if the CNI daemonset has not started pods on it yet, because kubelet independently applies node.kubernetes.io/network-unavailable until networking is confirmed working.

Most stuck-NotReady nodes fall into one of four categories, and diagnosing which one applies starts with kubectl describe node <name> and checking the Taints and Conditions sections directly.

The uninitialized taint never gets removed. This is a cloud-controller-manager problem, not a kubelet problem. Check the CCM pod logs first. The most frequent root cause is an IAM role, service account, or instance profile that lacks permission to describe the instance, ec2:DescribeInstances on AWS being the common missing permission. A second frequent cause on AWS specifically is IMDSv2 with a hop limit set to 1, which blocks CCM running in a pod, an extra network hop from the host, from reaching the instance metadata service to resolve the provider ID.

CCM never sees the node at all. If cloud-controller-manager is not running, crash-looping, or watching the wrong node selector, the create event mentioned above has no subscriber to act on it. kubectl get pods -n kube-system -l k8s-app=aws-cloud-controller-manager or the equivalent for the provider in use confirms whether it is even scheduled and healthy.

Kubelet and the control plane disagree on cloud provider mode. If kubelet was started with --cloud-provider=external but the cluster's kube-controller-manager still runs the legacy in-tree cloud provider, or the reverse, the taint gets applied but nothing is configured to remove it, since the component expected to act on it either does not exist or is not looking for that node. Checking the kubelet flags on the node itself against the cluster's controller-manager configuration resolves this quickly.

The node is cloud initialized but the CNI never started. This shows up as the cloud taint being gone while the node still reports NotReady, with NetworkUnavailable still True in the conditions. The fix here is checking the CNI daemonset, Calico, Cilium, or the AWS VPC CNI, is actually scheduled on that node and its pods are running, since a daemonset that failed to pull its image or hit a resource limit will silently leave every new node without networking.

🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)

Looking to promote your company, product, service, or event to 50,000+ DevOps and Cloud Professionals? Let's work together.