TechOps Examples
Hey — It's Govardhana MK 👋
Welcome to another technical edition.
Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.
Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities, and articles.
Top engineers at Anthropic and OpenAI say AI now writes 100% of their code.
If you're not using AI, you're spending 40 hours doing what they do in 4.
These 100+ Claude Code hacks fix that and help you ship 10x faster.
Sign up for The Code and get:
100+ Claude Code hacks used by top engineers — free
The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day
Looking to promote your company, product, service, or event to 50,000+ Cloud Native Professionals? Let's work together. Advertise With Us
🧠 DEEP DIVE USE CASE
How to Troubleshoot Unhealthy Kubernetes DaemonSets
Many DevOps engineers ask this all the time. Deployments create Pods. DaemonSets also create Pods. So what is really different here?
If both spin up containers inside the cluster, why not just use one instead of the other when something starts going wrong?
When you create a Deployment, you define a Pod template and how many replicas you want running.
The Deployment creates a ReplicaSet, and that ReplicaSet makes sure the total number of Pods you asked for is always maintained. If one Pod dies, another is created. If a node disappears, the Pods are rescheduled elsewhere.
The important part here is this. A Deployment thinks in terms of total replica count across the cluster. It ensures the number stays consistent, but it does not guarantee how those Pods are distributed across nodes.
What is Kubernetes DaemonSet?
A Kubernetes DaemonSet is a workload controller that ensures a specific Pod runs on every eligible node in a cluster.
Instead of asking for a replica count like a Deployment, you only define the Pod template. Kubernetes then automatically runs exactly one copy of that Pod on each node that matches the criteria.
If a new node is added, the DaemonSet creates a Pod on it. If a node is removed, the corresponding Pod disappears with it.
The focus here is not total replicas across the cluster. The focus is one Pod per node.
Now put both mental models side by side.
With a Deployment, you control a number. Three replicas means three Pods somewhere in the cluster. Kubernetes does not care which node runs them, only that the total count remains correct.
With a DaemonSet, you control coverage. Every eligible node must have one Pod. The number of Pods is therefore derived from the number of nodes.
In a Deployment:
A node can have zero Pods.
A node can have multiple Pods.
In a DaemonSet:
A node should never have zero Pods if it is eligible.
A node should never have more than one Pod from that DaemonSet.
When a Deployment is unhealthy, you usually think in terms of missing replicas.
When a DaemonSet is unhealthy, you think in terms of missing nodes, extra Pods, or Pods that are not aligned one-to-one with nodes.
With this basic understanding of how Deployments and DaemonSets behave, let us now look behind the scenes at how the DaemonSet controller works and how to troubleshoot unhealthy DaemonSets.
🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)
Behind the scenes of a DaemonSet
The DaemonSet controller runs continuously in the background using a reconciliation loop. Its behavior follows a clear flow.
It watches for changes in three resources: DaemonSets, Pods, Nodes
When any change occurs, the reconciliation process starts.
The controller builds two lists
All eligible Nodes
All Pods that match the DaemonSet selector
It compares these lists node by node and evaluates the state.
If a Node has no matching Pod, a new Pod is created from the DaemonSet template and assigned to that Node.
If a Node has more than one matching Pod, the extra Pod is deleted.
If a Pod exists for a Node that no longer exists, that Pod is removed.
If a Node has exactly one matching Pod, the controller takes no action.
Troubleshooting Unhealthy Kubernetes DaemonSets
Start with one focused question. For this DaemonSet, do all eligible nodes have the expected Pod, and is that Pod healthy?
1) Check the DaemonSet status first
kubectl get ds <daemonset-name>Look at these fields:
DESIRED tells how many nodes are eligible for this DaemonSet
CURRENT tells how many nodes currently have a DaemonSet Pod
READY tells how many of those Pods are ready
Now interpret it quickly:
DESIRED > CURRENT means some eligible nodes have no Pod scheduled
CURRENT > DESIRED means extra Pods exist and node mapping is drifting
READY < DESIRED means Pods exist but some are unhealthy
2) List the Pods and map them to nodes
Use the DaemonSet selector.
kubectl get ds <daemonset-name> -o wide
kubectl get pods -l <label-selector> -o wideYou are looking for:
Eligible nodes that do not show any Pod for this DaemonSet
Pods stuck in Pending, CrashLoopBackOff, ImagePullBackOff, or Terminating
More than one Pod for the same DaemonSet on the same node, which can happen during updates or misconfigurations
3) Pick one problematic Pod and inspect why
kubectl describe pod <pod-name>
kubectl logs <pod-name>From describe, focus on:
Events related to scheduling, Image pull failures, Probe failures, Resource related messages
From logs, focus on:
Startup failures, Config errors, Permission issues when touching host paths, host network, or devices
4) Validate node eligibility and node health
For the node where the Pod is missing or failing:
kubectl describe node <node-name>Check: Node Ready status, Taints that require tolerations, Labels expected by nodeSelector or affinity, Resource pressure conditions like memory or disk pressure
I hope this was an insightful edition. See you in the next edition on Tuesday.
Take care and have a nice weekend.
