TechOps Examples

Hey — It's Govardhana MK 👋

Welcome to another technical edition.

Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.

Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities, and articles.

You're spending 40 hours a week writing code that AI could do in 10.

While you're grinding through pull requests, 350k+ engineers at OpenAI, Google & Meta are using AI to ship faster. How?

The Code newsletter teaches them exactly which AI tools to use and how to use them. Here's what you get:

Sign up now and get instant access to the Ultimate Claude code guide to ship 5X faster.

Looking to promote your company, product, service, or event to 50,000+ Cloud Native Professionals? Let's work together. Advertise With Us

A logging agent that only runs on three out of ten nodes in a cluster is not actually collecting cluster wide logs, it is missing seven nodes worth of container output entirely. The same problem hits a node level metrics exporter monitoring disk and memory pressure, or a CNI networking plugin that has to be present on every node before pods on that node can even get an IP address. A regular Deployment cannot guarantee this kind of placement, it only guarantees a total replica count, and the scheduler is free to pack all of them onto a handful of nodes if that is where capacity happens to be available.

A Deployment deals with "how many copies of this pod should exist," not "where should they run." It delegates the actual replica count enforcement to a ReplicaSet, which in turn asks the scheduler to place each pod, and the scheduler's only concern is finding a node with enough free capacity, with no built in notion of spreading pods one-per-node unless something explicitly tells it to.

A Deployment creates a ReplicaSet configured for N replicas. The scheduler places those N pods wherever capacity exists, in this case one pod lands on Node A and two land on Node B. Nothing about this arrangement is wrong from the ReplicaSet's perspective, three pods exist, the desired count is satisfied. But if this were a log shipper meant to collect output from every container on every node, Node A would have coverage while any node beyond A and B would have none at all.

This gap is exactly why running a node level agent as a Deployment, even with a high replica count and pod anti affinity rules bolted on, is fragile. Anti affinity can discourage two replicas from landing on the same node, but it says nothing about ensuring every node gets exactly one, and it breaks the moment the cluster scales past the replica count or a new node joins after the Deployment already reached its target.

A DaemonSet solves this differently by inverting the question entirely. Instead of specifying a replica count and letting the scheduler decide placement, a DaemonSet specifies a pod template and Kubernetes guarantees exactly one copy of it on every node that matches the DaemonSet's node selector, with the count following the node count automatically rather than the other way around.

The API server holds the DaemonSet object as the source of truth. Every node in the cluster, Node A, Node B, and Node C, receives exactly one DaemonSet pod, shown here running alongside each node's regular application pods. A log shipper deployed this way reads every container's stdout on every node it runs on, a node exporter reads every node's actual disk and memory metrics, and a CNI plugin like Calico or Cilium gets installed on every node before any application pod on that node is allowed to schedule.

Kubernetes ships a handful of components as DaemonSets by default on most managed clusters specifically because they need this exact guarantee, kube-proxy maintaining iptables or IPVS rules on every node, the CNI plugin's node agent, and typically the log and metrics collectors a cluster operator installs, Fluent Bit, Datadog's node agent, or Prometheus's node-exporter.

🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)