TechOps Examples

Hey — It's Govardhana MK 👋

Welcome to another technical edition.

Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.

Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities, and articles.

👋 👋 A big thank you to today's sponsor LINDY

Lindy sorts your inbox, drafts replies in your voice, and preps you before every meeting over iMessage. 400,000+ professionals use it. Try it free.

Try Lindy Free

Powered by: Jobsurface.com

Browse 669 Worldwide Jobs Here →

Looking to promote your company, product, service, or event to 48,000+ Cloud Native Professionals? Let's work together. Advertise With Us

In the early days of cloud infrastructure, engineers coined a useful analogy: cattle are interchangeable, if one gets sick, you replace it without caring which one it was. Pets are individuals, each one has a name, a history, and data that matters.

Deployments are cattle. Pods are identical, interchangeable, and replaceable. Your web servers, API services, and worker processes work this way.

StatefulSets are pets. Each pod has a stable identity, a stable hostname, and its own persistent storage that follows it across restarts. Your databases, message queues, and distributed consensus systems need this.

A Deployment generates pod names by appending a random hash suffix to the ReplicaSet name. web-app-7d9f4b-xk2qp, web-app-7d9f4b-m8rt, web-app-7d9f4b-pz7wv. Every time a pod restarts, it gets a completely new name. The old name disappears. Nothing in the system depends on the name, so this is fine. Your application never needs to know which specific pod instance it is running on.

A StatefulSet generates pod names using a deterministic ordinal index. mysql-0, mysql-1, mysql-2. These names are permanent. When mysql-0 is deleted and recreated, it comes back as mysql-0. Not mysql-4 or mysql-0-new. The exact same name, bound to the exact same PersistentVolumeClaim, with the exact same DNS hostname. This stability is not cosmetic. It is the mechanism that allows mysql-1 to find the primary by dialing mysql-0.mysql-headless.default.svc.cluster.local regardless of how many times the primary pod has restarted.

The lock icon in the StatefulSet column of the diagram represents exactly this: the pod name is locked. It cannot change. Everything that depends on the pod, replication configuration, peer discovery, client connection strings, depends on that name being permanent.

The lifecycle differences between the two workload types are most visible during scaling and upgrades.

When a Deployment scales from 0 to 3 replicas, all three pods start simultaneously. Kubernetes hands the scheduling decisions to the kube-scheduler in parallel and the pods race to become Ready. The order in which they become Ready is irrelevant because the pods are interchangeable.

When a StatefulSet scales from 0 to 3 replicas, app-0 must start first and pass its readiness probe before app-1 is allowed to start. app-1 must pass its readiness probe before app-2 is allowed to start. This sequential enforcement exists because in most distributed stateful systems, the first pod bootstraps the cluster and subsequent pods join it. If they all start simultaneously, they attempt to form a quorum or connect to a primary that does not exist yet and the initialization fails.

The rolling update behavior is equally different. A Deployment rolling update creates a new ReplicaSet and replaces pods in parallel based on maxUnavailable and maxSurge settings. Old and new ReplicaSets run concurrently during the transition. The order of replacement has no significance.

A StatefulSet rolling update proceeds in reverse ordinal order: app-2 is updated first, then app-1, then app-0 last. This ordering is deliberate. In a primary-replica database cluster, the primary is always ordinal 0. Updating replicas first means the primary continues serving writes throughout the update process. The primary is only updated after all replicas have been successfully updated and have rejoined replication. This minimizes the window during which a failover could expose an older version of the database. Each pod must become Ready before the previous pod update begins, making the process serial but safe.

🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)