TechOps Examples

Hey — It's Govardhana MK 👋

Welcome to another technical edition.

Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.

Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities, and articles.

Stop guessing if your agents are failing and start monitoring them for failures. We surface agent failures by looking for behavioural anomalies your your agent traces and classify failures with RCA and remediation steps. Works on your existing observability stack, no new SDK required.

Looking to promote your company, product, service, or event to 50,000+ Cloud Native Professionals? Let's work together. Advertise With Us

A production incident that keeps showing up across on-call rotations looks like this: a pod shows Running in kubectl get pods, the container process is alive, yet the application inside stopped responding minutes ago.

Requests routed to it time out. Nothing in the pod status hints at the problem because Kubernetes, by default, only tracks whether the container process exists, not whether the application behind it is doing useful work. Before probes existed as a first-class concept, teams patched this gap with external monitoring agents polling endpoints from outside the cluster, then manually killing and recreating pods when checks failed. That approach worked, but it added latency between failure and recovery, and it kept healing logic outside the system that was already responsible for scheduling and restarting workloads.

Kubernetes closes this gap with three probe types, each answering a different question about a container's state:

Each probe is configured independently inside a container spec, runs on its own schedule, and triggers a different response when it fails. Confusing one for another is a common source of self inflicted outages, so it helps to walk through what each one actually does under the hood.

A liveness probe let us: is this container still functioning correctly, or has it entered a state where it needs a restart? The kubelet on each node runs this probe against every container that defines one, using the exact schedule specified in the pod spec.

Picture a payment processing container that hits a deadlock between two internal goroutines while writing to a connection pool. The process itself keeps running, so Kubernetes has no reason to touch it. But the application stopped serving requests permanently. This is exactly the scenario a liveness probe catches. If the check fails past the configured failure threshold, the kubelet kills the container and restarts it according to the pod's restartPolicy.

The YAML configuration for a liveness probe supports three check mechanisms: an HTTP GET request, a TCP socket connection, or an arbitrary command executed inside the container.

This example uses an exec probe that runs cat /var/run/techops-app/healthy inside the container. If that file is missing or the command exits non-zero, the probe counts as failed. initialDelaySeconds: 10 tells the kubelet to wait 10 seconds after container start before running the first check, giving the application time to boot. periodSeconds: 5 sets the check interval afterward.

🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)