TechOps Examples
Hey — It's Govardhana MK 👋
Welcome to another technical edition.
Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.
Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities and articles.
Top engineers at Anthropic and OpenAI say AI now writes 100% of their code.
If you're not using AI, you're spending 40 hours doing what they do in 4.
These 100+ Claude Code hacks fix that and help you ship 10x faster.
Sign up for The Code and get:
100+ Claude Code hacks used by top engineers — free
The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day
🛠 TOOL OF THE DAY
Inspektor Gadget - Open source eBPF debugging and data collection tool for Kubernetes and Linux.
Build and package eBPF programs into OCI images called Gadgets
Collect and export data to observability tools with a simple command
🧠 USE CASE
How to Fix Kubernetes Node Disk Pressure
Imagine you deploy an application, but after a few days, it starts throwing warnings like this:
Warning NodePressure [timestamp] kubelet Node [node-name] status is now: NodeHasDiskPressureYour application slows, pods get evicted, and new ones fail to schedule. This common error in Kubernetes is known as Node Disk Pressure, and if left unchecked, it can severely impact application performance.
What is Kubernetes Node Disk Pressure?
Node Disk Pressure occurs when a node’s filesystem is under strain due to low available disk space or inodes.
Kubernetes automatically detects these low resource conditions and sets a NodeHasDiskPressure status.
This status signals that the node has insufficient disk resources for further scheduling, evicting non-critical pods to prevent critical system disruptions.
How to Check Kubernetes Node Disk Pressure
To verify if your nodes are experiencing Disk Pressure, you can use:
kubectl describe node [node-name]Look for any nodes with the condition type DiskPressure and status True. In the output, focus on the Conditions section. Here’s an example where techops-node2 is experiencing disk pressure:
techops-node2 Ready worker 14d v1.28.1 DiskPressure=True,MemoryPressure=False,PIDPressure=False,Ready=True
Additionally, use this command to monitor disk usage:
kubectl top nodesThis shows overall CPU, memory, and disk usage for each node, helping you pinpoint where Disk Pressure is affecting your nodes.
Why Should You Care About Node Disk Pressure?
Ignoring Disk Pressure can lead to various issues:
Pod Eviction: Kubernetes evicts lower-priority pods to free up disk space, which can cause disruptions in non-critical workloads.
Scheduling Failures: New workloads may not deploy if nodes are in a Disk Pressure state.
Performance Degradation: Insufficient disk space impacts node performance and can lead to application latency.
How to Fix Kubernetes Node Disk Pressure
Here are some strategies to address Disk Pressure:
Clean Up Disk Space: Clear out unused images and containers, which can take up significant space.
Increase Node Disk Size: If your nodes are in a cloud environment, consider resizing disks. In AWS, for instance - Increase the EBS volume size.
Move Logs and Data to Separate Disks: If your node frequently generates large logs, consider mounting a separate disk for log storage to keep system space free.
Implement Resource Quotas and Limits:
apiVersion: v1
kind: ResourceQuota
metadata:
name: storage-quota
namespace: [namespace]
spec:
hard:
requests.storage: 10GiMonitor with Alerts:
Use Prometheus or another monitoring tool to set up alerts when disk usage exceeds a threshold. This proactive approach helps you intervene before Disk Pressure arises.
Hope this helps in tackling one of the common Kubernetes challenges.
You may even like:
Looking to promote your company, product, service, or event to 49,000+ TechOps Professionals? Let's work together.
