<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ramesh | The Platform Engineer on Medium]]></title>
        <description><![CDATA[Stories by Ramesh | The Platform Engineer on Medium]]></description>
        <link>https://medium.com/@rameshavutu?source=rss-c3bcf684ed87------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QrR_ZJAZebyf_9w6.jpg</url>
            <title>Stories by Ramesh | The Platform Engineer on Medium</title>
            <link>https://medium.com/@rameshavutu?source=rss-c3bcf684ed87------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:23:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rameshavutu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes 1.37: 10 Changes Every Kubernetes Engineer Should Know]]></title>
            <link>https://medium.com/@rameshavutu/kubernetes-1-37-10-changes-every-kubernetes-engineer-should-know-7482420a86fb?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/7482420a86fb</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 08:25:45 GMT</pubDate>
            <atom:updated>2026-09-01T08:25:45.163Z</atom:updated>
            <content:encoded><![CDATA[<h4>From KYAML to cgroups v1 enforcement, what just shipped and what you need to do right now</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CcwkFPMNRJk-9uQ3QJTJzQ.png" /><figcaption><a href="https://kubernetes.io/blog/2026/08/26/kubernetes-v1-37-release/">Kubernetes v1.37: Garhwal</a></figcaption></figure><p>Kubernetes 1.37 shipped on <strong>August 26, 2026</strong>. <br>Sixty-seven enhancements. <br>Sixteen graduating to stable.</p><p>Two deprecations that will break clusters if you ignore them. And one SELinux change that has already started silently failing pods in early-adopter clusters.</p><p>The release blog is 4,000 words. You don’t have time for that.</p><p>This is the version for engineers who need to know what matters, why it matters, and what to do before it bites them in production.</p><p>Ten changes. Real code. No fluff.</p><p>Let’s go.</p><h3>1. KYAML — GA: Better Kubernetes YAML Workflow</h3><p>KYAML is now locked to enabled, the feature gate is gone, and it’s on by default everywhere.</p><p>KYAML output is now available across <strong>all </strong><strong>kubectl commands</strong> that support the --output flag. That means structured, consistently typed YAML output from operations that previously returned inconsistent or loosely typed formats.</p><p>For engineers writing controllers, automation scripts, or GitOps pipelines that parse kubectl output, this matters. No more wrestling with inconsistent type coercions between YAML and JSON.</p><pre># With KYAML GA — typed output, schema-conformant<br>kubectl get pod my-pod -o kyaml<br># KYAML ensures types conform to the Kubernetes API schema<br># Integers are integers, not strings. No more patch surprises.<br><br><br># KYAML output now available across all --output flag commands<br>kubectl get deployment my-app -o kyaml<br>kubectl get service my-svc -o kyaml<br>kubectl get configmap my-config -o kyaml</pre><h3>2. Metrics API GA — metrics.k8s.io Finally Stable</h3><p>The metrics.k8s.io API, the one that powers kubectl top, HPA CPU/Memory autoscaling, and most monitoring stacks — has <strong>graduated to General Availability</strong> after years in Beta.</p><p>GA means API stability guarantees, SLO-level reliability expectations, and tools depending on metrics can drop version detection hacks.</p><pre># Verify your metrics-server is responding<br>kubectl top nodes<br>kubectl top pods --all-namespaces<br><br># The API is now stable — query it directly<br>kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq &#39;.items[0].usage&#39;<br><br># Check API service status<br>kubectl get apiservice v1beta1.metrics.k8s.io -o yaml</pre><h3>3. Node Declared Features — Smarter Version-Skew Handling</h3><p>Running mixed-version clusters during upgrades is one of the hardest operational problems in Kubernetes. Node Declared Features (now Stable) fixes this elegantly.</p><p>Nodes can now <strong>dynamically advertise their feature-gated capabilities</strong> to the control plane. The scheduler and admission controllers make intelligent scheduling decisions placing pods only on nodes that support required features without manual taint/label management.</p><pre># Pod spec - request a feature-gated capability<br># Scheduler will only place this on compatible nodes automatically<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: gpu-workload<br>spec:<br>  # No manual taints/labels needed anymore for version-skew<br>  # Node Declared Features handles compatibility automatically<br>  containers:<br>  - name: gpu-job<br>    image: nvidia/cuda:12.0-base<br>    resources:<br>      claims:<br>      - name: gpu-resource<br>  resourceClaims:<br>  - name: gpu-resource<br>    spec:<br>      devices:<br>        requests:<br>        - name: gpu<br>          deviceClassName: gpu.example.com</pre><p><strong>Action required:</strong> Stop manually managing compatibility taints during upgrades. Audit your upgrade runbooks and remove the manual taint workarounds.</p><h3>4. DRA Improvements — Hardware Allocation Grows Up</h3><p>Three DRA capabilities graduated to GA in 1.37:</p><ul><li><strong>Device-level taints and tolerations</strong> — hardware can be marked unavailable/degraded without removing it from the pool</li><li><strong>Resource Claim Status with standardized network interface data</strong> — structured device status</li><li><strong>DRA extended resources</strong> — bridge between classic extended resources and the new DRA model</li></ul><p>Plus four new Alpha capabilities: optional node preparation, device compatibility groups, standardized numaNode attribute, and derived attributes.</p><pre># Device-level taints and tolerations - now GA<br># Mark a degraded GPU without removing it from the pool<br>apiVersion: resource.k8s.io/v1beta1<br>kind: ResourceSlice<br>metadata:<br>  name: node-01-gpus<br>spec:<br>  nodeName: worker-node-01<br>  driver: gpu.example.com<br>  pool:<br>    name: gpu-pool<br>    generation: 1<br>  devices:<br>  - name: gpu-0<br>    basic:<br>      attributes:<br>        memory: 80Gi<br>      taints:<br>      - key: &quot;gpu.example.com/degraded&quot;<br>        effect: &quot;NoSchedule&quot;<br>        value: &quot;ECC-errors-detected&quot;<br>  - name: gpu-1<br>    basic:<br>      attributes:<br>        memory: 80Gi<br>      # No taint - healthy GPU, available for scheduling<br><br>---<br># Pod tolerates degraded GPUs if needed (e.g., batch jobs)<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: batch-gpu-job<br>spec:<br>  containers:<br>  - name: trainer<br>    image: pytorch/pytorch:latest<br>    resources:<br>      claims:<br>      - name: gpu-claim<br>  resourceClaims:<br>  - name: gpu-claim<br>    spec:<br>      devices:<br>        requests:<br>        - name: gpu<br>          deviceClassName: gpu.example.com<br>          tolerations:<br>          - key: &quot;gpu.example.com/degraded&quot;<br>            operator: &quot;Exists&quot;<br>            effect: &quot;NoSchedule&quot;</pre><p><strong>Action required:</strong> If running GPU or specialized hardware workloads, evaluate migrating from extended resources to DRA. GA graduation means production-ready.</p><h3>5. HPA Tolerance Per-Workload — Precision Autoscaling</h3><p>Per-workload HPA tolerance lets you override the cluster-wide default 10% tolerance individually per HPA. A latency-sensitive API and a batch job now scale differently, as they should.</p><p>HPAScaleToZero also moved to Beta and is now enabled by default — scale down to zero replicas when spec.minReplicas: 0</p><pre># Latency-sensitive service - tight scaleUp, conservative scaleDown<br>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: latency-sensitive-api<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: api-server<br>  minReplicas: 3<br>  maxReplicas: 50<br>  metrics:<br>  - type: Resource<br>    resource:<br>      name: cpu<br>      target:<br>        type: Utilization<br>        averageUtilization: 70<br>  behavior:<br>    scaleUp:<br>      tolerance: 0.02    # 2% - very responsive scale-up<br>    scaleDown:<br>      tolerance: 0.15    # 15% - conservative scale-down<br>      stabilizationWindowSeconds: 300<br><br>---<br># Event-driven workload - scale to zero when idle<br>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: event-consumer<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: queue-processor<br>  minReplicas: 0    # Can now scale to zero (HPAScaleToZero Beta/default)<br>  maxReplicas: 20<br>  metrics:<br>  - type: External<br>    external:<br>      metric:<br>        name: queue_depth<br>        selector:<br>          matchLabels:<br>            queue: &quot;orders&quot;<br>      target:<br>        type: AverageValue<br>        averageValue: &quot;10&quot;</pre><h3>6. SELinuxMount GA — The Silent Pod Killer</h3><p>This is the one that will bite clusters without warning.</p><p>SELinuxMount is now Stable and <strong>enabled by default</strong>. The kubelet now uses the Linux -o context mount option to apply SELinux labels to an entire volume at once — a massive startup performance improvement for large volumes.</p><p><strong>The breaking change:</strong> Pods sharing volumes between containers with <em>different</em> SELinux labels will fail to start. The old recursive relabeling could overwrite labels file by file. The new single-context mount cannot reconcile conflicting labels.</p><pre># THIS WILL BREAK in 1.37 - different SELinux labels on shared volume<br>spec:<br>  volumes:<br>  - name: shared-data<br>    emptyDir: {}<br>  containers:<br>  - name: writer<br>    securityContext:<br>      seLinuxOptions:<br>        level: &quot;s0:c100,c200&quot;    # Context A<br>    volumeMounts:<br>    - name: shared-data<br>      mountPath: /data<br>  - name: reader<br>    securityContext:<br>      seLinuxOptions:<br>        level: &quot;s0:c300,c400&quot;    # Context B - CONFLICT - pod fails<br>    volumeMounts:<br>    - name: shared-data<br>      mountPath: /data<br><br>---<br># FIX: Set SELinux at pod level - all containers inherit same context<br>spec:<br>  securityContext:<br>    seLinuxOptions:<br>      level: &quot;s0:c100,c200&quot;    # Single context for all containers<br>  volumes:<br>  - name: shared-data<br>    emptyDir: {}<br>  containers:<br>  - name: writer<br>    volumeMounts:<br>    - name: shared-data<br>      mountPath: /data<br>  - name: reader<br>    volumeMounts:<br>    - name: shared-data<br>      mountPath: /data</pre><h3>7. Pod Certificates — Native Workload Identity</h3><p><strong>Pod Certificates</strong> (Stable) delivers first-class mTLS without sidecars or third-party CAs. Pods receive X.509 certificates via a podCertificate projected volume, automatically provisioned and rotated.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: secure-service<br>spec:<br>  containers:<br>  - name: app<br>    image: myapp:latest<br>    volumeMounts:<br>    - name: workload-cert<br>      mountPath: /var/run/secrets/certs<br>      readOnly: true<br>  volumes:<br>  - name: workload-cert<br>    projected:<br>      sources:<br>      - podCertificate:<br>          signerName: &quot;kubernetes.io/kube-apiserver-client&quot;<br>          expirationSeconds: 3600    # 1 hour - auto-rotated before expiry</pre><p>Verify the mounted certificate</p><pre>kubectl exec secure-service -- \<br>  openssl x509 -in /var/run/secrets/certs/tls.crt -text -noout | \<br>  grep -E &quot;Subject:|Validity|Not After</pre><h3>8. ClusterTrustBundles — Certificate Trust Management</h3><p><strong>ClusterTrustBundles</strong> (Stable) provides a standard way to distribute CA certificate bundles cluster-wide. Pods reference a ClusterTrustBundle to get trusted CA certificates for mTLS peer verification with automatic rotation propagation.</p><pre># Define a trust bundle<br>apiVersion: certificates.k8s.io/v1beta1<br>kind: ClusterTrustBundle<br>metadata:<br>  name: internal-ca-bundle<br>spec:<br>  signerName: &quot;example.com/internal-ca&quot;<br>  trustBundle: |<br>    -----BEGIN CERTIFICATE-----<br>    MIIBpDCCAUqgAwIBAgI...  # Your internal CA cert<br>    -----END CERTIFICATE-----<br><br>---<br># Pod consumes the trust bundle<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: mtls-client<br>spec:<br>  containers:<br>  - name: app<br>    image: myapp:latest<br>    volumeMounts:<br>    - name: ca-bundle<br>      mountPath: /etc/ssl/internal<br>      readOnly: true<br>  volumes:<br>  - name: ca-bundle<br>    projected:<br>      sources:<br>      - clusterTrustBundle:<br>          name: internal-ca-bundle<br>          path: ca-bundle.pem<br>          # Auto-updated when ClusterTrustBundle is rotated</pre><h3>9. cgroups v1 — The Kubelet Will Now Refuse to Start</h3><p>Stop scrolling. This is the most operationally critical change in 1.37.</p><p>Starting with Kubernetes 1.37, <strong>kubelet will fail to initialize by default on nodes using cgroup v1</strong>. The failCgroupV1 flag is true by default. Your nodes will not join the cluster.</p><pre># Check if your nodes are on cgroup v1 or v2<br>stat -fc %T /sys/fs/cgroup/<br># cgroup2fs → cgroup v2 (safe to upgrade)<br># tmpfs     → cgroup v1 (BLOCKED in 1.37)<br><br># Check all nodes in cluster (run from a node or via debug pod)<br># systemd-based systems:<br>cat /proc/cmdline | grep cgroup<br># cgroup v1 indicator: systemd.unified_cgroup_hierarchy=0<br><br># Migrate to cgroup v2 on RHEL/CentOS/Fedora<br>grubby --update-kernel=ALL --args=&quot;systemd.unified_cgroup_hierarchy=1&quot;<br>reboot<br><br># Migrate to cgroup v2 on Ubuntu/Debian<br>sed -i &#39;s/GRUB_CMDLINE_LINUX=&quot;&quot;/GRUB_CMDLINE_LINUX=&quot;systemd.unified_cgroup_hierarchy=1&quot;/&#39; \<br>  /etc/default/grub<br>update-grub &amp;&amp; reboot<br><br># Verify after reboot<br>stat -fc %T /sys/fs/cgroup/<br># Expected output: cgroup2fs</pre><p><strong>Emergency override</strong> (temporary only — will be removed in a future version):</p><pre>kubelet --fail-cgroupv1=false</pre><h3>10. kube-dns on Life Support — Plan Your Migration</h3><p>kube-dns is no longer part of Kubernetes CI pipelines and receives no new feature development. If you&#39;re still running kube-dns (common in clusters from pre-2020), you&#39;re running unsupported DNS infrastructure.</p><p>The recommended path is CoreDNS. For high-traffic clusters, add NodeLocal DNSCache for node-level caching that dramatically reduces latency.</p><pre># Check your current DNS solution<br>kubectl get pods -n kube-system | grep -E &quot;coredns|kube-dns&quot;<br><br># If kube-dns is running, check the version<br>kubectl describe pod -n kube-system -l k8s-app=kube-dns | grep Image<br><br># Verify CoreDNS config after migration<br>kubectl get configmap -n kube-system coredns -o yaml# NodeLocal DNSCache DaemonSet - for high-traffic clusters<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: node-local-dns<br>  namespace: kube-system<br>spec:<br>  selector:<br>    matchLabels:<br>      k8s-app: node-local-dns<br>  template:<br>    metadata:<br>      labels:<br>        k8s-app: node-local-dns<br>    spec:<br>      hostNetwork: true<br>      containers:<br>      - name: node-cache<br>        image: registry.k8s.io/dns/k8s-dns-node-cache:1.23.0<br>        args:<br>        - &quot;-localip&quot;<br>        - &quot;169.254.20.10&quot;    # Link-local IP for node-local cache<br>        - &quot;-conf&quot;<br>        - &quot;/etc/Corefile&quot;<br>        - &quot;-upstreamsvc&quot;<br>        - &quot;kube-dns&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7482420a86fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Kubernetes Cluster Has No Internal Firewall. Here’s Why That Matters]]></title>
            <link>https://medium.com/@rameshavutu/your-kubernetes-cluster-has-no-internal-firewall-heres-why-that-matters-fa8689b30e32?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa8689b30e32</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[k8s-networking]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 05:46:01 GMT</pubDate>
            <atom:updated>2026-07-29T05:46:01.386Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mastering Kubernetes Network Policies</h4><h4>Why Default Allow-All Is a Security Disaster Waiting to Happen and How Network Policies Fix It</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wIE146ZZawVXS0WGvgn5BQ.png" /></figure><h3>The Problem Nobody Talks About in Kubernetes 101</h3><p>You deployed your application to Kubernetes. <br>The frontend is running. <br>The backend is healthy. <br>The database is up. <br>Everything works.</p><p>But here’s something most Kubernetes tutorials skip: <strong>every single one of those Pods can talk to every other Pod in your cluster, right now, without restriction.</strong></p><p>Your frontend can connect directly to your database bypassing the backend entirely. <br>A compromised microservice can reach the secrets endpoint of another service in a completely different namespace. <br>A buggy batch job can hammer your production database with unintended queries.</p><p>This isn’t a misconfiguration. <br>It’s the default behavior.</p><p>In this article, you’ll understand exactly why Kubernetes works this way, what the real risks look like, and why Network Policies are the first line of defense for any production cluster.</p><h3>What You’ll Learn</h3><ul><li>Why Kubernetes allows all Pod-to-Pod communication by default</li><li>What East-West traffic is and why it’s different from North-South</li><li>Three real-world attack scenarios made possible by unrestricted traffic</li><li>What Zero Trust networking means at the Kubernetes layer</li><li>How to observe unrestricted communication in a running cluster (hands-on)</li><li>Where Network Policies fit in the bigger security picture</li></ul><h3>Prerequisites</h3><ul><li>A running Kubernetes cluster (local: kind, minikube, or k3d works)</li><li>kubectl configured and working</li><li>Basic familiarity with Pods, Services, and Namespaces</li></ul><h3>Kubernetes Networking: The Flat Network Model</h3><p>Kubernetes is built on a flat network model. Every Pod gets its own IP address, and by design, <strong>every Pod can reach every other Pod IP directly</strong> — regardless of namespace, node, or workload type.</p><p>This is intentional. The Kubernetes networking model makes three guarantees:</p><ol><li>Every Pod gets a unique, routable IP within the cluster</li><li>Pods on the same node can communicate without NAT</li><li>Pods on different nodes can communicate without NAT</li></ol><p>This flat model makes service discovery and inter-service communication simple. No complex routing rules. No NAT tables to manage. A service in namespace-a can reach a service in namespace-b using a direct IP or a DNS name and it just works.</p><p>The problem is that this simplicity comes with zero isolation by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XEApvg-PCf_CXFbLAXEqzw.png" /></figure><p><strong>Namespaces provide organizational separation but not network isolation.</strong> A Pod in dev can open a TCP connection to a Pod in prod just as easily as if they were in the same namespace. There is no firewall between them.</p><h3>East-West vs. North-South Traffic</h3><p>Before we go further, it helps to understand two terms that appear constantly in Kubernetes security discussions:</p><p><strong>North-South traffic</strong> is traffic that flows into or out of your cluster from the internet to your application, or from your application to an external API. This is traffic that crosses the cluster boundary. Load balancers, Ingress controllers, and egress gateways live in this space.</p><p><strong>East-West traffic</strong> is traffic that flows between workloads <em>inside</em> your cluster — Pod to Pod, service to service, namespace to namespace. This is internal lateral movement.</p><p>Most Kubernetes security discussions focus on North-South: TLS termination, Ingress rules, external firewall policies. But the majority of modern attacks and the majority of <strong>blast radius</strong> <strong>from a compromised workload happen over East-West traffic.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zwv9dJyM7vRS5MgT1KeZFQ.png" /></figure><p>Once an attacker or a buggy workload is inside your cluster, unrestricted East-West traffic is the highway that lets them move laterally across every service you have.</p><h3>Hands-On: Observe Unrestricted Traffic in Action</h3><p>Let’s make this concrete. Deploy two applications in separate namespaces and prove that they can freely communicate — no configuration required.</p><p><strong>Step 1: Create two namespaces</strong></p><pre>kubectl create namespace app-team<br>kubectl create namespace data-team</pre><p><strong>Step 2: Deploy a simple web server in </strong><strong>data-team</strong></p><pre># data-server.yaml<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: data-server<br>  namespace: data-team<br>  labels:<br>    app: data-server<br>spec:<br>  containers:<br>  - name: nginx<br>    image: nginx:alpine<br>    ports:<br>    - containerPort: 80<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: data-service<br>  namespace: data-team<br>spec:<br>  selector:<br>    app: data-server<br>  ports:<br>  - port: 80<br>    targetPort: 80</pre><p>Now apply it.</p><pre>kubectl apply -f data-server.yaml</pre><p><strong>Step 3: Deploy a client Pod in </strong><strong>app-team</strong></p><pre># app-client.yaml<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: app-client<br>  namespace: app-team<br>spec:<br>  containers:<br>  - name: client<br>    image: curlimages/curl:8.21.0<br>    command: [&quot;sleep&quot;, &quot;3600&quot;]</pre><p>Now apply it.</p><pre>kubectl apply -f app-client.yaml</pre><p><strong>Step 4: Verify the Pods are running</strong></p><pre>kubectl get pods -n app-team<br>kubectl get pods -n data-team</pre><p><strong>Step 5: From </strong><strong>app-team, reach </strong><strong>data-team directly</strong></p><pre>kubectl exec -n app-team app-client -- \<br>  curl -s http://data-service.data-team.svc.cluster.local</pre><p>You’ll see the nginx welcome page. <strong>A Pod in </strong><strong>app-team just connected to a service in </strong><strong>data-team without any configuration, credentials, or permission.</strong></p><p>This is the default behavior. Every workload in your cluster is in this position right now.</p><p><strong>Clean up when done:</strong></p><pre>kubectl delete namespace app-team data-team</pre><h3>Three Real-World Attack Scenarios</h3><p>Understanding the default-allow model isn’t just theoretical. Here are three scenarios that happen in real clusters:</p><h3>Scenario 1: The Curious Debug Pod</h3><p>A developer deploys a debug Pod with kubectl run debug --image=busybox --rm -it to troubleshoot a DNS issue. While connected, they intentionally or not, are able to reach internal database endpoints, internal metrics endpoints, and other services that were never meant to be reachable from a debug session.</p><p>The access was never granted. It was never revoked. It simply existed because of the default allow-all model.</p><h3>Scenario 2: The Compromised Dependency</h3><p>A supply chain vulnerability in a third-party container image gives an attacker execution inside one of your Pods. With no network restrictions, that compromised Pod can:</p><ul><li>Scan internal service IPs on common ports (3306, 5432, 6379, 27017)</li><li>Connect to databases and caches directly</li><li>Exfiltrate data by calling internal APIs</li><li>Move laterally to higher-privilege services</li></ul><p>The attacker never touched your perimeter. They moved entirely inside your flat East-West network.</p><h3>Scenario 3: The Noisy Batch Job</h3><p>A misconfigured batch job loops on a database query due to a bug. Since there are no network-level restrictions on how many connections it can open or which services it can reach, it hammers your production database with thousands of connections causing cascading failures across services that have nothing to do with the batch job.</p><h3>What Zero Trust Means at the Network Layer</h3><p>Zero Trust is an architecture principle: <strong>never assume trust based on network location alone.</strong> In traditional data centers, being “inside the firewall” implied trust. In Kubernetes, the flat network model means location inside the cluster grants the same implicit trust problem, but at a finer granularity and at much larger scale.</p><p>Zero Trust at the Kubernetes network layer means:</p><ul><li><strong>Default deny</strong>: all traffic is blocked unless explicitly permitted</li><li><strong>Explicit allow</strong>: every communication path is defined intentionally</li><li><strong>Least privilege</strong>: each workload can reach only what it needs</li><li><strong>Verify continuously</strong>: traffic is inspected and logged, not just allowed</li></ul><p>Network Policies are the Kubernetes-native mechanism for the first three. They don’t give you L7 inspection out of the box, but they give you the foundational layer every Zero Trust architecture starts with.</p><p>The Zero Trust principle in a Kubernetes context looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AjaU62EqczQ6nqU9qjlmjw.png" /></figure><p>You should be able to draw every allowed communication path. If you can’t, you have an unmaintained network.</p><h3>Where Network Policies Fit</h3><p>A Kubernetes Network Policy is a resource that specifies which Pods are allowed to communicate with each other and with other network endpoints. They work at the IP and port level (L3/L4) and are enforced by the CNI plugin running in your cluster.</p><p>The critical detail: <strong>Network Policies are additive and Pod-scoped.</strong> A policy that selects a Pod says “for this Pod, allow this traffic.” Multiple policies combine with OR logic, if any policy allows traffic, the traffic is allowed.</p><p>The equally critical detail: <strong>if no policy selects a Pod, all traffic to and from that Pod is allowed.</strong> The default is open. You have to opt in to restriction.</p><p>This is why you start with a default-deny policy that selects all Pods, then add specific allow rules on top. We’ll cover that in later parts.</p><p>Network Policies are not the complete answer to Kubernetes security, they don’t cover authentication, secrets management, admission control, or runtime security. But they are the foundation. Everything else in cluster security assumes some form of network isolation is in place.</p><h3>One Important Caveat: CNI Support</h3><p>Network Policies are defined in the Kubernetes API, but <strong>enforcement happens at the CNI (Container Network Interface) plugin layer</strong>. Not all CNIs support Network Policies.</p><p>The most commonly deployed CNIs and their support:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNKAPsHx9oRTMHL-26yZ_A.png" /></figure><p>If your CNI doesn’t support Network Policies, you can create them and Kubernetes will accept them but nothing will be enforced. <strong>Always verify your CNI supports enforcement before relying on Network Policies for security.</strong></p><p>We’ll cover the CNI landscape in depth in next part.</p><h3>Common Mistakes at This Stage</h3><p><strong>“My cluster has namespaces, so services are isolated.”</strong><br>Namespaces are a logical boundary for organization, RBAC, and resource quotas. They are not a network boundary. Traffic flows freely across them.</p><p><strong>“My services use TLS, so I don’t need Network Policies.”</strong><br>TLS encrypts traffic in transit. It doesn’t prevent a Pod from initiating a connection to a service it shouldn’t reach. A TLS-encrypted connection to your database from a compromised Pod is still a compromised connection.</p><p><strong>“My cluster is internal, so East-West security doesn’t matter.”</strong><br>The most impactful lateral movement attacks start from a single compromised internal workload i.e a compromised CI/CD job, a vulnerable dependency, or a misconfigured debug container. “Internal” is not a trust boundary.</p><h3>Production Tips</h3><ul><li><strong>Audit your current state before you add policies.</strong> Run kubectl get pods -A -o wide and map out which namespaces and services you have. You can&#39;t secure what you haven&#39;t inventoried.</li><li><strong>Label everything now.</strong> Network Policies use Pod labels for selection. If your Pods don’t have meaningful labels today, add them before you start writing policies. A consistent label strategy (app, tier, component, environment) makes policies readable and maintainable.</li><li><strong>Plan your default-deny first.</strong> The most important decision isn’t what to allow, it’s what to deny by default. We’ll implement this in later parts.</li><li><strong>Test in a non-production namespace first.</strong> A default-deny policy applied to the wrong namespace will break DNS and take down all services in that namespace. Test your understanding in an isolated environment first.</li></ul><h3>Key Takeaways</h3><ul><li>Kubernetes uses a flat network model: every Pod can reach every other Pod by default, across namespaces</li><li>East-West traffic (internal lateral movement) is where most cluster security incidents occur</li><li>Network Policies are opt-in: a Pod with no policy has no restrictions</li><li>The default must be explicit deny, with specific allows layered on top, that’s Zero Trust at the network layer</li><li>Network Policy enforcement requires a compatible CNI; always verify support before depending on it</li></ul><h3>What’s Next?</h3><p>You’ve seen the problem. Now let’s understand the solution.</p><p>In Part 2, we’ll dissect the anatomy of a Network Policy podSelector, namespaceSelector, ipBlock, ingress, egress and understand exactly how Kubernetes evaluates them. Specifically, we&#39;ll cover the AND/OR combination rules that trip up almost every engineer writing their first policy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa8689b30e32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backstage IDP Explained: The Complete Guide to Building an Internal Developer Portal]]></title>
            <link>https://medium.com/@rameshavutu/backstage-docs-7023c73b3390?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/7023c73b3390</guid>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[backstage]]></category>
            <category><![CDATA[internal-developer-portal]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 03:36:01 GMT</pubDate>
            <atom:updated>2026-07-28T03:36:01.314Z</atom:updated>
            <content:encoded><![CDATA[<h4>Stop losing time in scattered microservices, outdated docs and onboarding chaos — learn how Backstage solves it all</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Kds06852oSVmwambLa_KA.png" /></figure><p>Most DevOps teams don’t have a tooling problem — they have a <strong>discoverability problem</strong>.<br> Microservices are scattered, documentation is outdated, and onboarding takes weeks.<br> Developers waste hours every day hunting for the right information.<br> Backstage fixes this by centralizing your software, documentation, and workflows into one portal.<br> This guide will show you how, step by step, even if you’re starting from zero.</p><h3>1. What is Backstage IDP?</h3><p>Backstage is an <strong>Internal Developer Portal (IDP)</strong> originally developed by Spotify. It’s designed to solve real-world DevOps problems:</p><ul><li><strong>Centralized service catalog</strong> — know where every service lives, who owns it, and what it depends on.</li><li><strong>TechDocs</strong> — documentation “like code,” integrated with your repositories.</li><li><strong>Plugins &amp; workflows</strong> — extend Backstage to fit your engineering ecosystem.</li><li><strong>Authentication &amp; security</strong> — control access across teams and services.</li></ul><p><strong>Why it matters:</strong></p><blockquote><em>Platform engineering is now a necessity for growing engineering teams. Backstage is the </em><strong><em>gold standard</em></strong><em> IDP.</em></blockquote><h3>2. Real Problems Backstage Solves</h3><ul><li>❌ Developers can’t find services → delays &amp; frustration</li><li>❌ Onboarding takes weeks → lost productivity</li><li>❌ Documentation is outdated → mistakes and outages</li><li>❌ CI/CD pipelines and microservices are fragmented → chaos</li></ul><p><strong>Backstage centralizes everything</strong>, providing a single source of truth for teams and management alike.</p><h3>3. High-Level Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*da3TpRspPDgl8qushBF-kg.png" /></figure><h3>4. What Most Tutorials Miss (Your Secret Sauce)</h3><ul><li><strong>Auth &amp; RBAC complexity</strong> — hard to get right at scale</li><li><strong>Catalog data quality</strong> — incomplete metadata = broken search</li><li><strong>TechDocs pitfalls</strong> — often ignored in production</li><li><strong>Scaling challenges</strong> — plugins, caching, and multiple teams</li></ul><p><em>Most articles only show “how to install Backstage.” We’re showing </em><strong><em>how to make it work in a real engineering org</em></strong><em>.</em></p><h3>7. Series Roadmap</h3><h4>Start with Part 1 — Build the Portal</h4><blockquote><em>Learn how to create a secure internal developer portal that organizes your engineering workflow and speeds up onboarding.<br> 👉 </em><a href="https://medium.com/@rameshavutu/mastering-backstage-idp-part-1-from-zero-to-secure-developer-portal-9c1991052d48"><em>How to Build a Secure Internal Developer Portal (Backstage Guide)</em></a></blockquote><h4>Part 2 — Software Catalog</h4><blockquote><em>Discover how to organize all your microservices, track ownership, and make your catalog the single source of truth for your team.<br> 👉 </em><a href="https://medium.com/stackademic/mastering-backstage-idp-part-2-building-the-software-catalog-589f864d7226"><em>Backstage Software Catalog Explained: Organize Your Microservices</em></a></blockquote><h4>Part 3 — TechDocs</h4><blockquote><em>Stop wasting time with outdated documentation — learn how to implement TechDocs “like code” so your docs are always up-to-date and useful.<br> 👉 </em><a href="https://medium.com/stackademic/engineering-docs-fail-backstage-techdocs-018e6b0d53a8"><em>TechDocs Done Right: Documentation Like Code in Backstage</em></a></blockquote><h4>Part 4 — Golden Paths &amp; Software Templates</h4><blockquote><em>Standardize your workflows and accelerate developer productivity with Golden Paths using Backstage software templates.<br> 👉 </em><a href="https://medium.com/@rameshavutu/backstage-idp-golden-paths-software-templates-170adce436fe"><em>Golden Paths with Backstage Software Templates: Streamline Development</em></a></blockquote><h4>Part 5 — Automating Tech Debt</h4><blockquote><em>Reduce tech debt automatically by integrating Renovate and custom Backstage plugins — stop issues before they hit production.<br> 👉 Automating Tech Debt with Renovate &amp; Custom Plugins in Backstage</em></blockquote><h3>6. When You Should (and Shouldn’t) Use Backstage</h3><p><strong>Use it if:</strong></p><ul><li>You run 10+ microservices</li><li>You have multiple dev teams</li><li>Onboarding is painful or slow</li></ul><p><strong>Don’t use it if:</strong></p><ul><li>You’re a small team (&lt;5 engineers)</li><li>Your architecture is simple and centralized</li></ul><blockquote><em>Backstage isn’t just a tool — it’s a </em><strong><em>platform engineering shift</em></strong><em>. Teams that adopt it early </em><strong><em>move faster, reduce mistakes, and scale effectively</em></strong><em>.</em></blockquote><p><em>Bookmark this guide and follow the series — by the end, you’ll have a fully functioning IDP ready for production.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7023c73b3390" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Guessing: Debug Istio Like a Pro with istioctl]]></title>
            <link>https://medium.com/@rameshavutu/how-to-debug-istio-with-istioctl-6c4b1e5007c3?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c4b1e5007c3</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[istioctl]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 06:34:18 GMT</pubDate>
            <atom:updated>2026-07-24T06:34:18.384Z</atom:updated>
            <content:encoded><![CDATA[<h4>Master istioctl to diagnose 503s, mTLS mismatches, routing issues, sidecar injection failures, and more… without guessing or restarting pods.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7kaDobpAHdZv7gy6Upc1g.png" /></figure><p>It’s 2 AM.</p><p>Your service is returning 503s.</p><p>Pod logs? Clean. Nodes? Fine. You’ve restarted the deployment twice. Nothing changed.</p><p>The culprit is Istio and you have no idea where to look.</p><p>Most engineers do the same thing at this point.</p><p>Open the Istio docs. Skim the architecture diagram. Start randomly toggling mTLS settings. Hope something sticks.</p><p>Some get lucky. Most burn two more hours until someone who’s <em>“seen this before”</em> shows up and runs three istioctl commands that surface the problem in 30 seconds.</p><p><strong>This guide makes you that person.</strong></p><h3>Why Istio Debugging Feels Impossible</h3><p>There’s one specific reason it’s so hard:</p><p><strong>The failure is never where you’re looking.</strong></p><p>Your pod is healthy. <br>Your Service is correct. <br>Your labels are right.</p><p>But traffic isn’t flowing because Istio is intercepting, mutating, or rejecting it at the sidecar layer.</p><p>None of that shows up in your app logs.</p><p>The istioctl CLI gives you a direct window into what Istio actually sees, enforces, and routes. Once you know eight commands, you stop guessing entirely.</p><h3>Your Debugging Arsenal</h3><p>Here’s the complete toolkit. Every one of these gets used in the scenarios below.</p><pre># Analyze your entire mesh for config issues<br>istioctl analyze<br><br># Human-readable summary of everything Istio sees for a pod<br>istioctl experimental describe pod &lt;pod&gt;.&lt;namespace&gt;<br><br># Check why sidecar injection did or didn&#39;t happen<br>istioctl experimental check-inject -n &lt;namespace&gt; deploy/&lt;deployment&gt;<br><br># Dump the full Envoy config for a pod<br>istioctl proxy-config all &lt;pod&gt;.&lt;namespace&gt;<br><br># What clusters does this pod&#39;s Envoy know about?<br>istioctl proxy-config clusters &lt;pod&gt;.&lt;namespace&gt;<br><br># Inspect TLS at the Envoy listener level<br>istioctl proxy-config listener &lt;pod&gt;.&lt;namespace&gt;<br><br># What AuthorizationPolicies are loaded for this pod?<br>istioctl experimental authz check &lt;pod&gt;.&lt;namespace&gt;<br><br># Sidecar sync status across the whole mesh<br>istioctl proxy-status</pre><p><strong>The mental model:</strong></p><p>analyze → catch declaration-level mistakes first<br>describe pod → human-readable cross-reference of policies, routes, TLS<br>proxy-config → raw Envoy layer when you need it<br>authz check → when the sidecar itself is blocking traffic</p><p>Let’s walk through nine real failure patterns.</p><h3>Scenario 1: 503 With No App Errors</h3><p><em>The most common Istio bug report on the internet.</em></p><p>Service A gets 503 from service B. Service B’s logs show nothing. The pod is Running and Ready.</p><p><strong>Step 1: Check sync status</strong></p><pre>$ istioctl proxy-status<br>NAME                           CLUSTER     CDS     LDS     EDS     RDS     ISTIOD                   VERSION<br>reviews-v1-6b8d97fc5b-xkp9z    Kubernetes  SYNCED  SYNCED  SYNCED  SYNCED  istiod-7c6b89b9f8-q4ml2  1.30.2<br>productpage-v1-85c7d7459-n7j2  Kubernetes  SYNCED  SYNCED  SYNCED  SYNCED  istiod-7c6b89b9f8-q4ml2  1.30.2</pre><p>SYNCED means istiod pushed config and Envoy accepted it.</p><p>STALE or NOT SENT? That’s your first lead. istiod can’t reach the proxy or the proxy can’t process the config.</p><p><strong>Step 2: Check if Envoy even knows about the target</strong></p><pre>$ istioctl proxy-config clusters productpage-v1-85c7d7459-n7j2.default | grep reviews<br>reviews.default.svc.cluster.local    9080    -      outbound  EDS</pre><p>If the target service doesn’t appear, Envoy doesn’t know it exists.</p><p>Check the Service definition, namespace, and label selectors.</p><p><strong>Step 3: Check for mTLS mode mismatch</strong></p><p>This is where most 503s live.</p><pre>$ kubectl get destinationrule reviews -n default -o yaml<br>spec:<br>  host: reviews.default.svc.cluster.local<br>  trafficPolicy:<br>    tls:<br>      mode: DISABLE    # &lt;-- this is the bug</pre><pre>$ kubectl get peerauthentication -n default<br><br>NAME      MODE     AGE<br>default   STRICT   14d</pre><p>The server requires mTLS (STRICT). The client sends plaintext (DISABLE).</p><p>Envoy transmits a plaintext request. The receiving sidecar rejects it. You get a 503 with absolutely nothing in the app logs.</p><p>Confirm it’s actually loaded into Envoy:</p><pre>$ istioctl proxy-config listener productpage-v1-85c7d7459-n7j2.default \<br>    --port 9080 --type outbound -o json | grep -A5 &#39;&quot;tlsContext&quot;&#39;<br><br># Empty output = client sending plaintext<br># tlsContext present = mTLS expected</pre><p><strong>The fix:</strong></p><pre>apiVersion: networking.istio.io/v1beta1<br>kind: DestinationRule<br>metadata:<br>  name: reviews<br>  namespace: default<br>spec:<br>  host: reviews.default.svc.cluster.local<br>  trafficPolicy:<br>    tls:<br>      mode: ISTIO_MUTUAL   # was DISABLE</pre><h3>Scenario 2: Run This Before You Do Anything Else</h3><p><em>Seriously. Make it a habit.</em></p><p>Before reaching for proxy-config, run analyze. It catches an entire class of problems before they become incidents.</p><pre>$ istioctl analyze --namespace production<br>Warn [IST0101] (VirtualService production/checkout-vs) Referenced host not found: &quot;checkout-v2.production.svc.cluster.local&quot;<br>Error [IST0150] (DestinationRule production/checkout-dr) No matching workloads for this resource.</pre><p>That first warning?</p><p>A VirtualService referencing a host that doesn’t exist. 100% of traffic to that route fails silently. No error in your app. No alert fires. It just disappears.</p><p>This is the most common misconfiguration during service renames and namespace migrations.</p><p>istioctl analyze reads your VirtualServices, DestinationRules, Gateways, ServiceEntries, and PeerAuthentication policies and validates them against each other.</p><p><strong>Run it after every change to your mesh. No exceptions.</strong></p><h3>Scenario 3: Sidecar Injection Silently Failed</h3><p>A pod outside the mesh is invisible to Istio telemetry, won’t enforce mTLS, and will cause mysterious 503s for anything trying to reach it under a STRICT policy.</p><pre>$ istioctl proxy-status | grep payment-svc<br>payment-svc-7dfb89c4d-k92pz   Kubernetes  NOT SENT  NOT SENT  NOT SENT  NOT SENT  NOT SENT  -</pre><p>The - in the istiod column: no sidecar. Let&#39;s find out why.</p><pre>$ kubectl get namespace production --show-labels<br>NAME         STATUS   AGE   LABELS<br>production   Active   94d   kubernetes.io/metadata.name=production<br># Missing: istio-injection=enabled</pre><p>No injection label. Every pod created in this namespace never got a sidecar and since injection happens at pod creation time, existing pods aren’t retroactively affected.</p><pre># Fix the namespace<br>kubectl label namespace production istio-injection=enabled<br><br># Restart to trigger injection<br>kubectl rollout restart deployment/payment-svc -n production</pre><p>After the rollout, proxy-status shows SYNCED.</p><h3>Scenario 4: 404 From a Service That’s Actually Running</h3><p><em>The app returns 200 from its own tests. The caller sees 404. Infuriating.</em></p><p>This is almost always a VirtualService routing rule with a prefix ordering problem.</p><pre>$ istioctl proxy-config routes frontend-v2-6c9d88f4b-p7xnq.production --name 9080 -o json[<br>  {<br>    &quot;name&quot;: &quot;9080&quot;,<br>    &quot;virtualHosts&quot;: [<br>      {<br>        &quot;name&quot;: &quot;catalog.production.svc.cluster.local:9080&quot;,<br>        &quot;routes&quot;: [<br>          {<br>            &quot;match&quot;: { &quot;prefix&quot;: &quot;/api/v2&quot; },<br>            &quot;route&quot;: { &quot;cluster&quot;: &quot;outbound|9080|v2|catalog.production.svc.cluster.local&quot; }<br>          },<br>          {<br>            &quot;match&quot;: { &quot;prefix&quot;: &quot;/&quot; },<br>            &quot;route&quot;: { &quot;cluster&quot;: &quot;outbound|9080|v1|catalog.production.svc.cluster.local&quot; }<br>          }<br>        ]<br>      }<br>    ]<br>  }<br>]</pre><p>Envoy evaluates routes <strong>top-to-bottom. First match wins.</strong></p><p>If your VirtualService placed the broad / prefix before the specific /api/v2 one, Envoy never reaches the specific route. The request lands on v1 which doesn&#39;t have that path, returns 404, and the caller has no idea why.</p><p><strong>Rule:</strong> Specific prefixes first. Catch-all last.</p><p>In the VirtualService YAML <em>and</em> as validated here in the Envoy route dump.</p><h3>Scenario 5: Egress Traffic Suddenly Blocked</h3><p><em>It worked yesterday. Nothing changed. The external API is unreachable.</em></p><pre>$ istioctl proxy-config clusters payment-svc-7dfb89c4d-k92pz.production | grep stripe<br># No output</pre><p>Envoy has no cluster entry for stripe.com.</p><p>If your mesh runs outboundTrafficPolicy: REGISTRY_ONLY (the recommended security baseline), Istio blocks all egress to unregistered hosts. Check if the ServiceEntry exists and is actually loaded:</p><pre>$ istioctl proxy-config endpoints payment-svc-7dfb89c4d-k92pz.production | grep stripe<br>outbound|443||stripe.com   HEALTHY  1   stripe.com</pre><p>HEALTHY and present? The ServiceEntry is loaded. Check for a DestinationRule that’s enforcing mTLS on port 443 to an external host that double-terminates TLS and silently breaks the connection.</p><p>While you’re there, check certificate expiry:</p><pre>$ istioctl proxy-config secret payment-svc-7dfb89c4d-k92pz.production<br>RESOURCE NAME   TYPE        STATUS  VALID CERT  NOT AFTER<br>default         Cert Chain  ACTIVE  true        2026-07-23<br>ROOTCA          CA          ACTIVE  true        2036-07-06</pre><p>Expired certs break all mTLS connections silently. Worth checking any time mesh communication breaks with no config changes.</p><h3>Scenario 6: Get the Exact Conflicting Resource Names</h3><p><em>Scenario 1 made you read resources manually. There’s a faster way.</em></p><p>istioctl experimental describe pod cross-references everything for you — and names the exact conflicting resources.</p><pre>$ istioctl experimental describe pod ratings-v1-f745cf57b-qrxl2.default<br>Pod: ratings-v1-f745cf57b-qrxl2<br>   Pod Ports: 9080 (ratings), 15090 (istio-proxy)<br>--------------------<br>Service: ratings<br>   Port: http 9080/HTTP targets pod port 9080<br>Exposed on Ingress Gateway http://35.180.68.104<br>VirtualService: ratings<br>   Weight 100%<br>WARNING Pilot predicts TLS Conflict on ratings-v1-f745cf57b-qrxl2 port 9080<br>(pod enforces mTLS, clients speak HTTP)<br>Check DestinationRule ratings/default and AuthenticationPolicy ratings-strict/default</pre><p>Those last two lines.</p><p>It tells you exactly which DestinationRule and which PeerAuthentication are in conflict by name and namespace. No hunting. No cross-referencing.</p><p>Those last two lines.</p><p>It tells you exactly which DestinationRule and which PeerAuthentication are in conflict — by name and namespace. No hunting. No cross-referencing.</p><p>After fixing the DestinationRule to ISTIO_MUTUAL:</p><pre>Pilot reports that pod enforces mTLS and clients speak mTLS</pre><p><strong>Use </strong><strong>describe pod as your first move on any suspected TLS issue.</strong></p><p>It’s faster than reading four resources individually.</p><h3>Scenario 7: Canary Getting Zero Traffic</h3><p><em>You deployed v2. Set up a VirtualService. Zero requests arrived.</em></p><p>The pod is healthy. The DestinationRule has the subset. What’s wrong?</p><pre>$ istioctl experimental describe pod reviews-v2-6c9d88f4b-p7xnq.default<br><br>Pod: reviews-v2-6c9d88f4b-p7xnq<br>   Pod Ports: 9080 (reviews), 15090 (istio-proxy)<br>--------------------<br>Service: reviews<br>   Port: http 9080/HTTP targets pod port 9080<br><br>VirtualService: reviews<br>   WARNING: No destinations match pod subsets (checked 2 HTTP routes)<br>   Route to non-matching subset v1 for (when headers are end-user=jason)<br>   Route to non-matching subset v1 for (everything)</pre><p>Every route points to v1. The VirtualService was never updated — or was updated incorrectly.</p><p>One command surfaces what proxy-config routes would require you to manually correlate across cluster names and subset labels.</p><p>Another common variant:</p><pre>VirtualService: reviews<br>   Warning: Route to subset v2 but NO DESTINATION RULE defining subsets!</pre><p>The VirtualService references v2 but no DestinationRule with a subset named v2 exists. Traffic has nowhere valid to go.</p><p>Once you fix it, verify the split is live:</p><pre>$ istioctl experimental describe pod reviews-v1-6b8d97fc5b-xkp9z.default | grep Weight<br>Weight 90%<br><br><br>$ istioctl experimental describe pod reviews-v2-6c9d88f4b-p7xnq.default | grep Weight<br>Weight 10%</pre><p>Split confirmed. Canary is receiving traffic.</p><h3>Scenario 8: Check Injection Before You Restart</h3><p><em>The Scenario 3 approach label the namespace, restart, hope works. But this is better.</em></p><p>check-inject tells you whether injection will happen and why, without touching a single running pod.</p><pre>$ istioctl experimental check-inject -n production deploy/payment-svc<br><br>WEBHOOK                     REVISION  INJECTED  REASON<br>istio-revision-tag-default  default   ✘         No matching namespace labels or pod labels</pre><p>INJECTED: ✘ with an explicit reason. Fix the label, then restart. No guessing.</p><p>Common reasons you’ll see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ciUAQQ4qrAIiL8npCCFK6w.png" /></figure><p>That last one deserves attention. During a canary Istio upgrade, two revisions may both target the same namespace:</p><pre>WEBHOOK                      REVISION   INJECTED  REASON<br>istio-revision-tag-default   default    ✔         Namespace label istio-injection=enabled matches<br>istio-revision-tag-1-30      1-30       ✔         Namespace label istio-injection=enabled matches</pre><p>Two ✔ rows means two sidecars racing to inject.</p><p>Fix: scope to a specific revision with istio.io/rev=default or istio.io/rev=1-30. Not both.</p><h3>Scenario 9: 403 — When the Sidecar Is the Gatekeeper</h3><p><em>Your app has no auth at this layer. The 403 is coming from inside the mesh.</em></p><p>AuthorizationPolicy is Istio’s RBAC system. When it fires, Envoy rejects the request before it reaches your app. Nothing appears in app logs.</p><pre>$ istioctl experimental authz check ratings-v1-f745cf57b-qrxl2.default<br><br>ACTION   AuthorizationPolicy        RULES<br>ALLOW    ratings-viewer.default     1<br>DENY     deny-all.default           1</pre><p>The output shows every policy loaded for this pod — its action type and rule count.</p><p>A DENY policy is active. Whether the ALLOW policy covers the actual caller is what you verify next.</p><pre>$ kubectl get authorizationpolicy ratings-viewer -n default -o yaml<br><br>spec:<br>  selector:<br>    matchLabels:<br>      app: ratings<br>  rules:<br>  - from:<br>    - source:<br>        principals: [&quot;cluster.local/ns/default/sa/productpage&quot;]<br>    to:<br>    - operation:<br>        methods: [&quot;GET&quot;]</pre><p>Only productpage is allowed. The actual caller is reviews.</p><pre>kubectl get pod reviews-v1-6b8d97fc5b-xkp9z -n default \<br>  -o jsonpath=&#39;{.spec.serviceAccountName}&#39;<br><br>reviews   # not productpage</pre><p>Add reviews as an allowed principal in the policy. Or confirm you&#39;re calling from the intended service.</p><blockquote><strong>Note:</strong> authz check shows what&#39;s <em>configured</em> — not what fired in real-time. The four possible ACTION values are ALLOW, DENY, LOG, and CUSTOM.</blockquote><blockquote>If authz check shows no policies at all for a pod returning 403s, the deny is coming from a namespace-level or mesh-wide policy. Widen your search.</blockquote><h3>When You’re Completely Stuck: The Escape Hatch</h3><p>proxy-config shows correct config. Traffic still fails. You need to see what Envoy actually does with the packet.</p><p>Enable debug logging on the sidecar:</p><pre># Verbose — only run this temporarily<br>istioctl proxy-config log payment-svc-7dfb89c4d-k92pz.production --level debug</pre><p>Watch the sidecar:</p><pre>kubectl logs payment-svc-7dfb89c4d-k92pz -n production -c istio-proxy --follow</pre><p>Every request appears. You’ll see:</p><ul><li>The upstream cluster it matched</li><li>The response flags</li><li>Any Envoy-level rejection reason</li></ul><p><strong>Key response flags:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K02d7ZH5a7IGaOquLtyC_Q.png" /></figure><p>Reset when done — this is loud:</p><pre>istioctl proxy-config log payment-svc-7dfb89c4d-k92pz.production --level warning</pre><h3>Try It Right Now: The 6-Command Drill</h3><p>Have Minikube? The Bookinfo demo works perfectly for this.</p><pre># 1. Mesh-wide sync picture<br>istioctl proxy-status<br><br># 2. Catch any config issues<br>istioctl analyze -A<br><br># 3. Human-readable pod summary<br>POD=$(kubectl get pod -n default -l app=productpage -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br>istioctl experimental describe pod ${POD}.default<br><br># 4. Will injection work correctly?<br>istioctl experimental check-inject -n default deploy/productpage-v1<br><br># 5. Raw route table<br>istioctl proxy-config routes ${POD}.default --name 9080<br><br># 6. Authorization policies loaded for the ratings pod<br>RATINGS=$(kubectl get pod -n default -l app=ratings -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br>istioctl experimental authz check ${RATINGS}.default</pre><p>Six commands. Under two minutes.</p><p>By step 3 you already have a summary of TLS posture, subset routing, and traffic weights, what used to require reading four separate resources by hand.</p><h3>What istioctl Can’t Do</h3><p>Worth being honest about:</p><ul><li><strong>Live packet capture</strong> → use tcpdump via ephemeral container</li><li><strong>Application-layer payload</strong> → not visible here</li><li><strong>Cross-service latency traces</strong> → Jaeger or Zipkin</li></ul><p>For production traffic analysis: pair istioctl with Kiali for topology and Jaeger for traces.</p><p>istioctl is your <strong>first-responder tool </strong>fast, authoritative, no instrumentation required.</p><h3>The Shift That Changes Everything</h3><p>Istio debugging isn’t random.</p><p>Every failure has a location:</p><ul><li>Config declarations → analyze</li><li>Control-plane sync → proxy-status</li><li>Injection state → check-inject</li><li>mTLS conflicts, subset routing → describe pod</li><li>Authorization denials → authz check</li><li>Routing table → proxy-config routes</li><li>Certificate layer → proxy-config secret</li></ul><p>Map the symptom to the layer.</p><p>Run the command.</p><p>The problem becomes visible.</p><p>From there, it’s just fixing YAML.</p><h3>You’re Not Guessing Anymore</h3><p>Most engineers never get past “restart and hope.”</p><p>They either stumble on the fix by luck or wait for someone else to show up.</p><p>That gap between engineers who guess and engineers who <em>know</em> is eight commands wide.</p><p>You now have all eight.</p><p><strong>If this saved you debugging time, follow me for more deep-dives on Kubernetes, Istio, and production SRE patterns.</strong></p><p><em>I publish practical guides like this weekly, the kind you actually want open during an incident, not after.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c4b1e5007c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Read 300 Pages. You Retained 3. Here’s What’s Actually Going Wrong]]></title>
            <link>https://medium.com/@rameshavutu/how-to-read-faster-without-losing-comprehension-6b2d10199b83?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b2d10199b83</guid>
            <category><![CDATA[speed-reading-exercises]]></category>
            <category><![CDATA[speed-reading]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 18:10:05 GMT</pubDate>
            <atom:updated>2026-07-09T18:10:05.746Z</atom:updated>
            <content:encoded><![CDATA[<h4>Most speed-reading advice hurts comprehension. Here’s the neuroscience-backed system to read faster, retain more and finally remember what you learn.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aWlTaY1TULlaNHTh" /><figcaption>Photo by <a href="https://unsplash.com/@nublson?utm_source=medium&amp;utm_medium=referral">Nubelson Fernandes</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You close the book.</p><p>You’ve been reading for forty-five minutes. You turned pages. You highlighted sentences. Your coffee went cold.</p><p>And now, sitting in the silence, you try to recall what you just read.</p><p>Chapter 1… something about habits? Chapter 2… there was a story. A person. You think their name started with M.</p><p>That’s it. That’s all that survived.</p><p>This isn’t a reading problem. You can read, you’ve been doing it since you were six. This is a <em>processing</em> problem. And the cruel irony? Most speed-reading advice makes it dramatically worse.</p><p>Because speed, without the right architecture underneath it, doesn’t make you read faster.</p><p>It just makes you forget faster.</p><h3>1. The Comprehension Illusion, Why Your Brain Lies to You</h3><p>Here’s the uncomfortable truth no speed-reading app will tell you.</p><p>Your eyes can move across a sentence at remarkable speed. The words register. Your brain fires recognitions. It <em>feels</em> like understanding.</p><p>It isn’t.</p><p>What’s actually happening is called <strong>fluency illusion</strong> — a well-documented cognitive trap where the effortless processing of words is mistaken for deep comprehension. You’ve seen the word “paradigm” a thousand times. Your brain recognizes it instantly. But recognition is not the same as retention.</p><p>Neuroscientists call it <em>shallow encoding.</em> The information reaches working memory, creates a momentary electrical signal, and then, without deeper processing it evaporates. Usually within 20–30 seconds.</p><p>The people who read fast <em>and</em> remember everything aren’t cheating the system. They built a different system. And that system starts with understanding what’s actually happening in your skull when you read.</p><h3>2. How Your Brain Actually Processes Text</h3><p><strong>Your working memory</strong>: the cognitive workspace where active thinking happens and holds roughly <strong>7 chunks of information</strong> at a time (Miller’s Law, 1956). That’s not 7 words. It’s 7 <em>meaningful units.</em></p><p>For a novice reader, “TCP/IP handshake protocol” is four chunks. For a network engineer, it’s one. That’s why domain experts appear to read faster, they’re not moving their eyes more quickly. They’re grouping information more efficiently.</p><p>This is the mechanism behind true reading speed: <strong>schema activation.</strong></p><p>When you already have a mental framework for a topic, incoming information slots in cleanly. When you don’t, your brain is building the framework <em>while</em> trying to absorb content simultaneously — like trying to pour water into a glass you’re still manufacturing.</p><p>The implication is profound: <strong>the best investment in reading speed isn’t a faster eye. It’s a richer mental library.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IifMlX2P-ZUIDhS_h3sSXA.png" /></figure><h3>3. The 3 Speed Killers Destroying Your Retention</h3><p>Before adding speed, eliminate these three comprehension assassins.</p><p><strong>Killer #1: Regression</strong></p><p>You’ve done this a thousand times without noticing. Your eyes move forward across a line, then silently dart back and re-reading words you already passed. Eye-tracking research shows regressions account for <strong>10–15% of all eye movements</strong> during reading a small percentage that adds up to significant wasted time and broken flow across a reading session. Many are habitual, not driven by genuine confusion.</p><p>Regression feels like diligence. It’s actually self-doubt on autopilot.</p><p><strong>Killer #2: Passive Reading</strong></p><p>Eyes moving. Brain disengaged. You’re physically present on the page but cognitively somewhere else running through your to-do list, half-composing a reply to that email.</p><p>Passive reading is the intellectual equivalent of sleepwalking through a museum. You technically went. You saw nothing.</p><p><strong>Killer #3: Cold Entry</strong></p><p>You pick up a book, open to page 47, and dive in without any cognitive warm-up. No context. No schema activated. No questions primed.</p><p>Your brain is essentially handed a jigsaw puzzle with no picture on the box. It can still assemble pieces slowly, painfully, with frequent backtracking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vdLvMmFupzn8Ag6N" /><figcaption>Photo by <a href="https://unsplash.com/@minimdesignco?utm_source=medium&amp;utm_medium=referral">Brad</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>4. Pre-Reading: The 5-Minute Ritual That Changes Everything</h3><p>Here’s the counter-intuitive truth: <strong>reading slower before you read fast is how you actually read faster.</strong></p><p>Navy SEAL intelligence analysts don’t dive into a classified brief and read word-by-word under pressure. They scan. They orient. They extract the information architecture before processing content. They’re answering the question: <em>“What am I about to understand?”</em></p><p>You can do the same with any book, article, or report.</p><p><strong>The 5-Minute Pre-Read Protocol:</strong></p><ol><li><strong>Read the title and subtitle</strong> Form your first hypothesis about the argument</li><li><strong>Scan all headings and subheadings </strong>Build the skeleton of the content in your mind</li><li><strong>Read the first and last paragraph of each section </strong>Capture the topic sentence and conclusion</li><li><strong>Identify 3 questions</strong> you want answered Write them down, literally. This creates <em>intentional reading.</em></li><li><strong>Estimate reading time</strong> Sets a mental commitment anchor</li></ol><p>This five-minute investment reduces total reading time by 20–40% in most studies, because your brain processes information it’s been primed to expect 2–3x faster than cold information.</p><p>You’re not reading slower. You’re building the road before driving on it.</p><h3>5. The Chunking Method — Read in Thought Groups, Not Words</h3><p>This is where genuine speed gains live.</p><p>Most people were taught to read word-by-word — a necessary scaffold for a six-year-old learning decoding. But you’re not six. You’re still using training wheels on a vehicle that can go 100 mph.</p><p><strong>Chunking</strong> means training your eye to capture 3–5 words in a single fixation, rather than landing on each individual word.</p><p><strong>Progression Drill (15 minutes daily):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SOEAAvGyjOamf3_G6BiOg.png" /></figure><p><strong>How to practice:</strong></p><ul><li><strong>Use a pen or finger as a pacer </strong>run it smoothly under lines without stopping</li><li>Force yourself to keep moving forward (eliminates regression simultaneously)</li><li><strong>Focus on the center of each line</strong> your peripheral vision handles the edges</li></ul><p>A critical nuance: don’t push speed until comprehension holds at each level. Speed without comprehension is just expensive page-turning.</p><h3>6. Active Recall — The Single Highest-ROI Retention Tool</h3><p>Here’s what the research is unambiguous about: <strong>highlighting is nearly useless for retention.</strong></p><p>It <em>feels</em> productive. You’re engaging with the text. The yellow marker makes the page look alive with your attention. But passive marking requires no cognitive effort — and learning requires effort to stick.</p><p><strong>The Forgetting Curve</strong> (Ebbinghaus, 1885 — still the most replicated finding in memory research): without active retrieval, you forget approximately <strong>67% of new information within 24 hours</strong>, and up to <strong>75% within six days</strong>. A week later, very little remains without reinforcement.</p><p>Active recall reverses this catastrophically fast forgetting.</p><p><strong>The Pause-and-Summarize Technique:</strong></p><p>Every 2–3 pages, stop. Close the book or look away from the screen. In your own words out loud or in writing summarize what you just read. Not what the author said. What <em>you</em> understood.</p><p>This one habit, practiced consistently, has been shown in multiple studies to <strong>substantially outperform passive re-reading for long-term retention</strong> — with researchers like Roediger &amp; Karpicke (2006) consistently finding that students who practiced active retrieval retained far more on delayed tests than those who spent the same time re-reading.</p><p><strong>The Feynman Integration:</strong><br>After finishing a chapter, explain the core idea as if teaching it to a twelve-year-old. Wherever your explanation breaks down — that’s the gap in your understanding. Go back and fix it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*B5n5dQwUVXqP6CAhDmruOA.png" /></figure><h3>7. Your 4-Week Reading Transformation Plan</h3><p>Don’t try to implement everything simultaneously. Stacking habits sequentially is the difference between lasting change and a three-day experiment that fades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sbiq7chFTpJdBQJT1LuRtw.png" /></figure><p><strong>Week 1 — Eliminate the Killers</strong></p><ul><li>Day 1: Take a baseline reading speed test (<a href="http://readingsoft.com">readingsoft.com</a>)</li><li>Days 2–7: Practice regression elimination with a pacer (pen/finger) for 20 minutes daily</li><li>No speed target just clean, forward-moving eyes</li></ul><p><strong>Week 2 — Build the Engine</strong></p><ul><li>Add 15-minute chunking drills with fiction (lower cognitive load)</li><li>Begin 5-minute pre-reading ritual on every non-fiction piece</li><li>Target: 2–3 word chunk fixations</li></ul><p><strong>Week 3 — Install the Memory System</strong></p><ul><li>Introduce pause-and-summarize every 2–3 pages</li><li>End each reading session with a 3-minute free-recall write (everything you remember, stream of consciousness)</li><li>Add Readwise or a simple flashcard system for key insights</li></ul><p><strong>Week 4 — Full Integration</strong></p><ul><li>All systems running together: pre-read → chunk → pause-recall → review</li><li>Measure speed again — most people see 60–100% improvement with near-identical comprehension scores</li></ul><h3>8. Tools That Actually Help (And Ones That Don’t)</h3><p><strong>✅ Tools Worth Your Time:</strong></p><ul><li><strong>Readwise</strong> — Resurfaces your highlights at optimal spaced repetition intervals. Turns passive marking into active review.</li><li><strong>Kindle + Export highlights</strong> — Use Clippings Kindle app to export and organize insights into a searchable personal database.</li><li><strong>Timer-based reading sessions</strong> — 25-minute focused blocks (Pomodoro) outperform marathon sessions. Cognitive fatigue is real and it destroys comprehension after 45–60 minutes.</li><li><strong>Spreeder (at conservative speeds)</strong> — Useful for chunking drills at controlled WPM. Don’t push past your comprehension ceiling.</li></ul><p><strong>❌ Tools to Avoid:</strong></p><ul><li><strong>RSVP (Rapid Serial Visual Presentation) apps</strong> — Flashing one word at a time on screen. Feels incredible. For most readers, meta-analyses show comprehension drops <strong>20–40%</strong> because RSVP eliminates the natural eye movements and saccades your brain uses to re-anchor and process context. (Note: a 2025 study found RSVP can <em>help</em> readers with ADHD by reducing visual distraction — so this caveat applies to neurotypical readers.)</li><li><strong>“PhotoReading” courses</strong> — Absorbing entire pages in a “subliminal” glance. No peer-reviewed evidence. None.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bDV1PKEdGO3QJvDP" /><figcaption>Photo by <a href="https://unsplash.com/@adryan_studio?utm_source=medium&amp;utm_medium=referral">Puscas Adryan</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The 1% Principle And Your Next Step</h3><p>No one becomes a better reader in a day.</p><p>But 1% better each session? That compounds. Over four weeks, a 1% daily improvement yields a 33% overall gain. Over three months? You’re an entirely different reader.</p><p>The readers who retain everything aren’t gifted with better memories. They built better systems. The pre-read ritual. The chunking habit. The pause and recall. These aren’t tricks, they’re the cognitive infrastructure that separates reading <em>at</em> a page from reading <em>into</em> a page.</p><p>Start with one thing this week. Eliminate regression with a pacer. Five minutes. One book. One change.</p><p>Then tell me how it goes. Drop it in the comments . what’s your baseline WPM, and what topic are you reading right now?</p><p>I’ll see you in Week 4.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b2d10199b83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[kubectl debug: The Command Every Kubernetes Engineer Underuses]]></title>
            <link>https://medium.com/beyond-localhost/kubectl-debug-distroless-containers-crashloopbackoff-5cdc850ed735?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cdc850ed735</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 01:31:00 GMT</pubDate>
            <atom:updated>2026-06-22T01:31:00.875Z</atom:updated>
            <content:encoded><![CDATA[<h4>Five real debugging scenarios: ephemeral containers, distroless images, CrashLoopBackOff, node-level access, and network tracing with copy-paste commands for each</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6TnzWC11Q93Ohlu0CpvBA.png" /></figure><p>You’re Staring at a Distroless Container. No Shell. No Tools. Nothing.</p><pre>kubectl exec mypod - /bin/sh<br>error: Internal error occurred: error executing command in container</pre><p>Sound familiar?</p><p>You built a minimal, secure container image with no shell, no extra binaries, just the app.</p><p>Best practice. Reduced attack surface. Completely undebugable.</p><p>And now something is wrong in production, the clock is ticking, and you have zero visibility inside that container.</p><p><strong>This is the exact problem </strong><strong>kubectl debug was built to solve.</strong></p><p>But here’s what most engineers don’t know: kubectl debug doesn&#39;t just fix the distroless problem.</p><blockquote>It fixes <em>all</em> the situations where kubectl exec falls apart and most engineers only know one of its three modes.</blockquote><h3>kubectl exec Has a Dirty Secret</h3><p>It has one hard requirement most people never think about until it’s 2am and something is broken:</p><p><strong>The container must have a shell or the exact binary you want to run.</strong></p><p>That requirement breaks completely for:</p><ul><li><strong>Distroless images:</strong> no shell, no tools, just the app binary</li><li><strong>Scratch-based images:</strong> same story</li><li><strong>CrashLoopBackOff pods:</strong> exits before you can get in</li><li><strong>Restricted namespaces: </strong>where exec is blocked by policy</li></ul><h4>What Is kubectl debug?</h4><p>kubectl debug is a Kubernetes troubleshooting command that lets you investigate running workloads without rebuilding container images or modifying production deployments.</p><p>It supports:</p><p>- Ephemeral containers<br>- Pod copy debugging<br>- Node-level debugging</p><p>kubectl debug became generally available (GA) in Kubernetes 1.25 and is now the preferred way to troubleshoot distroless containers, CrashLoopBackOff pods, and node-level issues.</p><p>It doesn’t require you to touch your image.</p><p>It doesn’t restart your pod.</p><p>It doesn’t disrupt your workload.</p><p>It just works once you understand the three modes.</p><h3>Three Modes. One Command.</h3><p>Before the scenarios, here’s the mental model:</p><p><strong>Mode 1: Ephemeral Container</strong></p><p>Adds a temporary debug container to a <em>running</em> pod.</p><p>The pod keeps running. The ephemeral container shares the pod’s namespaces. Once added, it cannot be removed.</p><p><strong>Mode 2: Pod Copy</strong></p><p>Creates a duplicate of the pod with modifications, a new debug container, swapped image, or overridden entrypoint.</p><p>Original pod is untouched. Delete the copy when you’re done.</p><p><strong>Mode 3: Node Shell</strong></p><p>Schedules a privileged pod on a specific node with full access to the host’s namespaces. The host filesystem is mounted at /host.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMSioZDjTmiIvslv61kZ9A.png" /></figure><p>Not sure which one fits your situation?</p><h3>Pick Your Mode in 30 Seconds</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afXBa27lfxw45I65BLN4Tw.png" /></figure><p>Got your mode? One more thing before the scenarios.</p><h3>The --profile Flag Nobody Reads (Until They Need It)</h3><p>Every kubectl debug command accepts --profile.</p><p>It controls what capabilities your debug container gets. Default is general.</p><p>Get this wrong and your tools silently fail — strace returns permission denied, tcpdump won&#39;t capture, iptables won&#39;t list rules.</p><p>Know these before you start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzlBS_8Y2RI3aHtkvDg-tA.png" /></figure><pre>kubectl debug -it mypod --image=busybox --target=mycontainer --profile=netadmin</pre><blockquote>Now the five scenarios.</blockquote><h3>Scenario 1: Something’s Wrong and Your Pod Is Running</h3><p>Unexpected behavior. A process you can’t explain. Wrong environment variable being picked up.</p><p>You want a shell <em>alongside</em> the running container without restarting it, without touching it.</p><pre>kubectl debug -it mypod --image=busybox:1.38.0 --target=mycontainer</pre><p>That’s it.</p><p>--target is the critical flag here. It attaches your debug container to the <em>process namespace</em> of mycontainer so you can see the main container&#39;s processes with ps.</p><p>Without --target, you get an isolated namespace. You&#39;ll be flying blind.</p><p>Once you’re inside:</p><pre># See what&#39;s actually running<br>ps aux<br><br># Check what env vars the app is reading<br>env<br><br># Browse the filesystem via shared process namespace<br>ls /proc/1/root/</pre><p>When you exit, the ephemeral container stops but it stays in the pod spec.</p><p>To resume the same session:</p><pre>kubectl attach mypod -c debugger-8xzrl -i -t</pre><p>The container name is printed when the session starts. Forgot it?</p><pre>kubectl describe pod mypod | grep -A5 &quot;Ephemeral Containers&quot;</pre><p>That’s the warm-up. Now the scenario most engineers get stuck on.</p><h3>Scenario 2: kubectl debug for Distroless Containers</h3><p>No shell. No ls. No curl. No ps.</p><p>kubectl exec is completely useless.</p><p>Here’s the fix and the command will look familiar:</p><pre>kubectl debug -it mypod --image=busybox:1.38.0 --target=mycontainer</pre><p><strong>Same command. Different target.</strong></p><p>You’re attaching a busybox ephemeral container to a distroless container’s process namespace. The distroless app has no idea. You now have full visibility:</p><pre># See the distroless binary running as PID 1<br>ps aux<br># PID   USER     TIME  COMMAND<br>#   1   root      0:00 /app/server          ← distroless binary<br>#  12   root      0:00 sh                   ← your debug shell<br><br># Browse its filesystem via /proc<br>ls /proc/1/root/<br>ls /proc/1/root/etc/<br><br># Check open file descriptors<br>ls /proc/1/fd<br><br># See what libraries it&#39;s linked against<br>cat /proc/1/maps</pre><p>Need heavier firepower? Swap busybox for my favouritenicolaka/netshoot it includes curl, dig, tcpdump, ss, strace, and 70+ other tools:</p><pre>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer</pre><p>Now:</p><pre># Network connections the distroless app has open<br>ss -tulnp<br><br># DNS from inside the pod<br>dig kubernetes.default.svc.cluster.local</pre><h4>Want to Run strace? Read This First.</h4><p>strace requires SYS_PTRACE. The default general profile doesn&#39;t grant it.</p><p>This is a silent failure, the command runs, nothing happens, you wonder why.</p><p>The fix:</p><pre>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer --profile=sysadmin<br><br># Then inside:<br>strace -p 1</pre><blockquote><strong>Note:</strong> Process namespace sharing requires the container runtime to support --target. Modern containerd and CRI-O both do. If ps doesn&#39;t show the main container&#39;s processes, your runtime may not.</blockquote><p>The ephemeral approach only works while the pod is alive.</p><p>When the pod keeps dying before you can get in, that’s the next scenario.</p><h3>Scenario 3: kubectl debug for CrashLoopBackOff Pods</h3><p>The pod crashes in 2 seconds.</p><p>kubectl exec? Gone before you type the command.</p><p>kubectl logs? Sometimes empty — crash happens before the app writes anything.</p><p><strong>The solution: pod copy mode. Freeze the environment, take the crash out of the equation.</strong></p><pre>kubectl debug mypod -it --copy-to=mypod-debug --container=mycontainer -- sh</pre><p>This creates mypod-debug a perfect clone of mypod with the same env vars, same volumes, same resource limits — but with the crash-causing command replaced by sh.</p><p>The container stays up. You’re now inside the exact environment where the crash was happening.</p><pre># Trigger the crash manually to see the real error<br>/app/server<br><br># Hunt for missing config<br>env | grep -i database<br>env | grep -i secret<br><br># Verify secrets and configmaps are actually mounted<br>ls /etc/secrets/<br>cat /etc/config/app.yaml</pre><p><strong>Two More Pod Copy Tricks</strong></p><p>Swap the image instead of the command:</p><pre>kubectl debug mypod --copy-to=mypod-debug --set-image=mycontainer=busybox</pre><p>Keeps the original entrypoint. Replaces the container image. Useful when the image itself is the problem.</p><p><strong>Add a debug sidecar without changing the crashing container at all:</strong></p><pre>kubectl debug mypod -it --image=busybox --share-processes --copy-to=mypod-debug</pre><p>When you’re done:</p><pre>kubectl delete pod mypod-debug</pre><p>Sometimes the pod is perfectly fine. The problem is the node it’s sitting on.</p><h3>Scenario 4: kubectl debug for Kubernetes Node Troubleshooting</h3><p>Disk pressure. Kubelet refusing to schedule. Container runtime throwing cryptic errors. Certificate expiry on the host.</p><p>You need to get <em>onto the node</em> — not just into a pod.</p><pre>kubectl debug node/worker-node-1 -it --image=ubuntu</pre><p>This schedules a pod on worker-node-1 with:</p><ul><li>Host PID, network, and IPC namespaces shared</li><li>Host root filesystem mounted at /host</li><li>No resource limits</li></ul><p>Once you’re in:</p><pre># Step into the host filesystem<br>chroot /host<br><br># Check kubelet<br>systemctl status kubelet<br><br># Inspect what containers the runtime sees<br>crictl ps<br>crictl logs &lt;container-id&gt;<br><br># Disk pressure diagnosis<br>df -h<br><br># System logs<br>journalctl -u kubelet -n 100<br>journalctl -n 200 --since &quot;10 minutes ago&quot;</pre><p>For network investigation at the host level:</p><pre>kubectl debug node/worker-node-1 -it --image=nicolaka/netshoot</pre><p>Once you’re in:</p><pre># Capture host-level traffic<br>tcpdump -i eth0 -n port 443<br><br># Check what&#39;s listening<br>ss -tulnp<br><br># Inspect iptables rules<br>iptables -L -n | grep &lt;service-ip&gt;<br><br># Verify node-level DNS<br>nslookup kubernetes.default</pre><p>Need cgroup access or kernel-level inspection?</p><pre>kubectl debug node/worker-node-1 -it --image=ubuntu --profile=sysadmin</pre><p>Clean up after:</p><pre>kubectl delete pod node-debugger-worker-node-1-xxxx</pre><p>The pod name is printed when the session starts.</p><p>One scenario left and it’s the one that catches most engineers off-guard.</p><h3>Scenario 5: kubectl debug for Kubernetes Network Troubleshooting</h3><p>The pod is running. The service exists. Traffic still isn’t flowing.</p><p>Wrong DNS. Broken NetworkPolicy. Misconfigured port. Silent packet drop.</p><p>You need traffic capture and routing visibility and that requires NET_ADMIN:</p><pre>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer --profile=netadmin</pre><p>Inside:</p><pre># Test reachability<br>curl -v http://my-service.default.svc.cluster.local:8080/health<br><br># Inspect routing table<br>ip route<br><br># Check iptables rules<br>iptables -L -n<br><br># DNS investigation<br>dig my-service.default.svc.cluster.local<br>nslookup kubernetes.default.svc.cluster.local</pre><h3>Capturing Traffic — The Gotcha</h3><p>kubectl cp <strong>does not work with ephemeral containers.</strong></p><p>Don’t try to write a .pcap file inside the container and copy it out — it won&#39;t work.</p><p>Pipe the output directly instead:</p><pre># Stream capture to a local file<br>kubectl exec mypod -c &lt;debug-container-name&gt; -- tcpdump -i eth0 -n -w - &gt; capture.pcap<br><br># Or watch live in terminal<br>kubectl exec mypod -c &lt;debug-container-name&gt; -- tcpdump -i eth0 -n</pre><h3>Ephemeral Containers: The Gotchas That Will Bite You</h3><p>Before you start using ephemeral containers in production, read this section.</p><p><strong>You can’t remove them.</strong></p><p>Once added to a pod, the ephemeral container stays until the pod is deleted. For production debugging, pod copies are usually cleaner, you can delete them when done.</p><p><strong>No resource limits.</strong></p><p>Ephemeral containers have no resource requests or limits. On a loaded node, an unbounded container can cause problems.</p><p><strong>No ports, no probes.</strong></p><p>You can’t expose ports or add health checks to ephemeral containers.</p><p><strong>Static pods don’t support them.</strong></p><p>Debugging kube-apiserver or other control-plane components running as static pods? Use node mode instead.</p><p><strong>kubectl cp is not supported.</strong></p><p>Pipe output out via kubectl exec, as shown in Scenario 5.</p><p><strong>Runtime support required.</strong></p><p>If --target isn&#39;t showing the main container&#39;s processes, your container runtime may not support process namespace sharing. Rare with modern containerd/CRI-O, but possible.</p><h3>The Full Cheat Sheet</h3><p>Bookmark this. You’ll come back to it.</p><pre># Basic ephemeral container on a running pod<br>kubectl debug -it mypod --image=busybox --target=mycontainer<br><br># Distroless debugging — share process namespace<br>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer<br><br># Distroless with strace (needs SYS_PTRACE)<br>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer --profile=sysadmin<br><br># CrashLoopBackOff — override entrypoint to freeze the environment<br>kubectl debug mypod -it --copy-to=mypod-debug --container=mycontainer -- sh<br><br># CrashLoopBackOff — swap container image<br>kubectl debug mypod --copy-to=mypod-debug --set-image=mycontainer=busybox<br><br># Add debug sidecar with shared process namespace<br>kubectl debug mypod -it --image=busybox --share-processes --copy-to=mypod-debug<br><br># Node shell — host filesystem at /host<br>kubectl debug node/mynode -it --image=ubuntu<br><br># Node shell with elevated capabilities<br>kubectl debug node/mynode -it --image=ubuntu --profile=sysadmin<br><br># Network debugging with NET_ADMIN<br>kubectl debug -it mypod --image=nicolaka/netshoot --target=mycontainer --profile=netadmin<br><br># Resume a detached ephemeral container session<br>kubectl attach mypod -c &lt;debug-container-name&gt; -i -t<br><br># Clean up pod copy<br>kubectl delete pod mypod-debug</pre><h3>What Engineers Did Before This Existed</h3><p>Before kubectl debug was GA, the workarounds were painful:</p><p><strong>Temporary debug deployments</strong> spin up a separate pod with debug tools in the same namespace, manually wire it up. Slow. Messy. Doesn’t share the target pod’s environment.</p><p><strong>Patching the pod spec to add a sidecar</strong> requires deleting and recreating the pod. Kills any transient state you were trying to capture. Defeats the purpose entirely.</p><p><strong>Maintaining debug image variants</strong> keeping myapp:debug alongside myapp:prod with extra tools baked in. Doubles your image maintenance. Increases attack surface of the debug build.</p><p>kubectl debug makes all three obsolete.</p><p>The distroless case is the clearest win: attaching a fully-tooled ephemeral container to a minimal image, without restarting the pod, without touching the image, without exposing a debug build to production.</p><p><strong>Most engineers still reach for the old workarounds out of habit.</strong></p><p>Now you don’t have to.</p><h4>kubectl debug FAQ</h4><h4><strong>Does kubectl debug restart a pod?</strong></h4><p>No. Ephemeral container mode adds a temporary container to a running pod without restarting the workload.</p><h4><strong>Can kubectl debug be used with distroless images?</strong></h4><p>Yes. This is one of its most common use cases. You can attach a debug container and inspect the distroless process from a shared namespace.</p><h4><strong>Can kubectl debug fix CrashLoopBackOff pods?</strong></h4><p>It cannot fix them automatically, but pod copy mode allows you to recreate the environment and investigate the root cause.</p><h4>Does kubectl debug work on Kubernetes nodes?</h4><p>Yes. Node debugging mode launches a privileged pod on the target node and mounts the host filesystem under /host.</p><h3>Resources</h3><ul><li><strong>Kubernetes docs:</strong> <a href="https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/">https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/</a></li><li><strong>Ephemeral containers reference:</strong> <a href="https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/">https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/</a></li><li><strong>kubectl debug reference:</strong> <a href="https://kubernetes.io/docs/reference/kubectl/generated/kubectl_debug/">https://kubernetes.io/docs/reference/kubectl/generated/kubectl_debug/</a></li><li><strong>nicolaka/netshoot:</strong> <a href="https://github.com/nicolaka/netshoot">https://github.com/nicolaka/netshoot</a></li></ul><p>If this saved you from a bad incident or reminded you of a time one of these scenarios burned you, let me know in the comments.</p><p><a href="https://medium.com/@rameshavutu">I write about Kubernetes, AWS, and production debugging every week. Follow along if you want more of this.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cdc850ed735" width="1" height="1" alt=""><hr><p><a href="https://medium.com/beyond-localhost/kubectl-debug-distroless-containers-crashloopbackoff-5cdc850ed735">kubectl debug: The Command Every Kubernetes Engineer Underuses</a> was originally published in <a href="https://medium.com/beyond-localhost">Beyond Localhost</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[If Charlie Munger Ran Your DevOps Team, He’d Probably Fire Your Heroes]]></title>
            <link>https://medium.com/@rameshavutu/charlie-munger-devops-mental-models-16ef777eb0af?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/16ef777eb0af</guid>
            <category><![CDATA[charlie-munger]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[mental-models]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 00:31:00 GMT</pubDate>
            <atom:updated>2026-07-07T09:34:43.482Z</atom:updated>
            <content:encoded><![CDATA[<h4>Applying Timeless Mental Models to Build Resilient, Scalable DevOps Systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yf0fBtMWnTYeo2u4qX2S4g.png" /><figcaption>Image created using Chatgpt</figcaption></figure><p>Charlie Munger never wrote a line of production code.</p><p>He never deployed a microservice.</p><p>He never debugged a Kubernetes cluster at 2 AM.</p><p>Yet if you handed him the keys to a large engineering organization tomorrow, he would immediately spot problems that most DevOps teams completely miss.</p><p>And he’d probably identify them faster than many experienced engineers.</p><p>Why?</p><p>Because software engineering isn’t fundamentally about code.</p><p>It’s about building systems that don’t collapse under pressure.</p><p>And nobody understood systems better than Charlie Munger.</p><p>For more than 70 years, Munger studied thousands of businesses.</p><p>He wasn’t evaluating APIs or cloud infrastructure.</p><p>He was evaluating something deeper:</p><p><strong>How systems succeed.</strong></p><p><strong>How systems fail.</strong></p><p><strong>And how small mistakes compound into massive disasters.</strong></p><p>The same mental models he used to identify fragile companies can be applied directly to modern DevOps.</p><p>If Munger walked into your engineering organization today, he wouldn’t ask:</p><p><em>“What language are you using?”</em></p><p>He’d ask:</p><p><em>“How does this system fail?”</em></p><p>And that’s where things get uncomfortable.</p><h3>Mental Model #1: Inversion — Design for Failure First</h3><p>One of Munger’s favorite thinking tools was something he called <strong>Inversion</strong>.</p><p>Instead of asking:</p><blockquote><em>“How do we succeed?”</em></blockquote><p>Ask:</p><blockquote><em>“How do we guarantee failure?”</em></blockquote><p>Most engineering leaders focus on optimization.</p><p>They ask:</p><ul><li>How do we ship faster?</li><li>How do we increase deployment frequency?</li><li>How do we improve velocity?</li></ul><p>Munger would ask a different question:</p><p><strong>If I wanted this platform to crash at 2:00 AM on a Friday night, how would I build it?</strong></p><p>The answers come quickly.</p><p>You’d probably:</p><ul><li>Require manual production deployments</li><li>Store critical knowledge inside one engineer’s head</li><li>Skip automated testing</li><li>Ignore monitoring</li><li>Depend on heroic interventions during incidents</li></ul><p>Sound familiar?</p><p>That’s because many organizations accidentally build exactly that.</p><p>This creates what I call <strong>Hero Engineering</strong>.</p><p>Everything works… until it doesn’t.</p><p>Then one exhausted senior engineer jumps into Slack, SSHs into production, performs emergency surgery, and saves the company.</p><p>Everyone celebrates.</p><p>The engineer becomes a legend.</p><p>The real problem never gets fixed.</p><p>Munger hated this type of thinking.</p><p>He didn’t trust heroes.</p><p>He trusted systems.</p><p>A famous principle he often referenced was simple:</p><blockquote><em>Don’t depend on brilliance when a checklist will do.</em></blockquote><p>In software, those checklists look like:</p><ul><li>Infrastructure as Code</li><li>Automated CI/CD pipelines</li><li>Runbooks</li><li>Monitoring</li><li>Incident response procedures</li></ul><p>If your production environment depends on someone being awake, available, and brilliant, your architecture has already failed.</p><p>A resilient system shouldn’t need heroes.</p><p>It should make heroics unnecessary.</p><h3>Mental Model #2: Margin of Safety</h3><p>The second principle comes from investing.</p><p>It’s called the <strong>Margin of Safety</strong>.</p><p>Investors know their predictions will be wrong.</p><p>Markets are unpredictable.</p><p>Reality never follows spreadsheets.</p><p>So instead of buying a $100 asset for $100, they buy it for $50.</p><p>That gap protects them when things inevitably go wrong.</p><p>Munger applied this idea everywhere.</p><p>He would apply it to your infrastructure too.</p><p>Imagine your application receives a sudden traffic spike.</p><p>Can your system absorb it?</p><p>What happens if:</p><ul><li>A database node fails?</li><li>A cloud region goes offline?</li><li>Traffic doubles overnight?</li><li>A deployment introduces unexpected load?</li></ul><p>If the answer is “we’ll figure it out,” you don’t have a margin of safety.</p><p>You have hope.</p><p>And hope is not architecture.</p><p>Fragile systems operate right at the edge of failure.</p><p>Resilient systems create buffers.</p><p>That’s why elite engineering teams invest heavily in:</p><ul><li>Auto-scaling</li><li>Geographic redundancy</li><li>Backups</li><li>Failover systems</li><li>Capacity planning</li></ul><p>This is also why Chaos Engineering is so powerful.</p><p>Intentionally breaking things sounds crazy.</p><p>Until you realize it’s one of the best ways to discover hidden weaknesses before customers do.</p><p>Think of infrastructure like a bridge.</p><p>If the heaviest truck weighs 10,000 pounds, you don’t build a bridge that supports 10,001.</p><p>You build one that supports 30,000.</p><p>That’s margin of safety.</p><p>That’s long-term thinking.</p><h3>Mental Model #3: Compounding Returns on Automation</h3><p>This is where Munger made most of his fortune.</p><p>He understood compounding better than almost anyone.</p><p>Small advantages repeated over long periods become enormous advantages.</p><p>The same principle applies to engineering.</p><p>Most teams think automation is about saving time.</p><p>That’s true.</p><p>But it’s only half the story.</p><p>Automation creates compound returns.</p><p>Imagine an engineer spends four hours automating a deployment process.</p><p>At first glance, that seems expensive.</p><p>After all, the manual process only took 15 minutes.</p><p>But Munger wouldn’t look at today’s cost.</p><p>He’d look at future returns.</p><p>Those 15 minutes are saved:</p><ul><li>Tomorrow</li><li>Next week</li><li>Next month</li><li>Next year</li></ul><p>And the savings keep accumulating.</p><p>Forever.</p><p>Now compare that to technical debt.</p><p>Technical debt compounds too.</p><p>Just in the opposite direction.</p><p>Every shortcut creates future friction.</p><p>Every skipped test increases future risk.</p><p>Every messy workaround becomes tomorrow’s bottleneck.</p><p>Over time, teams buried under technical debt spend most of their energy maintaining the past instead of building the future.</p><p>That’s negative compounding.</p><p>Automation is positive compounding.</p><p>One expands possibilities.</p><p>The other slowly suffocates them.</p><p>The best engineering organizations understand this intuitively.</p><p>They treat automation as an investment, not an expense.</p><h3>The Hidden Lesson Most Teams Miss</h3><p>Most companies think infrastructure exists to support software.</p><p>Munger would probably argue the opposite.</p><p>Infrastructure exists to support decision-making.</p><p>Good systems reduce mistakes.</p><p>Good systems reduce stress.</p><p>Good systems reduce dependency on individuals.</p><p>And over time, those advantages compound.</p><p>The strongest organizations aren’t built on genius.</p><p>They’re built on structures that continue working even when nobody is being a genius.</p><p>That’s the difference between a fragile operation and an antifragile one.</p><h3>Final Thought</h3><p>Charlie Munger’s advice was remarkably simple:</p><ul><li>Avoid unnecessary complexity.</li><li>Never rely on brilliance when a checklist will do.</li><li>Build huge margins of safety.</li><li>Invest in systems that compound.</li></ul><p>Apply those principles to your DevOps stack and something interesting happens.</p><p>You stop spending your time fighting fires.</p><p>You start building a fortress.</p><p>And the best part?</p><p>You don’t need to be a billionaire investor to think like Charlie Munger.</p><p>You just need to start treating your infrastructure like a long-term asset instead of a daily emergency.</p><p><strong>One question:</strong></p><p>What’s the worst example of “Hero Engineering” you’ve ever seen?</p><p>The story is probably more valuable than any textbook lesson.</p><p>Share it below.</p><p>And if you enjoy exploring the intersection of engineering, psychology, systems thinking, and investing, follow for more deep dives like this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16ef777eb0af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Dashboard Looked Fine. The Query Was Wrong.]]></title>
            <link>https://medium.com/@rameshavutu/promql-aggregation-explained-403cffb54fa5?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/403cffb54fa5</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 05:06:47 GMT</pubDate>
            <atom:updated>2026-07-07T09:34:55.320Z</atom:updated>
            <content:encoded><![CDATA[<h4>Master PromQL | Part 4 of 7</h4><h4>How a tiny PromQL aggregation mistake can produce convincing metrics, hide counter resets, and mislead your entire team.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YvIc8Ow2UfWvxTc-AOQV4w.png" /><figcaption>Image created using Chatgpt</figcaption></figure><p>Most broken PromQL queries don’t throw errors.</p><p>They return numbers.</p><p>That’s what makes them dangerous.</p><p>You can stare at a dashboard every day, make decisions from it, and never realize the result is subtly wrong.</p><p>A classic example? Aggregating before calculating a rate.</p><p>The graph still renders.</p><p>The values still look plausible.</p><p>But the math is no longer telling the truth.</p><p>Part 4 is about aggregation — the art of turning many time series into one useful metric.</p><p>We’ll cover the aggregation operators you’ll use daily, how labels affect your results, and the simple ordering rule that separates accurate dashboards from misleading ones.</p><h3>What Aggregation Actually Does to Your Data</h3><p>Most engineers learn aggregation by memorizing syntax.</p><p>That’s backwards.</p><p>If you understand what aggregation does to labels, the syntax becomes obvious.</p><p>Back in Part 1, we covered a foundational idea:</p><p><strong>Labels are identity.</strong></p><p>A Prometheus time series isn’t just a metric name.</p><p>It’s the metric name <strong>plus its complete set of labels</strong>.</p><p>Aggregation is where that concept starts paying dividends.</p><h3>Aggregation Is Really Label Reduction</h3><p>Think of aggregation as a process of removing dimensions.</p><p>You’re taking multiple time series and collapsing them into fewer time series by dropping one or more labels.</p><p>The labels you keep define the shape of the result.</p><p>The labels you remove are the dimensions you’re aggregating across.</p><p>Here’s a concrete example.</p><p>Before aggregation, Prometheus sees four separate time series:</p><pre>http_requests_total{job=&quot;api&quot;, instance=&quot;10.0.0.1&quot;, status=&quot;200&quot;} = 1200/s<br>http_requests_total{job=&quot;api&quot;, instance=&quot;10.0.0.2&quot;, status=&quot;200&quot;} = 1100/s<br>http_requests_total{job=&quot;api&quot;, instance=&quot;10.0.0.1&quot;, status=&quot;500&quot;} = 12/s<br>http_requests_total{job=&quot;api&quot;, instance=&quot;10.0.0.2&quot;, status=&quot;500&quot;} = 9/s</pre><p>Each unique label combination creates its own series.</p><p>Now aggregate with:</p><pre>sum by (job, status)</pre><p>The result becomes:</p><pre>{job=&quot;api&quot;, status=&quot;200&quot;} = 2300/s<br>{job=&quot;api&quot;, status=&quot;500&quot;} = 21/s</pre><p>Notice what disappeared.</p><p>The instance label is gone.</p><p>Prometheus merged all series that shared the same job and status values, summing their values together.</p><p>You started with four series.</p><p>You ended with two.</p><h3>The Mental Model That Makes Aggregation Click</h3><p>When writing an aggregation query, don’t ask:</p><p><em>“What should I aggregate?”</em></p><p>Ask:</p><p><strong>“What labels do I want to keep in the final result?”</strong></p><p>Those labels go inside by (...).</p><p>Everything else gets collapsed.</p><p>Once you start thinking in terms of label preservation instead of aggregation syntax, every PromQL aggregation becomes dramatically easier to reason about.</p><h3>The 5 Aggregation Operators You’ll Use 99% of the Time</h3><p>PromQL has several aggregation operators.</p><p>In practice, most engineers rely on the same five every day.</p><p>If you understand when each one answers the right question, you’ll solve most aggregation problems without memorizing syntax.</p><h3>1. sum() — &quot;What&#39;s the Total?&quot;</h3><p>This is the aggregation operator you’ll use most often.</p><p>It adds values together across multiple time series.</p><p>Want total API traffic across all instances?</p><pre>sum(rate(http_requests_total[5m]))</pre><p>Want total traffic per service?</p><pre>sum by (job)(rate(http_requests_total[5m]))</pre><p>Want traffic broken down by service and status code?</p><pre>sum by (job, status)(rate(http_requests_total[5m]))</pre><h3>Use sum() when values naturally add together</h3><p>Good examples:</p><ul><li>Request rates</li><li>Network traffic</li><li>Memory consumption</li><li>Disk usage</li><li>Total errors</li></ul><p>Ask yourself:</p><blockquote><em>If I combined all these machines into one giant machine, would adding these numbers make sense?</em></blockquote><p>If the answer is yes, sum() is usually the right choice.</p><p>One common mistake is summing percentages or ratios.</p><p>An error rate of 2% plus an error rate of 3% does <strong>not</strong> equal a 5% error rate.</p><p>That’s usually an avg() problem.</p><h3>2. avg() — &quot;What&#39;s Typical?&quot;</h3><p>Sometimes you don’t care about the total.</p><p>You care about the average experience.</p><p>That’s where avg() comes in.</p><p>For example, average CPU utilization per node:</p><pre>avg by (instance)(<br>  1 - rate(node_cpu_seconds_total{mode=&quot;idle&quot;}[5m])<br>      / rate(node_cpu_seconds_total[5m])<br>)</pre><p>Or average memory usage per namespace:</p><pre>avg by (namespace)(container_memory_working_set_bytes)</pre><h3>Use avg() when you&#39;re measuring behavior per unit</h3><p>Think:</p><ul><li>CPU utilization</li><li>Latency</li><li>Error rates</li><li>Cache hit rates</li><li>Resource efficiency</li></ul><p>A useful mental check:</p><p>sum() tells you how much.</p><p>avg() tells you how typical.</p><p>Those are very different questions.</p><h3>3. max() and min() — &quot;What&#39;s the Worst Case?&quot;</h3><p>Averages can hide problems.</p><p>Imagine nine servers running at 20% CPU and one server running at 100%.</p><p>The average looks healthy.</p><p>One server is on fire.</p><p>That’s why max() exists.</p><pre>max by (instance)(<br>  1 - rate(node_cpu_seconds_total{mode=&quot;idle&quot;}[5m])<br>      / rate(node_cpu_seconds_total[5m])<br>)</pre><p>Likewise, min() helps find the weakest point in a system.</p><pre>min by (instance)(node_memory_MemAvailable_bytes)</pre><h3>Use max() and min() when extremes matter</h3><p>Examples:</p><ul><li>Highest CPU usage</li><li>Highest latency</li><li>Lowest available memory</li><li>Most overloaded node</li><li>Slowest service instance</li></ul><p>A simple rule:</p><p>If your question starts with <strong>“Is any machine…”</strong>, you’re probably looking for max().</p><h3>4. count() — &quot;How Many?&quot;</h3><p>Unlike sum(), count() doesn&#39;t add values.</p><p>It counts time series.</p><p>That’s an important distinction.</p><p>For example:</p><pre>count by (job)(up == 1)</pre><p>This tells you how many healthy instances exist for each job.</p><p>Or:</p><pre>count by (namespace)(kube_pod_info)</pre><p>This tells you how many pods exist in each namespace.</p><h3>Use count() to understand cardinality</h3><p>It’s one of the best debugging tools in PromQL.</p><p>When a query suddenly returns dozens of unexpected lines, count() helps answer:</p><ul><li>How many series exist?</li><li>Which label values are creating them?</li><li>Where is the explosion coming from?</li></ul><p>If you’ve ever stared at a dashboard wondering why there are 47 lines on a graph that should have 3, count() is usually how you find the answer.</p><h3>by vs without: Two Ways to Think About Aggregation</h3><p>Every aggregation operator can be used with either by or without.</p><p>At first they seem different.</p><p>They’re really just two perspectives on the same operation.</p><h3>by(...) — Keep These Labels</h3><pre>sum by (job, status)(<br>  rate(http_requests_total[5m])<br>)</pre><p>The result keeps only:</p><ul><li>job</li><li>status</li></ul><p>Everything else disappears.</p><p>Think of by as:</p><blockquote><em>“These are the dimensions I care about.”</em></blockquote><h3>without(...) — Remove These Labels</h3><pre>sum without (instance)(<br>  rate(http_requests_total[5m])<br>)</pre><p>This removes only instance.</p><p>Every other label remains.</p><p>Think of without as:</p><blockquote><em>“This dimension is noise. Collapse it.”</em></blockquote><h3>Which One Should You Use?</h3><p>Most of the time, use by.</p><p>It’s explicit.</p><p>Future-you can immediately see which labels are expected in the output.</p><p>Use without when there&#39;s a specific label you want to eliminate—usually highly granular labels such as:</p><ul><li>instance</li><li>pod</li><li>container</li></ul><p>A good rule of thumb:</p><p>If you can easily list the labels you want in the result, use by.</p><p>If you only know the label you want to remove, use without.</p><p>Understanding that distinction makes aggregation queries much easier to read, reason about, and debug.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7IE_u825WwflGmhm_OIgDw.png" /></figure><p>Both produce the same output here because the only label not in by (job, status) is instance, and the only label in without (instance) is the same one. For simple cases, choose whichever reads more clearly.</p><h3>The One Rule That Silently Breaks Dashboards</h3><p>If you remember only one thing from this article, make it this:</p><blockquote><strong><em>Always apply </em></strong><strong><em>rate() before </em></strong><strong><em>sum()</em></strong></blockquote><p>Not sometimes.</p><p>Not usually.</p><p>Always.</p><pre><br># ✅ Correct<br>sum(rate(http_requests_total[5m]))<br><br><br># ❌ Wrong<br>rate(sum(http_requests_total)[5m:1m])</pre><p>At first glance, both queries seem like they should answer the same question.</p><p>They don’t.</p><p>And the difference can make your dashboard quietly lie to you.</p><h3>Why This Mistake Is So Dangerous</h3><p>The scary part isn’t that the wrong query crashes.</p><p>It’s that it often produces a number that looks reasonable.</p><p>The graph renders.</p><p>The dashboard loads.</p><p>Nobody notices anything is wrong.</p><p>Until months later.</p><p>To understand why, you need to remember what kind of metric you’re working with.</p><p>http_requests_total is a <strong>counter</strong>.</p><p>Counters only go up.</p><p>Except when they don’t.</p><p>Whenever a process restarts, the counter resets back to zero.</p><p>A server that had processed 50,000 requests suddenly reports:</p><pre>50,000 → 0</pre><p>That drop isn’t a traffic drop.</p><p>It’s just a restart.</p><p>Fortunately, rate() understands this.</p><p>It’s specifically designed to detect and compensate for counter resets.</p><h3>What Happens When You Sum First?</h3><p>Imagine two API instances.</p><pre>Instance A = 50,000<br>Instance B = 51,200</pre><p>Everything looks normal.</p><p>Then Instance B restarts.</p><pre>Instance A = 50,000<br>Instance B = 320</pre><p>Now look at the aggregated counter:</p><pre>Previous total = 101,200<br>Current total  = 50,320</pre><p>From Prometheus’s perspective, the combined counter just dropped by more than 50,000.</p><p>But traffic didn’t drop.</p><p>One machine simply restarted.</p><p>When you run rate() against that already-aggregated value, Prometheus is forced to interpret a counter reset that happened <strong>after aggregation</strong>.</p><p>The result is often:</p><ul><li>Artificial spikes</li><li>Sudden dips</li><li>Incorrect extrapolation</li><li>Misleading dashboards</li></ul><p>In other words:</p><p>The math is now operating on damaged data.</p><h3>Why rate() Must Come First</h3><p>The correct approach is to calculate the rate for each series independently.</p><pre>sum(rate(http_requests_total[5m]))</pre><p>Here’s what happens internally:</p><pre>Instance A → rate = 1,200/s<br>Instance B → rate = 1,100/s</pre><p>Each series handles its own reset correctly.</p><p>Only after those clean rates are calculated do we aggregate:</p><pre>1,200 + 1,100 = 2,300/s</pre><p>Perfect.</p><p>No spikes.</p><p>No dips.</p><p>No corrupted calculations.</p><p>Just the true request rate.</p><h3>A Small Detail That Trips Up Many Engineers</h3><p>Technically, this query won’t even parse:</p><pre>rate(sum(http_requests_total)[5m])</pre><p>PromQL doesn’t allow range selectors directly on aggregation results.</p><p>To express the same idea, you’d need a subquery:</p><pre>rate(sum(http_requests_total)[5m:1m])</pre><p>But don’t let the syntax distract you.</p><p>The real lesson isn’t about parsing.</p><p>It’s about order of operations.</p><p>Even when you make the query syntactically valid, the result is still semantically wrong.</p><h3>The Rule Generalizes Beyond rate()</h3><p>This isn’t just a rate() rule.</p><p>It’s a counter rule.</p><p>Any function that needs to understand counter behavior should run before aggregation.</p><p>That includes:</p><ul><li>rate()</li><li>irate()</li><li>increase()</li></ul><p>The pattern is always the same:</p><pre>Counter function first<br>Aggregation second</pre><p>Or, even simpler:</p><blockquote><strong><em>Transform each series before you combine the series</em></strong></blockquote><p>That’s the mental model.</p><p>And if you follow it consistently, you’ll avoid one of the most common PromQL mistakes engineers make in production.</p><h3>Aggregation Without by or without</h3><p>If you omit both by and without, all labels are dropped and you get a single scalar result.</p><pre># One number: total requests per second across everything<br>sum(rate(http_requests_total[5m]))<br><br># One number: total number of running pods in the cluster<br>count(kube_pod_info)</pre><p>This is often exactly what you want for high-level dashboard panels: “current total RPS”, “total pods”, “total error rate”. One number, no label dimensions.</p><h3>Aggregation and the topk / bottomk Operators</h3><p>Two additional operators are worth knowing for dashboards.</p><p>topk(n, expr) returns the N series with the highest current values:</p><pre># Top 5 endpoints by request rate<br>topk(5, sum by (handler)(rate(http_requests_total[5m])))</pre><p>bottomk(n, expr) returns the N series with the lowest values:</p><pre># 3 nodes with least available memory<br>bottomk(3, node_memory_MemAvailable_bytes)</pre><p>One gotcha: topk and bottomk preserve all labels from the input, unlike aggregation operators that reduce labels. They select a subset of series rather than combining them.</p><h3>Debug Angles</h3><h3>“Why did my labels disappear?”</h3><p>You ran sum(rate(http_requests_total[5m])) and got one number with no labels. Expected: per-job breakdown.</p><p>The cause: you forgot by. Without it, all labels are collapsed.</p><pre># You wrote (collapses everything):<br>sum(rate(http_requests_total[5m]))<br><br># You wanted (one result per job):<br>sum by (job)(rate(http_requests_total[5m]))</pre><h3>“Why am I getting one line instead of many?”</h3><p>Same root cause — over-aggregation. Check your by clause and confirm you included all the dimensions you want to split on.</p><h3>“Why are my numbers wrong?”</h3><p>If the rate looks plausible but slightly off, suspect rate(sum(...)) ordering. Compare:</p><pre># Suspect query (requires subquery syntax to run; wrong semantically either way)<br>rate(sum(http_requests_total)[5m:1m])<br><br># Correct equivalent<br>sum(rate(http_requests_total[5m]))</pre><p>Run both in the Prometheus UI at the same time. If they differ, the ordering is the problem.</p><h3>“Why do I get many-to-one errors after aggregation?&quot;</h3><p>Aggregation changes the label set of a result. If you then try to join two aggregated results, their label sets may no longer match. <br>The fix is to ensure both sides of a join use the same by clause or to use on() to specify the matching labels explicitly.</p><h3>Your Five Aggregation Exercises</h3><pre># 1. Total request rate — one number<br>sum(rate(http_requests_total[5m]))<br><br><br># 2. Per-job request rate — one line per job<br>sum by (job)(rate(http_requests_total[5m]))<br><br><br># 3. Same result using without — drop instance instead of keeping job<br>sum without (instance)(rate(http_requests_total[5m]))<br><br><br># 4. Count running instances per job<br>count by (job)(up == 1)<br><br><br># 5. Demonstrate the wrong order — compare with exercise 2<br># Uses subquery syntax (required to apply rate() to an aggregation result)<br># Does this match exercise 2? If your setup has restarts, it won&#39;t<br>rate(sum by (job)(http_requests_total)[5m:1m])</pre><p>Exercise 5 is the critical one. In a stable environment with no restarts, the difference may be small. Trigger a pod restart and watch the two queries diverge that’s the moment the ordering rule becomes visceral.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kvAo5YdG-i-ZJCI0" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=medium&amp;utm_medium=referral">Tim Mossholder</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What You Should Remember From This Article</h3><p>If everything else fades, remember these three ideas.</p><h3>1. Aggregation Is Really About Labels</h3><p>Aggregation isn’t primarily about numbers.</p><p>It’s about dimensions.</p><p>Every aggregation query is answering one question:</p><blockquote><em>Which labels do I want to keep?</em></blockquote><p>The labels you keep define the result.</p><p>Everything else gets collapsed.</p><p>Once you understand that, aggregation stops feeling like syntax and starts feeling predictable.</p><h3>2. Most of Your Work Happens With Five Operators</h3><p>You don’t need to memorize dozens of functions.</p><p>The vast majority of real-world PromQL aggregation uses just five:</p><ul><li>sum() → total values</li><li>avg() → typical values</li><li>max() → worst-case values</li><li>min() → best-case or least-available values</li><li>count() → number of series</li></ul><p>When choosing between them, focus on the question you’re trying to answer — not the operator itself.</p><p>The right operator usually becomes obvious.</p><h3>3. rate() Comes Before sum()</h3><p>This is the rule that saves dashboards.</p><p>Counter functions such as:</p><ul><li>rate()</li><li>irate()</li><li>increase()</li></ul><p>must operate on individual series first.</p><p>Only then should you aggregate.</p><p>Remember it like this:</p><pre>Transform first.<br>Aggregate second.</pre><p>Or even shorter:</p><pre>rate() → sum()</pre><p>Never the other way around.</p><h3>The Bigger Picture</h3><p>At first glance, aggregation feels like a standalone topic.</p><p>It’s not.</p><p>Everything we’ve covered today is laying the groundwork for what’s next.</p><p>In next Part , we’ll start combining metrics using binary operators and vector matching.</p><p>And here’s the important connection:</p><p>Vector matching works by comparing label sets.</p><p>The same label model that determines how aggregation behaves also determines how joins behave.</p><p>In fact, one of the most common reasons PromQL joins return empty results is incorrect aggregation that removes labels you later need for matching.</p><p>So if today’s lesson was:</p><blockquote><em>“Labels define identity.”</em></blockquote><p>The next lesson is:</p><blockquote><em>“Labels determine whether metrics can be combined.”</em></blockquote><p>And that’s where PromQL starts getting really powerful.</p><h3>📚 Series: Master PromQL</h3><ul><li><strong>Part 1:</strong> ✅ <a href="https://medium.com/beyond-localhost/promql-tutorial-time-series-labels-cardinality-53005d023c0b">The Prometheus Data Model</a></li><li><strong>Part</strong> 2: ✅ <a href="https://medium.com/devops-dev/why-your-promql-queries-return-nothing-607dc599d692">Selecting Data Correctly — Where Most Mistakes Start</a></li><li><strong>Part</strong> 3: ✅ <a href="https://medium.com/@rameshavutu/prometheus-rate-irate-increase-counters-gauges-explained-2e4a0eeedfa3">Rates — Understanding Change Over Time</a></li><li><strong>Part</strong> 4: ✅ Aggregation — Turning Noise into Meaning ← <em>You are here</em></li><li><strong>Part</strong> 5: ⏳ Joins &amp; Vector Matching — The Real Mastery Checkpoint</li><li><strong>Part</strong> 6: ⏳ Real-World Patterns — Apply Everything Together</li><li><strong>Part</strong> 7: ⏳ Debugging &amp; Thinking Like a PromQL Expert</li></ul><p>If this helped clarify how Prometheus metrics actually work, a small action goes a long way in supporting more deep-dive content like this.</p><p>👏 <strong>Clap 50 times if you found it useful</strong> — it helps surface this article to more engineers who are struggling with the same confusion.<br> <a href="https://medium.com/@rameshavutu">🔔 <strong>Follow on Medium</strong> for more practical breakdowns of observability, systems, and real-world debugging patterns.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=403cffb54fa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Prometheus Graphs Are Lying to You]]></title>
            <link>https://medium.com/@rameshavutu/prometheus-rate-irate-increase-counters-gauges-explained-2e4a0eeedfa3?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e4a0eeedfa3</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 07:14:20 GMT</pubDate>
            <atom:updated>2026-06-10T07:24:39.526Z</atom:updated>
            <content:encoded><![CDATA[<h4>Master PromQL | Part 3 of 7</h4><h4>Counters don’t measure rates. Gauges don’t need rate(). And irate() isn&#39;t as real-time as you think.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-jkGnKfa10IT8sMPCbfcbw.png" /></figure><p>Your HTTP request counter just hit <strong>847,293</strong>. Ten seconds later? <strong>847,301</strong>.</p><p>Congrats, you just watched a counter do exactly what it’s supposed to do. Exciting, right?</p><p>Now, try to graph <strong>requests per second</strong> from that data.</p><ul><li>Plot the raw counter → a steadily climbing line that tells you… nothing.</li><li>Divide by time naively → hello, spikes at every counter reset.</li><li>Use irate() blindly → it looks responsive, but hides the truth during scrape gaps.</li></ul><p>There <em>is</em> a right way to do this.</p><p>In this Part, I’ll show you exactly how and, more importantly, <strong>why it works</strong>.</p><h3>Counters and Gauges: The Two Metric Types You Must Distinguish</h3><p>Before you write rate(), you need to understand which metric types exist and why only one of them makes sense with rate functions.</p><h3>What Exactly Is a Counter?</h3><p>A counter is one of the simplest metric types you’ll encounter.</p><p>It has one job: <strong>go up.</strong></p><p>Every time an event happens, the counter increments.</p><p>A request is served? +1.</p><p>An error occurs? +1.</p><p>A few more bytes are sent over the network? The counter increases again.</p><p>The value keeps growing forever — unless the process restarts, in which case it resets back to zero and starts counting again.</p><p>Some common examples:</p><ul><li>http_requests_total → Total requests served since the application started</li><li>http_request_duration_seconds_count → Total number of recorded request durations</li><li>node_network_receive_bytes_total → Total bytes received since the machine booted</li><li>process_cpu_seconds_total → Total CPU time consumed by the process</li></ul><p>So far, so good.</p><p>But here’s the catch:</p><p>A raw counter value is almost useless when you’re trying to understand what’s happening <strong>right now</strong>.</p><p>Imagine seeing this number on a dashboard:</p><p><strong>847,301</strong></p><p>What does it tell you?</p><p>Not much.</p><p>Is your service handling 10 requests per second?</p><p>1,000 requests per second?</p><p>10,000 requests per second?</p><p>You have no idea.</p><p>A counter only tells you how much has happened since the beginning.</p><p>To understand current activity, you need something more important:</p><p><strong>How fast the counter is changing.</strong></p><p>In other words, you need the <strong>rate</strong>.</p><h3>Gauges: Metrics That Tell You What’s Happening Right Now</h3><p>Unlike counters, gauges aren’t trying to keep a running total.</p><p>They’re taking a snapshot.</p><p>A gauge represents the <strong>current state</strong> of something, which means it can move in either direction.</p><p>Memory usage goes up.</p><p>Then it goes down.</p><p>The number of ready pods increases after a deployment.</p><p>Then drops when a node fails.</p><p>That’s exactly what gauges are designed to capture.</p><p>Some common examples:</p><ul><li>node_memory_MemAvailable_bytes → Current available system memory</li><li>container_memory_working_set_bytes → Current memory usage of a container</li><li>kube_deployment_status_replicas_ready → Number of replicas currently ready</li><li>node_load1 → Current 1-minute load average</li></ul><p>The key difference from counters is simple:</p><p><strong>A gauge’s current value is already meaningful.</strong></p><p>If a memory usage graph shows 8 GB, you immediately know how much memory is being used.</p><p>If a deployment shows 3 ready replicas, you know its current state.</p><p>No extra math required.</p><p>And that’s where many people make a costly mistake.</p><p>They learn about rate() for counters and start applying it everywhere.</p><p>Including gauges.</p><p>Almost every time, that’s wrong.</p><p>Why?</p><p>Because gauges already represent the current measurement.</p><p>Applying rate() doesn&#39;t tell you the current value but it tells you how quickly that value is changing, which is usually not what you&#39;re trying to measure.</p><p>For gauges, the value itself is often the answer.</p><h3>Why You Can’t Just Divide a Counter by Time</h3><p>It seems simple, right?</p><p>If you saw <strong>1,000 requests in 5 minutes</strong>, you might think:</p><pre>1,000 ÷ 300 seconds = 3.33 requests/sec</pre><p>Easy. Problem solved.</p><p>Not so fast.</p><h3>The Counter Reset Problem</h3><p>Counters aren’t perfect. They <strong>reset to zero</strong> whenever a process restarts.</p><p>Imagine this:</p><ul><li>Counter at <strong>847,000</strong></li><li>Process restarts → counter goes to <strong>0</strong></li><li>Next scrape shows <strong>1,000</strong></li></ul><p>Do you really think dividing by time will give the right rate?</p><p>Nope. You’ll see either:</p><ul><li>A negative rate</li><li>A massive spike</li></ul><p>Neither reflects reality.</p><h3>The Sampling Problem</h3><p>There’s more. Your data isn’t continuous but it comes in <strong>discrete scrape intervals</strong>.</p><p>Dividing by the time between two samples doesn’t account for:</p><ul><li>Scrape alignment</li><li>Missed scrapes</li><li>Uneven intervals</li></ul><p>This makes naive division unreliable.</p><h3>How Prometheus Fixes It</h3><p>This is why Prometheus has the rate() function.</p><ul><li>Detects counter resets automatically</li><li>Extrapolates a <strong>per-second rate</strong> across all samples in the window</li><li>Gives you an accurate, smooth view of real traffic</li></ul><p>Bottom line: <strong>rate() exists for a reason</strong>. Dividing counters by time won’t cut it.</p><h3>rate() The Standard Tool for Counters</h3><p>When you want to know <strong>how fast something is happening</strong>, rate() is your go-to.</p><p>It calculates the <strong>average per-second increase</strong> of a counter over a specific time window.</p><pre># Requests per second, averaged over the last 5 minutes<br>rate(http_requests_total[5m])</pre><h3>What Actually Happens Behind the Scenes</h3><p>Prometheus doesn’t just divide numbers. It:</p><ol><li>Gathers all the http_requests_total samples from the last 5 minutes</li><li>Detects and accounts for any <strong>counter resets</strong></li><li>Extrapolates a <strong>per-second rate</strong> across the entire window</li></ol><p>The result? A <strong>smooth, averaged rate</strong>.</p><p>Even if traffic spiked 30 seconds ago and dropped, rate(...[5m]) still reflects that spike—just dampened by the averaging window.</p><h3>When to Use rate()</h3><ul><li>Dashboards tracking traffic trends over time</li><li>Alerts on <strong>sustained error rates</strong> (you want stable averages, not noise)</li><li>Anywhere you need a <strong>representative, smoothed rate</strong></li></ul><h3>Picking the Right Window</h3><p>The [5m] in rate(...[5m]) defines <strong>how much history you average</strong>.</p><ul><li><strong>Short windows</strong> ([1m]) → respond quickly, but noisy</li><li><strong>Long windows</strong> ([10m]) → smooth, but slower to react</li></ul><p>Practical guideline from the community:</p><ul><li>Minimum: 2× your <strong>scrape interval</strong> (to ensure at least 2 samples)</li><li>Better: 4× your <strong>scrape interval</strong> → more samples, smoother rate</li></ul><p>Examples:</p><ul><li>1-minute scrape → [4m] is a solid default</li><li>15-second scrape → [1m] works well</li></ul><p>These aren’t hard rules — adjust based on how much smoothing your dashboards need.</p><h3>irate() — The Instantaneous Rate</h3><p>irate() calculates the <strong>instantaneous per-second rate</strong> using only the last two samples in the window.</p><pre># Requests per second, calculated from the last two scrapes only<br>irate(http_requests_total[5m])</pre><p>Despite the [5m] range window, irate() only ever uses the two most recent data points within that window. The window&#39;s only purpose is to bound how far back it looks for those two points, it doesn&#39;t average across the window.</p><p>This gives you a very different result from rate():</p><ul><li><strong>More responsive:</strong> reacts immediately to traffic spikes</li><li><strong>More volatile: </strong>a single unusual scrape interval produces a spike in the graph</li><li><strong>Degrades with scrape gaps:</strong> if one scrape is missed, irate() spans two scrape intervals instead of one; the calculated rate is still approximately correct (larger delta over larger interval), but you lose instantaneous resolution exactly the thing irate() is supposed to provide</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHLRVtujIcM964XPz7l1ag.png" /></figure><p><strong>When to use </strong><strong>irate():</strong></p><ul><li>High-resolution graphs where you need to see spikes clearly</li><li>Short-lived metrics where averaging would mask the signal</li><li>When you <em>know</em> your scrape interval is consistent and gaps are rare</li></ul><p><strong>When NOT to use </strong><strong>irate():</strong></p><ul><li>Alerting rules — a single missed scrape can cause a false positive or false negative</li><li>Long-term trend dashboards — the volatility makes them unreadable</li><li>Any environment with unreliable scrape timing</li></ul><h3>increase() When You Need the Actual Count</h3><p>Sometimes you don’t care about requests <em>per second</em>.</p><p>You just want to know:</p><p><strong>“How many requests did we receive?”</strong></p><p>That’s exactly what increase() is for.</p><p>Instead of returning a rate, it returns the <strong>total increase of a counter</strong> over a time window.</p><pre># Total requests received in the last hour<br>increase(http_requests_total[1h])</pre><p>The result is a count, not a rate.</p><p>If your service handled 250,000 requests during the last hour, increase() returns approximately <strong>250,000</strong>.</p><p>Simple.</p><h3>How Is It Different From rate()?</h3><p>Under the hood, they’re closely related.</p><p>In fact, you can think of increase() as:</p><pre>increase() ≈ rate() × time</pre><p>For example:</p><pre>increase(http_requests_total[1h])<br>≈<br>rate(http_requests_total[1h]) * 3600</pre><p>Both functions:</p><ul><li>Handle counter resets automatically</li><li>Work with the same underlying counter logic</li><li>Use multiple samples within the selected range</li></ul><p>The difference is in the output.</p><ul><li>rate() answers: <strong>&quot;How fast?&quot;</strong></li><li>increase() answers: <strong>&quot;How many?&quot;</strong></li></ul><h3>When Should You Use increase()?</h3><p>Whenever the total count matters more than the rate.</p><p>Common examples include:</p><ul><li>Total requests during the last hour</li><li>Total errors during the last day</li><li>SLO calculations over weeks or months</li><li>Billing dashboards</li><li>Audit and compliance reporting</li></ul><p>If someone asks, <em>“How many?”</em>, increase() is usually the right tool.</p><h3>The Gotcha Most People Miss</h3><p>There’s one subtle detail that surprises a lot of engineers.</p><p>increase() doesn&#39;t simply subtract the first value from the last value.</p><p>It <strong>extrapolates</strong>.</p><p>That means Prometheus estimates what happened across the entire range window, even if it doesn’t have data covering every second of that window.</p><p>Imagine you run:</p><pre>increase(http_requests_total[1h])</pre><p>But Prometheus only has 45 minutes of samples available.</p><p>Instead of returning a count for those 45 minutes, it scales the result to estimate what the full hour would have looked like.</p><p>That’s usually what you want for monitoring.</p><p>But it also means the result is an <strong>estimate</strong>, not an exact accounting figure.</p><h3>The Rule of Thumb</h3><p>Use increase() for monitoring, reporting, and trend analysis.</p><p>But if you’re building systems where every event must be counted precisely billing, audits, financial reporting, or compliance — you’ll want a more authoritative source, such as:</p><ul><li>Recording rules that persist calculated values</li><li>Long-term storage systems</li><li>Event logs or transactional databases</li></ul><p>For operational visibility, increase() is excellent.</p><p>For exact accounting, treat it as an approximation.</p><h3>The Decision Framework</h3><p>Use this as your mental checklist every time you need to query a counter:</p><pre>Is this metric a counter (only goes up) or a gauge (can go up/down)?<br>│<br>├── Gauge → query the raw value directly, no rate function needed<br>│          e.g.: node_memory_MemAvailable_bytes<br>│<br>└── Counter → you need a rate function<br>           │<br>           ├── Do I need a smooth trend for dashboards or alerting?<br>           │   └── Use rate()    e.g.: rate(http_requests_total[5m])<br>           │<br>           ├── Do I need to see instantaneous spikes clearly?<br>           │   └── Use irate()   e.g.: irate(http_requests_total[5m])<br>           │<br>           └── Do I need a total count over a period?<br>               └── Use increase()  e.g.: increase(http_requests_total[1h])</pre><p>The most important branch: <strong>don’t apply </strong><strong>rate() or </strong><strong>irate() to a gauge</strong>. <br>It produces results that look numeric but are meaningless. Memory usage doesn&#39;t have a &quot;rate of increase per second&quot; in the same sense. <br>what you want is the current value or maybe a delta, not a rate.</p><h3>Counter Resets: What Actually Happens</h3><p>When a Prometheus-instrumented process restarts, its in-process counters reset to zero. The next scrape will show a value much lower than the previous one.</p><p>Without reset detection, calculating the rate across a restart would give you a massive negative delta (or a nonsensical large rate). rate() and irate() both detect resets automatically and adjust for them, Prometheus documents this behaviour without specifying the internal algorithm.</p><p>Here’s what that looks like in practice:</p><pre>Scrape 1: http_requests_total = 50,000<br>Scrape 2: http_requests_total = 51,200   ← normal, delta = 1,200<br>Scrape 3: http_requests_total = 52,400   ← normal, delta = 1,200<br>Scrape 4: http_requests_total = 320      ← RESET (process restarted)<br>Scrape 5: http_requests_total = 1,520    ← normal post-restart, delta = 1,200</pre><p>When rate() encounters the drop from 52,400 to 320 between scrapes 3 and 4, it automatically adjusts for the reset. <br>The result is a rate that correctly reflects post-restart traffic not a spike caused by the discontinuity.</p><p>Prometheus handles this automatically, you don’t need to write reset-detection logic yourself. The behaviour is documented; the exact internal mechanism isn’t something you need to reason about when writing queries.</p><p>This is precise enough for dashboards and most alerting. For exact accounting across restarts, you’d need a persistent counter mechanism outside of Prometheus.</p><h3>Debug Angle: Spikes and Misleading Graphs</h3><h3>Spike at process restart</h3><p><strong>Symptom:</strong> A massive spike in your rate graph exactly when a service restarted.<br><strong>Cause:</strong> You’re using a function or expression that doesn’t handle counter resets — for example, dividing raw counter values directly.<br><strong>Fix:</strong> Use rate() instead. It handles resets correctly.</p><h3>Unexpected flatness or coarse spikes with irate()</h3><p><strong>Symptom:</strong> Your irate() graph looks less responsive than expected, or shows coarser spikes than you&#39;d see at the actual scrape interval.<br><strong>Cause:</strong> A scrape was missed. irate() spans two scrape intervals to find its two samples — the calculated rate is still approximately correct in value, but you&#39;ve lost the instantaneous precision that makes irate() useful. Its entire advantage is the per-scrape-interval resolution; a missed scrape eliminates that.<br><strong>Fix:</strong> Switch to rate() for any graph where scrape reliability isn&#39;t guaranteed. Reserve irate() for high-frequency, well-scraped environments where you specifically need sub-window spike visibility.</p><h3>Flat line at zero when you expect a rate</h3><p><strong>Symptom:</strong> rate(my_metric[5m]) returns a flat zero even though you know traffic is flowing.<br><strong>Cause:</strong> The metric might be a gauge, not a counter. A gauge that isn&#39;t changing returns zero rate.<br><strong>Fix:</strong> Check the metric&#39;s type. Run the raw metric name and look at the values over time — if they go up and down, it&#39;s a gauge. Query it directly. If they only go up, it&#39;s a counter and rate() should work.</p><h3>The “rate of a rate” mistake</h3><pre># Type error: rate() expects a range vector, not an instant vector<br>rate(rate(http_requests_total[5m])[5m])</pre><p>This is a hard type error, Prometheus will reject it at parse time. <br>The inner rate() returns an instant vector; a range selector [5m] cannot be applied to an instant vector. <br>Beyond the type system, it wouldn&#39;t make conceptual sense either: rate() values go up and down (traffic can decrease), so they&#39;re not counters and have no meaningful &quot;rate of change.&quot;</p><h3>Your Five Rate Exercises</h3><p>Run these in order each one builds on the previous:</p><pre># 1. Raw counter — watch it only go up<br>http_requests_total{job=&quot;&lt;your-service&gt;&quot;}<br><br># 2. Rate — now you can see per-second traffic<br>rate(http_requests_total{job=&quot;&lt;your-service&gt;&quot;}[5m])<br><br># 3. irate — compare side-by-side with rate() in Grafana<br># Notice: more volatile, reacts faster to spikes<br>irate(http_requests_total{job=&quot;&lt;your-service&gt;&quot;}[5m])<br><br># 4. Increase — total requests in the last hour<br>increase(http_requests_total{job=&quot;&lt;your-service&gt;&quot;}[1h])<br><br># 5. Wrong: rate on a gauge — see what happens<br>rate(node_memory_MemAvailable_bytes[5m])<br><br># Compare with the correct query:<br>node_memory_MemAvailable_bytes</pre><p>Exercise 5 is the most instructive. Running rate() on a gauge metric returns a value — Prometheus won&#39;t error — but the number is meaningless noise. This is why knowing the metric type before choosing a function matters.</p><h3>What You Now Know</h3><p><strong>Counters vs. gauges:</strong></p><ul><li>Counter → only goes up; reset on restart. Use rate(), irate(), or increase().</li><li>Gauge → can go up or down. Query the raw value directly.</li></ul><p><strong>The three rate functions:</strong></p><ul><li>rate() → smooth average per-second rate across the window. Default choice.</li><li>irate() → instantaneous rate from last two samples. Responsive but volatile.</li><li>increase() → total count over the window. For absolute reporting, not rates.</li></ul><p><strong>Counter resets:</strong></p><ul><li>Both rate() and irate() detect and handle resets automatically.</li><li>A drop in counter value = restart; Prometheus doesn’t treat it as negative delta.</li></ul><p><strong>Window size matters:</strong></p><ul><li>Too small → not enough samples, empty or unstable results.</li><li>Too large → over-smoothed, slow to reflect real changes.</li><li>Community guideline: 2× scrape interval is the minimum; 4× gives a stable result for most dashboards.</li></ul><h3>What’s Next: Aggregation</h3><p>You now have per-second rates per time series. In Part4 we covers how to collapse those into meaningful summaries: total rate across a service, average across instances, or a breakdown by status code.</p><h3>📚 Series: Master PromQL</h3><ul><li><strong>Part 1:</strong> ✅ <a href="https://medium.com/beyond-localhost/promql-tutorial-time-series-labels-cardinality-53005d023c0b">The Prometheus Data Model</a></li><li><strong>Part</strong> 2: ✅ <a href="https://medium.com/devops-dev/why-your-promql-queries-return-nothing-607dc599d692">Selecting Data Correctly — Where Most Mistakes Start</a></li><li><strong>Part</strong> 3: ✅ Rates — Understanding Change Over Time ← <em>You are here</em></li><li><strong>Part</strong> 4: ⏳ Aggregation — Turning Noise into Meaning</li><li><strong>Part</strong> 5: ⏳ Joins &amp; Vector Matching — The Real Mastery Checkpoint</li><li><strong>Part</strong> 6: ⏳ Real-World Patterns — Apply Everything Together</li><li><strong>Part</strong> 7: ⏳ Debugging &amp; Thinking Like a PromQL Expert</li></ul><p>If this helped clarify how Prometheus metrics actually work, a small action goes a long way in supporting more deep-dive content like this.</p><p>👏 <strong>Clap 50 times if you found it useful</strong> — it helps surface this article to more engineers who are struggling with the same confusion.<br> <a href="https://medium.com/@rameshavutu">🔔 <strong>Follow on Medium</strong> for more practical breakdowns of observability, systems, and real-world debugging patterns.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e4a0eeedfa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon EKS 1.36 Upgrade: What Will Break and How to Fix It Before It Does]]></title>
            <link>https://medium.com/@rameshavutu/amazon-eks-1-36-upgrade-c534973172b6?source=rss-c3bcf684ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/c534973172b6</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Ramesh | The Platform Engineer]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 05:19:36 GMT</pubDate>
            <atom:updated>2026-06-26T05:02:30.896Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Upgrade Guide AWS Forgot to Write</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*2uMG5iBOe8zTeA9u6UMDgw.png" /><figcaption>source:<a href="https://kubernetes.io/blog/2026/04/22/kubernetes-v1-36-release/k8s-v1.36.svg"> k8s 1.36</a></figcaption></figure><p>AWS just released Amazon EKS 1.36.</p><p>Which means Kubernetes 1.35 has officially entered its countdown phase.</p><p>If you’ve managed enough production clusters, you know the upgrade itself usually isn’t the scary part.</p><p>The surprises are.</p><p>The deprecated APIs you forgot existed.</p><p>The add-on version mismatch that breaks deployments.</p><p>The GitOps pipeline that suddenly starts failing after what looked like a routine version bump.</p><p>AWS provides all the information you need to upgrade safely.</p><p>The problem is that it’s spread across multiple documentation pages, release notes, console guides, and changelogs that most engineers only discover after something goes wrong.</p><p>This guide brings everything together in one place.</p><p>You’ll learn what’s new in EKS 1.36, which changes can break existing workloads, and the exact upgrade sequence to move your clusters forward without creating a late-night incident.</p><p>We’ll cover:</p><ul><li>Control plane upgrades</li><li>Managed node groups</li><li>Self-managed nodes</li><li>EKS add-on updates</li><li>Deprecated and removed APIs</li><li>Pre-upgrade validation checks</li></ul><p>EKS Cluster Insights: It scans your cluster for upgrade blockers before you start and can save hours of troubleshooting later.</p><p>If you’re planning an EKS 1.36 upgrade, run those checks first.</p><p>Your future self will thank you.</p><p><strong>TL;DR — the four things that matter:</strong></p><ul><li><strong>New in 1.36:</strong> User Namespaces GA, Mutating Admission Policies, In-Place Pod Scaling, Resource Health Status — <a href="https://medium.com/stackademic/kubernetes-1-36-haru-9a1b4927fe0f">full feature breakdown here</a></li><li><strong>What breaks:</strong> flowcontrol.apiserver.k8s.io/v1beta2 removed — silent failure in ArgoCD/Helm if not fixed first</li><li><strong>Upgrade order:</strong> Control plane → Add-ons (kube-proxy → vpc-cni → CoreDNS) → Node groups</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*LV7cwIwpvwPYjPlt9J3Z0w.png" /></figure><ul><li><strong>Run first:</strong> EKS Cluster Insights + the pre-flight script at the bottom of this article</li></ul><blockquote><strong>Already familiar with how Kubernetes API deprecations work?</strong> If you haven’t read <a href="https://medium.com/@rameshavutu/kubernetes-upgrade-deprecated-apis-kubent-pluto-30c917c77835">The Kubernetes Upgrade Mistake That Breaks Production Every Time</a>, start there first — it covers the full deprecation lifecycle and the kubent + Pluto toolchain for catching removed APIs before they catch you. This guide builds on that foundation with EKS-specific steps for the 1.35 → 1.36 path.</blockquote><h3>Before we continue</h3><p>If you found this story insightful, …</p><p>👏 <strong>Clap 50 times </strong>(yes, you can, simply hold the button), it will help me a lot. Medium’s algorithm favors this, increasing visibility to others who then discover the article.</p><p>🔔 <a href="https://medium.com/@rameshavutu">Follow me on Medium and subscribe to get my latest articles straight to your inbox.</a></p><h3>What’s Actually New in EKS 1.36</h3><p>Before you touch anything, it helps to know <em>why</em> you’re upgrading, not just that the version clock is ticking.</p><h3>Kubernetes 1.36 Upstream Changes</h3><p>AWS highlighted four features in the EKS Distro 1.36 release:</p><ul><li><strong>User Namespaces (GA): </strong>Maps container root to an unprivileged host user. Opt-in via hostUsers: false in the pod spec. Biggest security win in this release.</li><li><strong>Mutating Admission Policies: </strong>CEL-based mutations directly in the API server. No webhook, no TLS certs to rotate, no external server to operate.</li><li><strong>In-Place Pod Vertical Scaling:</strong> Resize CPU/memory on running pods without a restart.</li><li><strong>Resource Health Status:</strong> GPU and accelerator health surfaces in Pod status. No more silent hardware crash loops.</li></ul><p>For the full breakdown of these plus HPA scale-to-zero, Node Log Query GA, PSI metrics, and SELinux volume labeling see: <a href="https://medium.com/stackademic/kubernetes-1-36-haru-9a1b4927fe0f">Kubernetes 1.36 Just Killed Webhooks (And 7 Other Massive Changes)</a>.</p><blockquote><strong>EKS users with Ingress NGINX: action required.</strong> Ingress NGINX was retired on March 24, 2026 — no new releases, no security fixes. If you’re running it on EKS today, this upgrade is a good forcing function to plan your migration to Gateway API (Envoy Gateway or Cilium are the leading replacements). It won’t break on day one of the upgrade, but you’re now running unsupported routing infrastructure.</blockquote><h3>EKS-Specific Changes in 1.36</h3><ul><li><strong>Updated default AMI: </strong>Amazon Linux 2023 (AL2023) is now the default AMI for new managed node groups. AL2 nodes continue to work but the new default matters for auto-provisioned clusters.</li><li><strong>CoreDNS 1.11.x: </strong>Updated from 1.10.x. Compatible but requires add-on version update.</li><li><strong>kube-proxy update: </strong>Aligned to 1.36 API. Must upgrade alongside control plane.</li><li><strong>VPC CNI 1.19.x: </strong>New minor version with IPv6 prefix delegation improvements.</li><li><strong>Pod Identity Agent: </strong>If you’re using EKS Pod Identity (not IRSA), verify the agent version compatibility.</li></ul><h3>Breaking Changes: Read This Before You Touch Anything</h3><p>Most EKS upgrade incidents don’t happen during the upgrade. <br>They happen because something deprecated slipped through unnoticed and breaks silently the moment the new API server starts rejecting it.</p><h3>Removed and Deprecated APIs to Check</h3><p>Here’s the honest framing: these APIs were deprecated across earlier Kubernetes releases and have been accumulating in codebases ever since. If you’ve been upgrading diligently, you may have already dealt with them. If you’re jumping from an older cluster or have long-lived Helm charts, run the scan, don’t guess.</p><p><strong>APIs commonly found in older EKS clusters that no longer work in recent versions:</strong></p><ul><li>flowcontrol.apiserver.k8s.io/v1beta2 (FlowSchema, PriorityLevelConfiguration) removed in 1.29. Still shows up in third-party Helm charts that haven&#39;t been updated.</li><li>autoscaling/v2beta2 (HorizontalPodAutoscaler) removed in 1.26. Older monitoring stacks sometimes ship this.</li><li>networking.k8s.io/v1beta1 (Ingress) removed in 1.22, but surprisingly still found in legacy GitOps repos.</li></ul><p>The point isn’t the specific removal version — it’s that <strong>any</strong> of these in your manifests or Helm releases will cause apply failures after the upgrade. The tools below catch them all in one pass.</p><p>Don’t chase these manually with grep. Use <strong>kubent</strong> and <strong>Pluto</strong> they scan your live cluster, Helm releases, and source manifests in one pass and tell you exactly what will break at your target version:</p><pre># kubent: scan live cluster against 1.36 target, exit non-zero for CI<br>kubent --target-version 1.36 --exit-error --output json<br><br># Pluto: scan manifest files and Helm charts<br>pluto detect-files -d ./k8s/ --target-versions k8s=v1.36.0<br>pluto detect-all-in-cluster -o wide --target-versions k8s=v1.36.0<br><br># Render Helm chart and pipe directly to Pluto (no cluster needed)<br>helm template my-release ./my-chart | pluto detect -</pre><h3>Pre-Flight Checklist</h3><p>Run through this before initiating the upgrade. Don’t skip items each takes 2 minutes and saves potentially hours.</p><p><strong>Start with EKS Cluster Insights</strong> — AWS added a built-in upgrade blocker scanner in the EKS console. Under your cluster → “<strong>Upgrade Insights</strong>” tab, it automatically flags deprecated API usage, incompatible add-on versions, and node group issues. Run it first then it catches most of the manual checks below automatically.</p><pre># Run cluster insights via CLI<br>aws eks list-insights \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --filter &#39;{&quot;categories&quot;:[&quot;UPGRADE_READINESS&quot;]}&#39; \<br>  --query &quot;insights[?insightStatus.status!=&#39;PASSING&#39;].{name:name,status:insightStatus.status,description:insightStatus.reason}&quot;</pre><p><strong>Step 1 — Confirm your cluster is on 1.35:</strong></p><pre>aws eks describe-cluster --name &lt;cluster-name&gt; --query &quot;cluster.version&quot; --output text</pre><p><strong>Step 2 — Check add-on compatibility for 1.36:</strong></p><pre>aws eks describe-addon-versions \<br>  --kubernetes-version 1.36 \<br>  --query &quot;addons[?contains([&#39;kube-proxy&#39;,&#39;vpc-cni&#39;,&#39;coredns&#39;], addonName)].{name:addonName, recommended:addonVersions[0].addonVersion}&quot;</pre><p><strong>Step 3 — Scan for deprecated APIs</strong> (don’t use grep — use the right tools):</p><pre>kubent --target-version 1.36 --exit-error<br>pluto detect-all-in-cluster --target-versions k8s=v1.36.0</pre><p><strong>Step 4 — Verify all node groups are ACTIVE</strong> (upgrade will fail if any are in UPDATING state):</p><pre>aws eks describe-nodegroup \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --nodegroup-name &lt;nodegroup-name&gt; \<br>  --query &quot;nodegroup.{status:status, version:version, releaseVersion:releaseVersion}&quot;</pre><p><strong>Terraform users</strong> — If you manage EKS with Terraform, check that your aws_eks_cluster resource has version = &quot;1.36&quot; ready to apply and that your aws_eks_node_group resources will accept the updated release_version after control plane upgrade.</p><pre>resource &quot;aws_eks_cluster&quot; &quot;main&quot; {<br>  name    = &quot;my-cluster&quot;<br>  version = &quot;1.36&quot;  # Update this<br><br>  # ... rest of config<br>}<br><br>resource &quot;aws_eks_node_group&quot; &quot;workers&quot; {<br>  cluster_name    = aws_eks_cluster.main.name<br>  node_group_name = &quot;workers&quot;<br>  version         = &quot;1.36&quot;  # Must match cluster version<br><br>  # release_version is optional — omit to get latest recommended AMI<br>  # release_version = &quot;1.36.0-20260501&quot;<br>}</pre><h3>The Upgrade Sequence</h3><p>EKS upgrades follow a strict order: control plane first, then add-ons, then nodes. Reversing any step will cause issues.</p><pre>Control Plane → Core Add-ons → Managed Node Groups → Self-Managed Nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*naqxYlnGb35DLgXHD5fdDQ.png" /></figure><h3>Step 1: Upgrade the Control Plane</h3><pre># Using AWS CLI<br>aws eks update-cluster-version \<br>  --name &lt;cluster-name&gt; \<br>  --kubernetes-version 1.36<br><br># Watch the upgrade status<br>aws eks describe-cluster \<br>  --name &lt;cluster-name&gt; \<br>  --query &quot;cluster.status&quot;<br><br># Wait for ACTIVE status (typically 10-20 minutes)<br>while true; do<br>  STATUS=$(aws eks describe-cluster --name &lt;cluster-name&gt; --query &quot;cluster.status&quot; --output text)<br>  echo &quot;$(date): $STATUS&quot;<br>  [ &quot;$STATUS&quot; = &quot;ACTIVE&quot; ] &amp;&amp; break<br>  sleep 30<br>done</pre><p><strong>Terraform:</strong></p><pre># Update version in your config, then:<br>terraform plan -target=aws_eks_cluster.main<br>terraform apply -target=aws_eks_cluster.main</pre><p>During control plane upgrade, your worker nodes continue serving traffic. The API server briefly experiences rolling restarts, kubectl commands may time out for 30–60 seconds. This is expected.</p><h3>Step 2: Update Core Add-ons</h3><p>After the control plane is on 1.36, update your add-ons. Do these in order: CoreDNS last, since it’s most critical for in-cluster DNS.</p><pre># Get the recommended add-on versions for 1.36<br>aws eks describe-addon-versions \<br>  --kubernetes-version 1.36 \<br>  --query &quot;addons[?contains([&#39;kube-proxy&#39;,&#39;vpc-cni&#39;,&#39;coredns&#39;], addonName)].{name:addonName, version:addonVersions[0].addonVersion}&quot;</pre><h4><strong>Update kube-proxy first</strong></h4><pre>aws eks update-addon \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --addon-name kube-proxy \<br>  --addon-version v1.36.0-eksbuild.7 \<br>  --resolve-conflicts OVERWRITE</pre><h4>Update VPC CNI</h4><pre>aws eks update-addon \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --addon-name vpc-cni \<br>  --addon-version v1.22.1-eksbuild.2 \<br>  --resolve-conflicts OVERWRITE</pre><h4><strong>Update CoreDNS last</strong></h4><pre>aws eks update-addon \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --addon-name coredns \<br>  --addon-version v1.14.3-eksbuild.2 \<br>  --resolve-conflicts OVERWRITE</pre><h4><strong>Check add-on status</strong></h4><pre>aws eks describe-addon \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --addon-name coredns \<br>  --query &quot;addon.status&quot;</pre><p><strong>Note:</strong> Use --resolve-conflicts OVERWRITE only if you haven&#39;t customized the add-on configuration. If you have custom CoreDNS Corefile entries or custom VPC CNI environment variables, use PRESERVE instead and verify manually after the update.</p><h3>Step 3: Upgrade Managed Node Groups</h3><p>EKS managed node groups support in-place version upgrades with rolling replacement. The default behavior replaces one node at a time, honoring PodDisruptionBudgets.</p><pre># Upgrade the node group (triggers rolling AMI replacement)<br>aws eks update-nodegroup-version \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --nodegroup-name &lt;nodegroup-name&gt;<br><br># Watch node group update status<br>aws eks describe-nodegroup \<br>  --cluster-name &lt;cluster-name&gt; \<br>  --nodegroup-name &lt;nodegroup-name&gt; \<br>  --query &quot;nodegroup.{status:status, health:health}&quot;<br><br># Verify nodes are running 1.36<br>kubectl get nodes -o wide</pre><p>The managed node group upgrade performs these steps automatically:</p><ol><li>Launches new node with 1.36 AMI</li><li>Cordons the old node</li><li>Drains pods (respects PDBs)</li><li>Terminates old node</li><li>Repeats until all nodes are updated</li></ol><p><strong>If a drain stalls:</strong> Check for pods violating PDBs or pods without a controller (standalone pods won’t migrate).</p><pre># Find pods blocking drain<br>kubectl get pods --all-namespaces --field-selector=status.phase=Running \<br>  -o jsonpath=&#39;{range .items[*]}{.metadata.namespace}/{.metadata.name} — node: {.spec.nodeName}{&quot;\n&quot;}{end}&#39; | \<br>  grep &lt;stalled-node-name&gt;</pre><h3>Step 4: Self-Managed Nodes (If Applicable)</h3><p>For self-managed nodes (Launch Templates, Karpenter, custom ASGs), the process is manual:</p><pre># 1. Update your Launch Template to use the new EKS-optimized AMI for 1.36<br># Find the latest AMI ID:<br>aws ssm get-parameter \<br>  --name /aws/service/eks/optimized-ami/1.36/amazon-linux-2023/x86_64/standard/recommended/image_id \<br>  --query &quot;Parameter.Value&quot; --output text<br><br># 2. Cordon the old node<br>kubectl cordon &lt;node-name&gt;<br><br># 3. Drain the node (respect PDBs)<br>kubectl drain &lt;node-name&gt; \<br>  --ignore-daemonsets \<br>  --delete-emptydir-data \<br>  --timeout=300s<br><br># 4. Terminate the EC2 instance (ASG will launch replacement with new template)<br>aws ec2 terminate-instances --instance-ids &lt;instance-id&gt;<br><br># 5. Verify new node joins at 1.36<br>kubectl get nodes --watch</pre><blockquote><strong>Karpenter users: </strong>Update your EC2NodeClass to remove any pinned AMI selectors that would pull 1.35 images, then let Karpenter handle rolling replacement via kubectl rollout restart on your deployments.</blockquote><h3>Post-Upgrade Validation</h3><p>Don’t mark the upgrade complete until you’ve run through these.</p><pre># 1. Verify all nodes are on 1.36<br>kubectl get nodes -o custom-columns=&#39;NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,STATUS:.status.conditions[-1].type&#39;<br><br># 2. Check for any pods in non-Running state<br>kubectl get pods --all-namespaces --field-selector=&#39;status.phase!=Running,status.phase!=Succeeded&#39; <br><br># 3. Verify add-on pods are healthy<br>kubectl get pods -n kube-system<br><br># 4. Test in-cluster DNS resolution<br>kubectl run dns-test --image=busybox --restart=Never -it --rm \<br>  -- nslookup kubernetes.default.svc.cluster.local<br><br># 5. Check for deprecated API warnings in audit logs<br>aws logs filter-log-events \<br>  --log-group-name /aws/eks/&lt;cluster-name&gt;/cluster \<br>  --filter-pattern &quot;deprecated&quot; \<br>  --start-time $(date -d &#39;1 hour ago&#39; +%s000) \<br>  --query &#39;events[].message&#39; \<br>  --output text | head -20</pre><h3>Common Problems and How to Fix Them:</h3><h3>Unable to drain node PodDisruptionBudget blocks eviction</h3><p>A PDB with minAvailable equal to the total number of replicas will block drain forever.</p><pre># Find the blocking PDB<br>kubectl get pdb --all-namespaces<br>kubectl describe pdb &lt;pdb-name&gt; -n &lt;namespace&gt;<br><br># Temporary fix (use only in maintenance windows):<br>kubectl patch pdb &lt;pdb-name&gt; -n &lt;namespace&gt; \<br>  --type=&#39;json&#39; \<br>  -p=&#39;[{&quot;op&quot;:&quot;replace&quot;,&quot;path&quot;:&quot;/spec/minAvailable&quot;,&quot;value&quot;:0}]&#39;<br><br># Restore after upgrade<br>kubectl patch pdb &lt;pdb-name&gt; -n &lt;namespace&gt; \<br>  --type=&#39;json&#39; \<br>  -p=&#39;[{&quot;op&quot;:&quot;replace&quot;,&quot;path&quot;:&quot;/spec/minAvailable&quot;,&quot;value&quot;:&lt;original-value&gt;}]&#39;</pre><h3>CoreDNS pods crashing after upgrade</h3><p>Usually caused by custom Corefile entries that conflict with the new default config.</p><pre># Check CoreDNS logs<br>kubectl logs -n kube-system -l k8s-app=kube-dns --previous<br><br># Compare your Corefile with the default for 1.36<br>kubectl get configmap coredns -n kube-system -o yaml<br><br># If custom entries are lost, re-apply your customization<br>kubectl edit configmap coredns -n kube-system</pre><h3>ArgoCD showing sync errors after upgrade</h3><p>If ArgoCD is trying to apply resources with removed API versions, you’ll see sync errors immediately.</p><pre># Check ArgoCD application status<br>kubectl get applications -n argocd<br><br># Look for &quot;resource not found&quot; errors pointing to removed APIs<br>kubectl describe application &lt;app-name&gt; -n argocd | grep -A5 &quot;ComparisonError\|SyncError&quot;</pre><p>Fix: Update the source manifests in your GitOps repo to use v1 API versions or latest versions, then trigger a sync.</p><h3>Summary</h3><p>The EKS 1.35 → 1.36 upgrade is straightforward if you follow the sequence and don’t skip the pre-flight checks. The control plane is the riskiest step to schedule (brief API server restarts), but the actual risk is in the preparation. Deprecated APIs and misconfigured PDBs cause more incidents than the upgrade itself.</p><p><strong>The short version:</strong></p><ol><li>Audit and fix v1beta2 FlowControl API usage</li><li>Upgrade control plane → add-ons → node groups (in that order)</li><li>Validate with the post-upgrade checks</li></ol><p>If you’re running multiple clusters, establish this as a runbook, the commands are identical across clusters, only the cluster name changes.</p><p><a href="https://medium.com/@rameshavutu/mcp-servers-for-devops-b29ea0742ee5">Build an AI Kubernetes Incident Debugger in Python</a></p><p><a href="https://medium.com/@rameshavutu"><strong>If you work on Kubernetes platforms or production debugging, follow for more deep dives into Linux internals, observability, and SRE tooling.</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c534973172b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>