When the Kubernetes API went silent: A three-layer networking mystery

extract

When the Kubernetes API went silent: A three-layer networking mystery Akash Thakur Sr. Technical Solutions Engineer Every support engineer has a case that stays with them. The one that comes up at team lunches. The one where, even after it's resolved, you find yourself staring at a packet capture at midnight just to make sure you really understand what happened. This is that case. It started in January 2026 with a support ticket that read deceptively simply: pods are intermittently failing to reach the Kubernetes API server. The error was a familiar-looking timeout. The customer had already ruled out their own operators. What none of us knew yet was that we were looking at three separate bugs — in three different layers of the stack — all conspiring to produce the same symptom at the same time. By the time it was over, around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt. In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself. This is the story of how we got there. Part one: The alarm bell that wouldn't stop The customer ran a large multi-cluster GKE environment in Google Cloud's europe-west region. Their clusters used Private Service Connect — Google Cloud's mechanism for exposing the GKE control plane privately — and had a rich ecosystem of system-level workloads: Twistlock security scanners, External Secrets Operator for secrets management, Cilium for network policy, as well as several internal Kubernetes operators. The first errors appeared in one of its user acceptance testing (UAT) clusters in europe-west, and a few days later in a second cluster in the same region. They looked like this: Application logs — pods across multiple namespaces Initially the errors were intermittent — spikes in the logs, not a hard outage. The customer suspected their own code. Their team had recently deployed a custom operator built with the kopf framework, and it was the first thing they looked at. But the logs showed the errors had started eight days before the operator was ever deployed. When they shut it down completely, the timeouts kept coming. The issue persists even after completely scaling down the External Secrets Operator. The errors are appearing from customer workloads, and system-level pods we haven't touched. The growing list of suspects When a case like this arrives, the first instinct is to ask: what changed? The customer's timeline gave us several candidates, and we worked through them methodically. - The custom operator (kopf): Ruled out immediately — errors predated deployment by a week and continued after shutdown. - External Secrets Operator: Removed entirely on customer request. Timeouts continued in customer workloads that relied on Kubernetes control plane (KCP) connectivity, and cluster-internal system pods. - Security agents: Uninstalled. Same result. We were now watching system-level Kubernetes components fail to reach their own control plane. - conntrack table exhaustion: One theory was that too many half-open connections were filling the OS connection tracking table on the control plane nodes. We checked. The conntrack table was at roughly 50% capacity — of a default maximum of 1.3M entries. Not even close to exhaustion. - Control plane overload: We scaled up the KCP VM size. kube-apiserver latencies looked healthy. No improvement on the timeout side. What made this case genuinely strange was the scope. We confirmed the timeouts were happening across five different clusters, affecting DaemonSets, Deployments, and system pods alike. External clients — developer laptops, GitHub Actions runners — were also hitting the same wall. This was not a broken workload. This was something wrong with the path between everything and the API server. The timeline so far - Day 1: First I/O timeout errors appear (One of the UAT clusters in europe-west3. Intermittent, easy to dismiss.) - Day 10: Errors become persistent (Frequency increases significantly. The issue is no longer dismissible.) - Day 15: Case escalated — P0 bug filed (Multiple clusters confirmed affected. ~800 developers blocked. Production releases are frozen. Formal escalation begins.) - Day 20: All obvious suspects ruled out (Custom operator, security agents, conntrack exhaustion, control plane resizing — all cleared.) - Day 22: Packet captures collected (Captures taken simultaneously at the control-plane node NIC and from inside the kube-api container. The investigation enters its most critical phase.) With the obvious suspects cleared, and the scope undeniably wide, we did what you do when the application layer has nothing left to say: We went to the packets. Part two: Tracing the ghost in the wire To understand what the packet captures revealed, you first need to understand the architecture that a GKE pod traverses when it makes an API call. In a Private Service Connect-based GKE cluster, the path is deceptively long for something that looks like a local connection. (Note for formatting: Consider including the Pod → kube-apiserver flow diagram here: Pod -> Worker Node -> PSC NAT+Proxy -> ILB -> KCP VM NIC -> kube-apiserver) The key component here is Private Service Connect. When a pod dials 1.2.3.4:443, it is not actually going directly to the control plane; that IP is a virtual endpoint. The connection is intercepted by Google's networking fabric, NAT'd through a Private Service Connect endpoint, passed through an internal passthrough Network Load Balancer, and ultimately delivered to the KCP VM, where Envoy acts as a local proxy in front of the actual kube-apiserver process. Private Service Connect adds a proxy protocol header to each new connection, so the kube-apiserver can see the real client IP. This will turn out to be important later. What the packets revealed When we captured simultaneously at the KCP VM's NIC (the PIGv interface, Google's programmable virtual NIC) and from inside the kube-apiserver container, we saw something striking. SYN packets from the pods were arriving at the KCP VM's NIC. They were getting that far. But instead of receiving a SYN-ACK in response — the expected second step of a TCP handshake — the client was getting back an out-of-place ACK. Not an RST. Not silence. An ACK that logically belonged to a different, earlier connection. Wireshark, the network protocol analyzer, was flagging these streams with a label that initially seemed innocuous: "TCP Port numbers reused." Wireshark — packet stream showing the failed handshake The pattern was clear. Something was causing the KCP VM to respond to a fresh SYN with an ACK that belonged to a different connection that had ended minutes ago. The old connection had used the same source port (53426 in the example above), and the KCP VM had never fully let go of it. "The KCP server sends FIN,ACK; client responds with FIN,ACK and then ACK. But KCP receives ACK before FIN,ACK. KCP server never ACKs the FIN,ACK — continuous retransmission of ACK. RST does not clear state." The "Invalid ACK" spike While one thread of the investigation was analyzing packet captures, a.k.a., pcaps, another team was digging into internal metrics for the Private Service Connect infrastructure — specifically the user space packet switch (USPS), Google's software switching layer that handles NAT and proxy protocol header injection for Private Service Connect connections. They found something alarming in the metrics for Andromeda, Google's networking fabric. A counter called "Proxy protocol drop: invalid ACK packet" had been flat for months — and then, in February 2026, it spiked sharply upward to several drops per second across the affected europe-west regions. That date matched a USPS version rollout. This version included a change which was a fix for a different bug but something about it had changed the behavior of how Private Service Connect handled a specific class of packet. Challenge ACKs: The RFC 5961 mechanism To understand the next piece of the puzzle, we need to talk about a TCP security mechanism called the challenge ACK, defined in RFC 5961. Imagine a TCP connection has closed — gone through the FIN/ACK dance and terminated. The server-side (here, the KCP VM) may still briefly remember the connection's sequence numbers in its TCP state machine. Now imagine that shortly afterward, a new client connection arrives on the same source port and destination port (the same 5-tuple). From the kernel's perspective, this could be a stale packet from the old connection, or it could be an attacker trying to inject data into what the kernel thinks is still an open session. RFC 5961 says: when you receive a SYN that matches an existing or recently-closed connection's remembered state, don't accept it blindly. Send a challenge ACK — an acknowledgment that carries the last known sequence number of the old connection. If the new client is legitimate, it will see this unexpected ACK, recognize the connection is stale, and send a RST to clear the state. The server then accepts the next SYN normally. The challenge ACK flow — how it's supposed to work This mechanism works perfectly — when the challenge ACK is allowed through. But that was exactly what the new USPS change was blocking. The culprit: A well-intentioned fix The change in USPS had been designed to fix a legitimate security issue in Private Service Connect's proxy protocol handling. Part of its effect was to drop packets it classified as "invalid ACKs" during the connection handshake phase — specifically, ACKs that arrived before the initial SYN-ACK had been sent, which in normal circumstances would be anomalous. Unfortunately, challenge ACKs look exactly like this: They arrive from the server before the new handshake has completed, in response to a SYN. From the USPS's new perspective, they looked like invalid packets. They were dropped silently. Why this broke GKE specifically: Private Service Connect uses NAT to share a single public endpoint across many client pods. This means many different pods — running on different nodes, from different namespaces — can all end up using the same source port after NAT, creating a high rate of 5-tuple collisions. The challenge ACK mechanism was being triggered far more often than it would be in a normal direct-connection scenario. The USPS fix hadn't introduced challenge ACKs — they had always been there. It had introduced the dropping of them, which broke a flow that GKE's TCP stack had silently depended on to clear stale state. Mitigation: Rolling back the change In February, we rolled back the USPS change, restoring the old behavior where challenge ACKs were passed through instead of dropped. The rate of "Proxy protocol drop: invalid ACK packet" dropped back toward baseline almost immediately. By early March , 2026, the customer confirmed that all affected clusters had recovered. No more timeout errors. But the rollback wasn't a fix. It was a band-aid. It restored the behavior the system had accidentally depended on. The deeper question — why was the KCP VM in FIN_WAIT_2 for over two minutes in the first place? — was still completely unanswered. That question led to Part 3. And it's where things got genuinely strange. Part three: Three bugs, one crime scene The mitigation bought us time, but it left a question hanging in the air: Why did the KCP VM keep connections in FIN_WAIT_2 for so long? A well-behaved TCP stack processes a FIN, acknowledges it, and closes the connection. The KCP VM was failing to do this, leaving half-open connections that persisted for more than two minutes — long enough for the port to be reused by Private Service Connect's NAT layer, long enough to cause the challenge ACK scenario we'd uncovered. To answer this, we needed to understand not just one bug, but three distinct failure modes that had intersected at the worst possible moment. Bug #1: Packet reordering at the PIGv The first piece was discovered through careful analysis of simultaneous captures at two points: the KCP VM's host NIC (inside the physical host running the VM) versus the PIGv — the virtual NIC as seen by the VM. At the host NIC level, packets arrived in perfect order. But by the time they crossed through USPS and appeared at the PIGv, they had been subtly reordered. This happens because of how Private Service Connect's miss-thread processing works. When Private Service Connect intercepts a new flow it hasn't seen before (a "miss"), it processes that packet through a slower path that adds the proxy protocol header and sets up NAT state. Subsequent packets in the same flow take a fast path. The consequence: a FIN or RST packet belonging to a flow in mid-close might arrive slightly out of order relative to the ACKs that followed it. What the KCP VM actually received (reordered sequence) The reordering put the client's FIN after the server had already sent its own FIN. The server entered FIN_WAIT_1, then FIN_WAIT_2 after seeing the client's ACK of the server's FIN. But then the client's own FIN arrived — and the server's kernel ignored it. Bug #2: Envoy's half-close and the frozen timer Why couldn't the operating system clean up on its own? There is a Linux kernel parameter called tcp_fin_timeout that governs exactly this: how long a socket in FIN_WAIT_2 waits before being forcibly closed. On the KCP VM, this was set to 60 seconds. But tcp_fin_timeout only applies to orphaned sockets — sockets that no process is currently holding open. Envoy, the proxy that sits in front of kube-apiserver on the KCP VM, was performing a half-close: it called shutdown(SHUT_WR), which closes the write side of the connection but keeps the socket open so it can still receive data. This is perfectly valid TCP behavior and is how graceful half-duplex termination is supposed to work. The side effect: As long as Envoy held the socket with SHUT_WR, the kernel did not consider it orphaned. tcp_fin_timeout never started. The socket sat in FIN_WAIT_2 indefinitely, waiting for a client FIN it would never be able to process correctly. Kernel socket state — captured via netstat just before a new connection Bug #3: The Linux kernel's silent failure But there was still a missing piece. Even with the reordered FIN, a healthy kernel should still process a received FIN in FIN_WAIT_2 and move to TIME_WAIT. The socket should eventually close. Why wasn't it? This led us deep into the Linux TCP state-machine implementation. When the reordered client FIN arrived at the KCP VM, the kernel's tcp_ack() function was called to process it. And here, in kernel version 6.1 (which underpinned the Container-Optimized OS (COS) running on the KCP VMs), there was a bug. RFC 5961 added stricter validation for acceptable ACK numbers — specifically, an ACK could be considered valid even if it was slightly "old" (within a sequence number window), to handle network reordering. But in kernel 6.1, there was a code path where tcp_ack() returned zero — indicating "this packet was not fully processed" — when it encountered an ACK that was old but within the RFC 5961 acceptable window. The caller of tcp_ack() in the FIN processing path checked this return value, saw zero, and did… nothing. The FIN was received and silently discarded. The Linux kernel commit that fixed it (upstream) The bug had existed since kernel 6.1 and was already fixed in kernel 6.9.7 — but GKE's KCP VMs were running COS builds based on 6.1 and 6.6. The fix needed to be backported and shipped as a COS image update. How the three bugs combined Individually, none of these three issues would have caused an outage of this magnitude. It took all three acting together to create the failure: - Bug #1 alone: Packet reordering is mildly disruptive, but the kernel would normally still eventually process the FIN and clean up, just slower than ideal. - Bug #2 alone: Envoy's half-close is perfectly correct behavior. It only becomes a problem if the FIN is never properly processed. - Bug #3 alone: The kernel's failure to process old-but-acceptable ACKs would cause some connections to linger, but the impact would be limited without the port-reuse pressure from Private Service Connect NAT. - All three together: Reordering causes the FIN to arrive out of sequence; the kernel bug prevents it from being processed; Envoy's SHUT_WR prevents the timeout from firing; the socket stays in FIN_WAIT_2 indefinitely. Private Service Connect NAT reuses the port. challenge ACK is sent. The USPS fix drops it. Connection fails. The resolution: A three-pronged fix Resolving this required action at three different levels, coordinated across multiple teams over several weeks: - Immediate mitigation (Feb, 2026): Rollback of change in USPS. Restored the previous behavior where challenge ACKs were forwarded rather than dropped. Customer clusters recovered by early March. - Correctness fix: A replacement change to handle the challenge ACK scenario properly in the Private Service Connect proxy protocol layer, rather than simply restoring the old drop behavior. A PacketDrill test was added to the test suite to prevent regression. - Root cause fix — Linux kernel patch: Upstream patch ("net: tcp: accept old ack during closing"), authored by Eric Dumazet. Merged into Linux 6.9.7. Backported to 6.6.y and 6.1.y LTS branches. — shipping in GKE 1.33.5 and later. The bug is fully resolved at the OS level for all GKE versions running these COS images. What made this case different Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — i/o timeout on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete. What finally cracked it was the discipline of simultaneous packet capture at multiple points in the path. Without captures at the host NIC, the PIGv, and inside the container at the same time, we would never have seen the reordering. Without correlating those pcaps to the USPS metric spike on February 18, we would never have found the dropped challenge ACKs. And without reproducing the issue in a lab environment, we would never have found the kernel bug that made the FIN_WAIT_2 state permanent. The other thing that made this work was the team. Over the course of two weeks, engineers from seven or eight different specialties contributed: GKE networking, Private Service Connect infrastructure, Hostnet, Andromeda, KCP, and ultimately the kernel community. No single person could have held all of this in their head. The case notes, the shared bug tracker, the meticulously collected pcaps — they were the connective tissue that let a distributed team work as one. In the end, the customer got back to production. The kernel got a better bug fix. And the rest of us got a very good story about why packet reordering, half-closed sockets, and a well-intentioned security patch should never be allowed in the same room at the same time. This blog is based on real case data from an anonymized customer support engagement and internal bugs. The Linux kernel fix referenced is commit 795a7dfbc3d9, authored by Eric Dumazet. Special thanks to the engineers across GKE, Private Service Connect, Hostnet, Andromeda, COS, and the Linux networking community whose work made this resolution possible.