Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs

retained text (truncated)

Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs
Mark Lohmeyer
VP & GM, AI & Computing Infrastructure
Sean Horgan
Product Manager
An open, Kubernetes-native serving stack with cache-aware routing, multi-tenant traffic control, tiered KV cache, and high availability ready for production on Google Cloud
Today at the PyTorch Conference, we released llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference. Co-founded by Google with Red Hat, IBM and others, we worked with a community of more than 500 contributors over 25 organizations on this Cloud Native Computing Foundation (CNCF) Sandbox project, bringing together two important communities: PyTorch, where vLLM and SGLang serve models, and Kubernetes, where distributed systems run and scale in production.
llm-d 1.0 arrives at a time when agents are changing the nature of inference. A year ago, most inference traffic was chat: one prompt in, one response out. Today, workloads are shifting to multi-step agents and reinforcement learning (RL) rollouts that fan out across tools, share long prefixes, and arrive in bursts. Serving them on a single machine — or behind a generic round-robin load balancer — breaks down fast. Replicas recompute the same prompts, accelerator memory fills up with duplicate state, and one bursty agent loop can stall interactive users.
llm-d solves this for teams serving their own models by turning a cluster of model servers into a coordinated serving pool. During the PyTorch Conference keynote, we showed this in action with 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs. As traffic scaled across the three stages below, cache-aware routing and tiered KV cache kept each agent's steps on the replicas holding its context, while flow control queued bursty agent loops so interactive requests stayed fast.
Figure 1: Evolution of distributed inference from round-robin load balancing to cache-aware routing, tiered KV cache, and priority flow control in llm-d 1.0
What 1.0 provides
llm-d already delivered prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference across Google Cloud TPUs and GPUs. These capabilities provide:
- Multi-tenant quality of service. Priority and fairness-aware routing via llm-d flow control, helping latency-sensitive requests and bursty agent workloads share a fleet to meet SLOs without starving one another.
- A stable foundation. v1 APIs and a hardened request path, validated on multiple clouds and accelerator types.
- Safer upgrades. Router high availability and zero-downtime updates, alongside DisaggregatedSet, which orchestrates prefill/decode-aware rolling updates without dropping context or disrupting active sessions.
- Agent-aware serving. Session-aware routing, turn-priority scheduling, and native Responses API state, helping agents reuse context across turns.
- Expanded TPU integration. Native PyTorch model serving with TorchTPU co-operative time-slicing to raise throughput and utilization across Google Cloud TPU fleets.
The components are designed so that you can compose them into your own bespoke production environment. Keep your engine, model and cluster, and adopt only the serving components and configurations you need.
llm-d under the hood
Across the areas below, Google engineers worked with the llm-d community to connect upstream Kubernetes primitives, PyTorch serving engines, and accelerator-aware scheduling into a composable stack.
Figure 2: Multi-node llm-d orchestration and single-node TorchTPU model engines on Google Kubernetes Engine and Google TPUs
One endpoint, many accelerators
An agent fleet is not one workload. Interactive turns need low latency, background loops and reinforcement learning (RL) rollouts need high throughput, and demand swings by the minute. llm-d puts your accelerator pools behind a single endpoint, with one router, one KV cache index, and one flow-control policy in front of the whole fleet. Agents call a single API; the router picks the replica for each step and keeps a session's context on the replicas that already hold it. Adding capacity, mixing accelerator generations, or moving a model to a different pool becomes a platform change, not an agent rewrite. TorchTPU brings Google Cloud TPUs into that fleet, running PyTorch models natively inside upstream vLLM and SGLang without model rewrites. For more, see the llm-d on Google Kubernetes Engine (GKE) TPU guide.
Routing that remembers context
The llm-d router builds on Gateway API Inference, the Kubernetes project for inference-optimized routing that we launched. On GKE, GKE Inference Gateway runs the llm-d router as a managed service, simplifying installation and management and extending routing across regions. The router tracks the replicas holding each cached prefix and weighs load and predicted latency when it picks one. When accelerator memory fills up, rather than dropping it, llm-d uses vLLM’s OffloadingConnector, Mooncake, or SGLang’s HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. This setup helps prevent prefix-eviction storms under heavy, multi-turn, and long-context conversational loads across mixed TPU and GPU fleets.
On an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token (TTFT) by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026 (source).
Disaggregation that upgrades safely
Prefill is compute-heavy and decode is memory-bound, so running both on the same replicas makes each wait on the other. llm-d runs prefill, decode, and multimodal encode as separate pools, with wide expert parallelism spreading large mixture-of-experts models across nodes. In 1.0, DisaggregatedSet — built on LeaderWorkerSet, the Kubernetes API for multi-host inference that we started — manages these pools as one unit. Rolling updates move prefill and decode together on matching versions while each phase scales independently. For more, see the disaggregated serving guide.
Async inference that runs alongside real-time inference
For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Google Cloud Pub/Sub or Redis and routes them alongside real-time traffic. Using queue and pool-dispatch gates, the processor checks available capacity and enforces strict priority lanes and quotas so background jobs run in slack capacity without starving interactive users.
Time-slicing to put idle RL capacity to work
During RL training, accelerators often sit idle 40% to 60% of the time during rollout generation or weight updates. llm-d now supports Google Cloud TPU and GPU time-slicing to interleave async inference into those quiet cycles, raising aggregate accelerator duty cycles from a ~40% baseline to >70% (source).
Benchmarks you can trust
Tuning a distributed inference stack across routers, KV cache tiers, and prefill/decode pools takes repeatable measurement under realistic multi-turn load. Prism is llm-d’s open-source analysis dashboard for exploring published benchmarks across llm-d’s well-lit paths, backed by standardized llm-d-benchmark recipes and inference-perf workloads. Inspect the exact configurations behind the numbers, or upload runs from your own cluster to compare throughput, latency, and cost curves side by side.
Running llm-d 1.0 on Google Cloud
llm-d builds on Kubernetes components you may already run: Gateway API, LeaderWorkerSet, Dynamic Resource Allocation, Prometheus, and OpenTelemetry. Running llm-d on Google Cloud brings these pieces together with managed infrastructure:
- GKE Inference Gateway is the llm-d router operating in Gateway Mode, a managed GKE Gateway that handles the L7 proxy and load balancing and calls the llm-d Endpoint Picker on every request, simplifying installation and management so you don't run the proxy yourself. More importantly, GKE Inference Gateway is how you scale llm-d across clusters and regions with global failover and Model Armor safety checks.
- Autoscaling on inference signals. KEDA and Horizontal Pod Autoscaler (HPA) scale pools based on queue depth, saturation, and predicted latency, while Kueue enforces per-model quota on shared accelerators.
- Mixed accelerator node pools. Google Cloud TPU and GPU pools run behind the same llm-d endpoint, deployed with verified Helm charts.
- KV cache offloading across layers. Google Cloud Managed Lustre provides a shared remote tier beneath local TPU and GPU HBM and host CPU RAM so long-context workloads run without exhausting accelerator memory.
In production
This GKE llm-d stack carries production traffic, and it has been tested at fleet scale:
- Gemini platform (formerly Vertex AI) serves production traffic through GKE Inference Gateway, doubling its prefix cache hit rate from 35% to 70%, cutting TTFT by more than 35% for coding agents and cutting P95 TTFT by more than 50% under bursty chat traffic (source).
- In a published benchmark across a 17,000-node multi-region GKE deployment serving a mixture-of-experts model in the US and Europe, multi-cluster GKE Inference Gateway and the llm-d router scaled throughput 2.9x across three clusters with a 99.9% success rate and less than 1% routing overhead (source).
- In a published benchmark on A3 Mega cluster serving a 70B-parameter model, llm-d with multi-connector KV cache offloading to Google Cloud Managed Lustre in addition to Host CPU RAM, delivered an approximately 40% improvement in TTFT and a 30% reduction in end-to-end latency responding to 50k-token prompts compared to CPU RAM offload only, while reducing GPU-hour requirements by nearly 60% (source).
Organizations running llm-d on Google Cloud are putting these same capabilities to work in different ways, for instance prefix-aware routing to raise cache hit rates, open interfaces that fit an existing service mesh, and flow control to hold latency for priority tiers when traffic spikes.
“At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%. We appreciate the open-source nature of llm-d, as it enables seamless integration with our Envoy-based Service Mesh." - Vinay Kola, Senior Manager, Software Engineering, Snap Inc.
"At ZenteiQ, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign Scientific Foundation Models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler.” - Prof. Sashikumaar Ganesan, Founder, Zenteiq.ai
“Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d's prefix-cache-aware routing, flow control and disaggregated serving support we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects.” - Mathis Felardos, Lucas Fernandes-Martins and Nikhil Thomas, Mistral, Inference Engineers and llm-d contributors
Teams serving their own models have been asking for an open stack they can take apart an

document fetched 2026-10-24 11:20 · canonical https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference