loading
loading
ZenteiQ scales batch inference 4x with GKE Inference Gateway powered by llm-d Prof. Sashikumaar Ganesan Founder, ZenteiQ Fisayo Feyisetan Product Manager, Google Kubernetes Engine Today we hear from ZenteiQ, an artificial intelligence startup spun out of the Indian Institute of Science (IISc) and selected under the IndiaAI Mission to build sovereign scientific foundation models from the ground up. To pre-train models that understand physical laws, governing equations, and numerical solvers, ZenteiQ designed and built a large-scale batch synthetic data-generation pipeline that runs across multiple model families and feeds verified datasets into training runs. Google Kubernetes Engine (GKE) and GKE Inference Gateway gave ZenteiQ the inference-routing layer within its pipeline to scale aggregate inference throughput 4.07x and grow daily output from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators. - Fisayo Feyisetan, Product Manager, Google Cloud Unlike general-purpose language models trained primarily on web text and code, scientific foundation models must understand complex physical systems, mathematical governing equations, and numerical solvers across domains such as thermodynamics, computational fluid dynamics, and electromagnetics. At ZenteiQ, we are building sovereign scientific foundation models from scratch that interpret simulation outputs, evaluate equation formulations, and work through step-by-step mathematical derivations that obey physical laws. Pre-training models on large datasets of governing equations takes hundreds of billions of reasoning tokens. Alongside curated scientific literature and simulation datasets, we also rely on a batch synthetic data-generation pipeline that we independently designed and implemented to generate, evaluate, and filter candidate derivations and solver validation problems before feeding them into our MaxText pre-training jobs. Because we operate on a fixed compute allocation, every hour our accelerators spend generating and verifying training data is an hour taken away from pre-training. When data generation falls behind, multi-host accelerator clusters reserved for pre-training sit idle, waiting on datasets. When data generation slows down pre-training To maximize data generation speed, our pipeline separates prompt preparation on worker pods from model serving on vLLM across multi-node accelerator pools. Each batch moves through four stages: expanding scientific seed concepts into prompt batches, generating candidate reasoning traces with generator models, scoring responses using evaluator models, and running guardrail checks with a separate model family before packaging verified datasets for pre-training. We originally ran these model pools on Kubernetes with standard ClusterIP round-robin load balancing across inference endpoints. As we scaled our pipeline to feed our 9B- and 35B-parameter model schedules, two routing bottlenecks began slowing down dataset creation. - Uneven engine queues: Our upstream worker pods expand seed concepts into prompts of widely varying complexity and response length. Because standard round-robin load balancing had no visibility into vLLM engine queues, it kept sending new requests to serving pods that were already busy processing long reasoning traces, while neighboring nodes had capacity to spare. - Redundant prefix computation: Prompts within a generation batch share roughly 70% to 80% of their instructional context and task specifications. Round-robin routing scattered requests with identical instructions across different serving instances, forcing each pod to recompute the same shared prompt prefix from scratch instead of reusing warm caches in accelerator memory. Together, uneven queues and redundant prefill work caused our generation throughput to swing by as much as half within a single run, stretching out dataset creation and delaying the start of scheduled pre-training runs. We needed routing that understood live vLLM queue depths and prompt caches, without pulling our researchers off model development to build and maintain a custom inference scheduler. Building on GKE Inference Gateway powered by llm-d We already run our data preparation and MaxText pre-training jobs on GKE, which lets our team manage data generation and model training on the same Kubernetes platform. Following a recommendation from Google Cloud to address our routing bottlenecks without changing our underlying scientific methodology or pipeline design, we deployed GKE Inference Gateway, powered by the open-source llm-d distributed inference framework, in front of our vLLM accelerator pools. In our pipeline, worker pods pull scientific seed documents and domain taxonomies from Cloud Storage and expand them into batches of multi-step prompts. Those batches flow through a single GKE Inference Gateway deployment that manages traffic across all of our model families and serving nodes. Rather than maintaining isolated routing for each model tier, we route generation requests, evaluation queries, and guardrail checks across their respective model pools through the same gateway. Once candidate responses pass evaluation and guardrail checks, worker pods deduplicate and package the verified datasets for direct consumption by downstream MaxText pre-training jobs on GKE. GKE Inference Gateway routes batch generation, evaluation, and guardrail requests across multi-node accelerator pools running vLLM before verified datasets are fed into MaxText pre-training. From round-robin to load- and cache-aware routing With GKE Inference Gateway and llm-d, routing decisions use live serving signals from our vLLM instances rather than static round-robin rotation. GKE Inference Gateway continuously monitors real-time queue depth and key-value (KV) cache utilization across our vLLM endpoints. When a serving node is busy decoding a long equation derivation, the gateway routes incoming requests to nodes with open capacity, so heavy prompts no longer pile up behind busy workers while neighboring nodes sit idle. At the same time, the gateway routes requests with matching scientific instructions to the serving node where that prefix’s KV cache is already warm in accelerator memory. Because 70% to 80% of each prompt is shared instructional context while the generated reasoning trajectories vary from request to request, combining prefix-cache locality with live queue balancing lets our pods skip repeated prefill work without overloading any single replica. Higher throughput, predictable training runs Adopting GKE Inference Gateway powered by llm-d transformed the speed and predictability of our data generation pipeline. Across our production clusters, intelligent routing delivered: - 4x higher fleet throughput: Aggregate throughput jumped from 16,964 to 69,114 tokens per second per 1,000 accelerators — growing daily output from 1.47 billion to nearly 6 billion generated tokens. - 3.5x to 5.5x gains across individual model pools: Real-time queue balancing kept serving nodes evenly loaded throughout each run, preventing the mid-run slowdowns caused by long reasoning derivations. - 14-percentage-point higher prefix-cache hit rate: Routing prompts with matching scientific instructions to warm caches avoided redundant prefill work before each request branched into its own derivation. For our research and infrastructure teams, the most important outcome was schedule predictability. Quadrupling daily output meant verified datasets were ready on time, so our multi-host pre-training runs started on schedule instead of waiting on data generation. What’s next for physical AI at ZenteiQ As we scale our scientific foundation models from 9B to 35B parameters and beyond, we plan to use the same GKE Inference Gateway and llm-d routing setup for post-training reinforcement learning (RL) and automated numerical solver evaluation loops. Because each turn in those multi-turn rollouts builds on the context of the turns before it, they depend even more heavily on live queue balancing and warm prefix caches. To try these capabilities on your own workloads, explore the GKE Inference Gateway documentation and check out the open-source llm-d project. You can also read how the Gemini platform team cut latency by 35% with GKE Inference Gateway.
document fetched 2026-10-24 11:20 · canonical https://cloud.google.com/blog/products/containers-kubernetes/zenteiq-scales-physical-ai-inference-4x-with-gke-and-llm-d