What’s new in AI infrastructure and orchestration this month
extract
What’s new in AI infrastructure and orchestration this month Alex Barrett Editor, Google Cloud blog At Google, AI is a soup-to-nuts endeavor. Obviously, we make leading AI models like Gemini and Nano Banana. We incorporate AI into the tools you use every day (think Gmail, BigQuery, AlloyDB, Google Cloud Code and Google Cloud Assist). We make software frameworks to help you build with AI, like Gemini Enterprise Agent Platform, JAX, or MaxTest. And we co-design the powerful infrastructure platform that runs underneath it all, including a broad range of standard compute, accelerators like TPUs and GPUs, optimized networks and storage, as well as orchestration software like GKE and Cluster Director. Then we package them all up into supercomputing platforms like AI Hypercomputer to power the industry-wide transformation to the AI and agentic future. This is critical in today’s agentic era, where AI is evolving from answering questions to reasoning and taking action. Companies that want to lead in this next phase of AI need computing infrastructure that’s designed and optimized for these new requirements, so they can innovate faster, deliver compelling user and customer experiences, and optimize for cost and energy efficiency — all at massive scale. To support this, we are making AI infrastructure and orchestration news at a furious pace. In this blog, we provide a monthly snapshot of the recent launches and milestones that you need to know about to keep up-to-date, deep dives on architecture and performance tuning, and discussions of specialized use cases, always with pointers to where you can learn more. Keep an eye out for updates to this blog every month. July 2026 Product, technology, and tools updates - Product update: Google Cloud Managed Lustre is now GA, and available in four distinct performance tiers that deliver throughput ranging from 125 MB/s, 250 MB/s, 500 MB/s, to 1000 MB/s per TiB of capacity — with the ability to scale up to 8 PB of storage capacity. The Managed Lustre solution is powered by DDN’s EXAScaler, combining DDN's decades of leadership in high-performance storage with Google Cloud's expertise in cloud infrastructure. - Product update: C4N network and storage optimized VMs are now GA. C4N is our first network- and block-storage-optimized VM series built to eliminate data-transfer bottlenecks. Powered by 5th Gen Intel Xeon Scalable processors and built on Google's Titanium offloading hardware, it achieves 400 Gbps network bandwidth, 95 million packets per second (MPPS), and up to 25 GiB/s of block storage throughput when paired with Hyperdisk Extreme. - New feature: GKE Dataplane V2 up to 15K Nodes with Network Policies (GA). This capability enables standard GKE clusters to scale up to 15,000 nodes while maintaining full active Network Policy enforcement, supporting the massive infrastructure needs of large enterprise and AI/ML customers. - New feature: Co-operative time-slicing in llm-d. If you’re running reinforcement learning (RL) workloads, you can now interleave independent RL jobs onto shared physical hardware, increasing aggregate accelerator duty cycles from a ~40% baseline up to 70% without impacting model convergence or accuracy. - New AI security tool: Looking to secure your AI supply chain on GKE, deploy AI workloads safely, and cut down on shadow AI? We open-sourced k8s-aibom, a lightweight, unprivileged Kubernetes controller that continuously monitors container clusters to automatically detect running AI runtimes (like vLLM and Triton) and generate standard CycloneDX Machine Learning Bill of Materials (ML-BOMs). Check out the k8s-aibom project and get involved. Practitioner guides and how-tos - How-to guide: On July 27, Google announced Day 0 support for Moonshot AI’s Kimi K3 2.8-trillion-parameter open-weight model, the day weights were released. Whichever your preferred deployment path — via Model Garden, custom orchestration, or GKE with llm-d recipes — this guide offers detailed step-by-step instructions to help you evaluate and pilot Kimi K3 in Google Cloud. - How-to guide: Google Kubernetes Engine (GKE) managed DRANET supports both GPUs and TPUs. There are several configurations to use this implementation, including standard cluster (where you have full control) and autopilot cluster (where Google does the heavy configs for you). Take a deeper dive in the hands-on lab, GKE Autopilot clusters with TPUs, GKE managed DRANET and Gemma 4. - How-to guide: Learn to run Ray on TPUs, not GPUs. In Part 1 of this two-part series, we discuss TPU slices (hint: Ray thinks of them as just another accelerator on which to schedule), then walk through Ray’s various AI libraries (Part 2). - How-to guide: Evaluate TPUs for sample workloads using a new microbenchmark suite that helps you accurately assess whether a device is achieving its theoretical performance specifications, and to identify specific performance gaps or architecture-specific bottlenecks. Dive in here. - How-to guide: Scale your agents without killing your budget. Learn how GKE orchestration can help you safely pack more agents onto a fixed compute footprint with GKE Agent Sandbox and Pod snapshots. Whether your goal is performance or cost optimization, we teach you how to turn the right dials for optimal agent efficiency. - Technical blueprint: Inside the optimization of Mistral 3 large inference on Ironwood. This blog outlines how one Google team optimized Mistral 3 large MoE model inference on Google’s Ironwood (TPU v7x), achieving a 1.5x performance gain. They did so with hybrid sharding, replacing linear VPU summations with tree reductions, optimizing GMM/MLA kernels, and adopting asynchronous scheduling. As a result, they boosted throughput by up to 48% while maintaining benchmark accuracy neutrality. Read the full blog here. Research, reports and deep-dives - Report: Google was named a Leader in the inaugural GartnerⓇ Magic Quadrant™ for AI Infrastructure, positioned highest for ‘Ability to Execute’ and furthest for ‘Completeness of Vision’. Gartner called out Google’s proprietary scalable compute, integrated AI Hypercomputer architecture, and the scale of our AI compute capacity as key strengths. Download a copy here. - Report: We recently surveyed more than 1,400 senior IT leaders for our State of AI Infrastructure report, and a resounding pattern emerged: The gap between AI ambition and infrastructure reality is widening. In fact, 83% of organizations say they require infrastructure upgrades to support production-grade agentic AI. Read the accompanying blog to understand how adapting your infrastructure to meet the demands that agentic applications place on your systems will help you move from pilot to production. June 2026 Product, technology and tool updates - Product update: Protecting sensitive data used with AI is a critical part of advanced and secure cloud infrastructure. Confidential Computing cryptographically protects data in use in hardware-based Trusted Execution Environments (TEEs) with verifiable data integrity, and is now available on the accelerator-optimized G4 machine series, featuring NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Get started with Confidential G4 VMs and Confidential G4 GKE Nodes. - Developer resource: The new TPU Developer Hub is the place to go for model builders, optimizers, and developers to learn to unlock the full performance of Google Cloud TPUs. Read more in this blog. - New product: Scale your AI workloads with the new OpenTelemetry-Based TPU AI Telemetry Collector Agent. For the first time, you can route high-fidelity TPU hardware telemetry to Google Cloud Monitoring, Google Managed Prometheus, or your own self-hosted Grafana stack. Practitioner guides and how-tos - How-to guide: Learn how to build high availability into an AI inference workload running on GKE Inference Gateway with TPUs, Cloud Storage FUSE and Dynamic Resource Allocation (DRA). This blog provides an overview, or you can get all the technical details in the hands-on codelab. - How-to guide: Did you know you can connect your AI agents to unstructured data in Cloud Storage via Model Context Protocol (MCP)? In this blog, learn about why would want to do that from three customer examples, then how to do it, choosing either a fully managed service, or a self-managed local server for more customization and control. Research, reports and deep-dives - Report: According to an independent benchmark report, GKE Inference Gateway outperforms the next leading managed Kubernetes service with 15.7% higher throughput, 92.8% shorter wait times, and 62.6% lower inter-token latency. This performance can be attributed to its use of prefix caching, which optimizes LLM performance by storing the KV cache (activation states) of long, repetitive prompt prefixes. Learn more in the blog. - Architecture deep dive: A closer look at the cold start problem, this time for TPUs and GKE, and how the Run:ai Model Streamer can help change the dynamic. Customer and partner updates - Customer win: Leveraging GKE, BigQuery, Cloud SQL, and Gemini Enterprise Agent Platform, Pager Health is eliminating operational fragmentation to deliver a simplified, personalized U.S. healthcare experience that transforms lives. - Customer win: Trustpilot, the customer review platform, built a high-volume streaming pipeline using fine-tuned Gemma models with Dataflow and Gemini Enterprise Agent Platform running on cost-optimized A2 VMs using A100 GPUs, as well as optimized version of vLLM maintained by Gemini Enterprise Agent Platform. May 2026 Product, technology and tool updates - Product update: GKE Agent Sandbox is now generally available. - New open-source project: Agent Substrate is a new open-source project aimed at continuing to push the limits of agentic infrastructure density - New feature: Google AI Edge Portal, a solution for testing and benchmarking on-device machine learning (ML) at scale, now supports benchmarking and debugging on-device LLMs. Read more here. - Product deep dive: We went into depth about Cloud Storage Rapid, a new family of high-performance storage offerings for AI workloads. At launch, offerings include Rapid Bucket (formerly Rapid Storage), a high-performance zonal object storage offering, and Rapid Cache (formerly Anywhere Cache), which accelerates reads on-demand and colocates compute and data for workloads in existing buckets. Research, reports and deep dives - Architecture deep dive: Google Global Infrastructure VP Bikash Koley and Engineering Fellow Arjun Singh provide a high-level overview of the challenges that AI workloads pose to network infrastructure, and discuss the deep enhancements we’ve made to our data center fabrics, WAN, and global networks to better support them. - Architecture deep dive: We unveiled a new cluster-level reliability model for developing frontier AI models on TPUs, ditching instance-level reliability Customer and partner updates - Customer win: Visual media provider Imgix serves more than 8 billion images and videos from AI Hypercomputer equipped with G4 VMs powered by NVIDIA RTX PRO 6000 Blackwell GPUs.
retained text (truncated)
What’s new in AI infrastructure and orchestration in October Alex Barrett Editor, Google Cloud blog Dave Bartoletti Product Manager, AI & Computing Infrastructure The Google Cloud AI Infrastructure team had another busy month in October, filled with new product launches and updates to power today’s AI and agentic workloads scalably and efficiently. We were also honored to share our progress with customers, partners, and the broader community at leading industry events, including NVIDIA GTC, PyTorch Conference, and AGNTCon + MCPCon. Let’s dive in! Product, technology, and tools updates - New compute service: Agent Compute Engine is a purpose-built compute service providing high-performance, hardware-isolated microVM sandboxes alongside expanded agent governance capabilities like Agent Gateway and Agent Registry, all in the service of securing and scaling autonomous AI agents. Its primary benefits include rapid sub-second activation, extreme workload density through copy-on-write memory sharing, enterprise-grade security and identity management, and cross-cloud observability. - AI Hypercomputer updates: At NVIDIA GTC Berlin 2026 we announced updates to our agentic infrastructure, including to A5X instances powered by the NVIDIA Vera Rubin NVL72 platform, Virgo Network scale-out fabric, and Managed Lustre storage. These co-engineered advancements provide enterprises with accelerated performance, enhanced sovereign AI controls, while advanced Confidential Computing capabilities help you train and deploy complex multi-step models securely and efficiently. - GKE product update: CPU startup boost for GKE, in preview, uses Kubernetes In-place Pod Resize to dynamically elevate container CPU resources during initialization and scale them back down to steady-state levels without having to restart the containers. This feature removes startup throttling and accelerates cold starts by up to 2x for demanding workloads, helping to lower cloud infrastructure costs. - GKE product update: The preview of the Wiz Runtime Sensor add-on for Google Kubernetes Engine (GKE) brings low-touch deployment, automated lifecycle management, enterprise credential security, and fleet-wide governance to Wiz's eBPF runtime threat detection for both Standard and Autopilot GKE clusters. - Modernization tools: Together at last. Google Cloud Modernize brings Google Cloud’s migration and modernization tools, including Migration Center, Google Cloud VMware Engine, Google Cloud Mainframe Modernization and our new EKS-to-GKE Migration Agent, into a single portfolio. Enterprise teams now have purpose-built agentic capabilities across infrastructure assessment, platform modernization, and application modernization. - Cloud Run product update: Temporal Serverless Workers on Cloud Run automatically scale worker pools based on Temporal task queue demands and scale down to zero when idle. This integration eliminates cluster management, lowers costs, using Temporal Worker Versioning to safely handle stateful, long-running workflows and AI agent workloads. - New open-source product: llm-d 1.0 is an open-source, Kubernetes-native serving stack that turns a cluster of model servers into a coordinated serving pool, unifying distributed LLM inference across mixed Google Cloud TPU and GPU fleets. Co-founded by Google and partners, it provides production-ready benefits such as cache-aware routing, tiered KV cache offloading, and priority-based flow control to dramatically improve context reuse, reduce time-to-first-token latency, and efficiently scale multi-tenant agent workloads. - New open-source product: Working with Meta, we open-sourced TorchTPU, a native PyTorch stack that integrates Google TPUs directly into the PyTorch ecosystem alongside GPUs and CPUs. Now developers can train, serve, and deploy models natively on TPUs using familiar frameworks like Hugging Face, vLLM, and TorchTitan. - Google Distributed Cloud updates: Organizations that operate under strict sovereignty mandates would love to use AI to help them process data, but can’t send it to the cloud. The latest version of Google Distributed Cloud (GDC) air-gapped brings frontier AI models and multimodal enterprise search with Gemini into these isolated environments. In addition, GDC air-gapped supports NVIDIA HGX B300 systems based on the NVIDIA Blackwell Ultra architecture, and features enhanced networking and security capabilities. - Edge inference product: Google open-sourced ML Drift, a cross-platform on-device GPU compute engine built for LiteRT that abstracts hardware complexities across OpenGL ES, OpenCL, Metal, and WebGPU. The framework delivers peak performance, enhanced 5D tensor support, and memory efficiency for both classical machine learning workloads and advanced generative AI models at the edge. - Open-weight model support: Reflection's Beam is an open-weight, 501-parameter model released under an Apache 2.0 license designed specifically for frontier software engineering and long-horizon agentic tasks. We announced day 0 support and ready-to-run deployment recipes for service Beam on AI Hypercomputer and NVIDIA AI Infrastructure. Practitioner guides and how-tos - AI network reference architectures: In Networking for AI inference model serving - GKE only and for all other backends, learn about reference architectures for AI inference model serving that provide centralized governance and private IP routing using Private Service Connect, Apigee API Management, and Model Armor. These architectures simplify multi-model scaling and prompt security across GKE, Cloud Run, and hybrid clouds. - How-to guide: A Tale of TPU Observability on GKE: Part 2 - Dynamic Slicing Enhancements breaks down GKE's new Dynamic Slicing observability tools and metrics for next-generation v7 TPUs through an engaging, story-driven observability guide featuring Alex, a lead AI platform engineer. Explore the step-by-step diagnostic workflows with real-world examples and handy command-line snippets. Check out Part 1 here. Customer and partner updates - AI21 achieves an 83% reduction in time-to-start for AI workloads with AI Hypercomputer: Frontier lab AI21 needed a new stack on which to train its Jamba family, and chose AI Hypercomputer configured with GKE on A3 and A3 Ultra instances. By doing so, AI21 says it cut high-priority job wait times from 72 hours to 12, and manual scheduling interventions from 20 per week to zero. - Visko streams real-time world models on Google Cloud TPUs: Visko, an AI startup developing interactive world models, deployed an end-to-end live inference pipeline that combines video generation, super-resolution upscaling, and synchronized audio onto a single Cloud TPU slice. In this blog, the team discusses how it engineered the pipeline, the architectural optimizations it made, and practical lessons for other startups building ultra-low-latency generative AI. - ZenteiQ scales physical AI inference 4x with GKE and llm-d: In order to pre-train models that understand physical laws, governing equations, and numerical solvers, Indian AI startup ZenteiQ designed and built a large-scale batch synthetic data-generation pipeline that runs across multiple model families and feeds verified datasets into training runs. GKE and GKE Inference Gateway gave ZenteiQ the inference-routing layer within its pipeline to scale aggregate inference throughput 4.07x and grow daily output from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators. - Palo Alto Networks modernizes cloud-delivered security service: Palo Alto has worked with Google Cloud since 2018 to deliver its Prisma Access security service, and it was time to update it. In order not to compromise its security guarantees, Palo Alto separated the platform into two independently scalable layers: an ingress layer that handles traffic routing and forwarding running on GKE, with Axion C4A instances; and a security layer that handles threat inspection, running on dedicated single-tenant VMs fronted by an Internal Load Balancer. Each layer can now scale, update, and operate independently. - Oracle is the first hyperscaler to provide TPUs outside Google Cloud: By deploying Google TPUs, Oracle is broadening the range of AI chips available through its Oracle Cloud Infrastructure (OCI), and giving customers more options for running AI workloads. Google TPUs on OCI will be available in 2027. Research, reports and deep-dives - New resource: As enterprise organizations shift from isolated AI experiments to production-scale systems, different departments — from engineering and finance to executive leadership — often speak different technical dialects; The Encyclopedia of AI Infrastructure bridges that communication gap by demystifying complex infrastructure jargon. - Analyst report: We are thrilled to announce that Google was named a Leader in the inaugural 2026 IDC MarketScape for Agentic AI Platforms. - Deep dive: A companion technical guide describing the architecture of Agent Compute Engine and how Google’s Gemini team needed both the isolation of a VM and the speed and density of a container to deliver personal agents for every Gemini user. September 2026 Product, technology, and tools updates Google Kubernetes Engine updates: The GKE team is all about improving the scalability of the platform, and in September, those improvements came in many shapes and sizes: - New feature: Need an execution runtime with higher density for your agentic workloads? We engineered the new open-source GKE Agent Substrate to run millions of sandboxes with 10x higher density than standard container runtimes. Agent Substrate also delivers sub-500ms resume operations at over 500 suspend/resume activations per second with a native zero-trust kernel and network isolation. - Product update: GKE now has scale-to-zero capabilities built-in. No need to configure complex components to scale your workloads down, thanks to the HPA with the Autoscaling Metric and support for KEP-2021, which do the job for you, out of the box. Read the blog to learn more. - Product update: Further, the GKE HPA (with the above-mentioned Autoscaling Metric) now lets you scale up and down based on custom PromQL metrics, in addition to standard metrics, allowing you to trigger workloads according to conditions that are meaningful and unique to your business. Read more here. - New feature: Yet another scalability feature is GKE Pod snapshots, which lets you save the running state of your workload, including CPU and GPU memory, and restore it on demand. According to internal tests, GKE Pod snapshots can reduce AI inference start-up by as much as 89%. Learn more here. - New migration tool: Finally, if you’ve always wanted to migrate your container workloads from AWS EKS to GKE but feared a daunting, high-friction engineering endeavor, we’ve just launched GKE agentic migration, a purpose-built agent plugin that replaces brittle, ad-hoc prompting with an AI-assisted migration pipeline protected by deterministic guardrails. Designed as a compilation of agent skills and a local Model Context Protocol (MCP) server, it uses AI to translate complex AWS EKS IaC and Kubernetes manifests directly into GKE landing zones. Get started with the onboarding guide. - Feature updates: Reinforcement learning (RL) and evaluation workloads are a beast: In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, infrastructure bottlenecks emerge, for instance idle accelerators, image cardinality, and a saturated control plane. To help, we developed GKE Agent Sandbox optimized for RL, plus an Agent Sandbox RL orchestration SDK and native integrations for popular RL gyms and harnesses. All are now generally available, and you can learn more here. Storage updates: AI trains and creates lots of da
document fetched 2026-10-31 02:54 · canonical https://cloud.google.com/blog/topics/ai-infrastructure/whats-new-in-ai-infrastructure-this-month