The Economics of Continuous Inference

extract

An AI model may prove its capabilities during evaluation, but the real test begins when users start using it at scale. A benchmark measures model capability under controlled conditions; production serving must meet latency, concurrency, availability and cost targets under variable demand. Long prompts increase prompt-processing demand; variable output lengths make capacity harder to predict; bursty traffic creates queues; and KV-cache pressure, replication for resilience, and idle capacity required for peak demand all raise the effective cost of serving each request. Inference is not a one-time compute event. It is a continuous production workload whose cost and performance depend on traffic patterns, response-time targets, model size, context length, concurrency, and the compute required to complete each task. Those economics also vary sharply by workload type: batch inference can prioritise throughput and scheduling flexibility, while interactive chat, voice, retrieval-augmented generation, agentic workflows and real-time decisioning have very different latency, availability and capacity requirements. That changes the economics of AI infrastructure. GPU hourly rates and API prices tell only part of the story. The real question is the cost per successful task at an agreed service level, measured by time-to-first-token, end-to-end latency, availability, quality and error rate. The Cost Per Token is Falling. The Cost of the Application May Not Be An important paradox lies at the heart of AI infrastructure. Inference has become dramatically cheaper, but the decline in token prices is not uniform across model classes, geographies, input and output tokens, or reserved and on-demand capacity. The cost of querying a model with GPT-3.5-level performance on the MMLU benchmark has fallen sharply. Stanford’s 2025 AI Index puts the price at $20 per million tokens in November 2022, compared with $0.07 per million tokens for Gemini 1.5 Flash-8B in October 2024, a more than 280-fold decline in about 18 months. The report also found that, across the models and tasks included in its analysis, the cost of inference fell by roughly nine to 900 times per year, depending on the task and model-performance level measured. This broader estimate refers to inference price reductions, rather than hardware efficiency or a uniform decline across production workloads. Lower unit prices have coincided with models being used for increasingly demanding workloads. Reasoning models can generate substantially more tokens, while agents can make multiple model calls, invoke tools and carry context from one step to the next. For example, a reasoning workflow may use a smaller number of requests but produce far more generated tokens and hold accelerator memory for longer. The result is that cost per token and cost per completed task are increasingly different metrics. A completed task could be a customer case resolved without escalation, a KYC document correctly classified, or a claims summary accepted by a reviewer. Even a nominally simple assistant may involve retrieval, safety checks and a generation model; an agent adds planning, repeated model calls, tool execution and retries. The unit price may be falling while the amount of inference required to complete a task is increasing. That is why production AI economics cannot stop at the API price sheet. This is particularly true when output tokens, context windows, retries and tool-call loops increase faster than the decline in unit price. Throughput Matters-But Only at the Required Latency It is useful for any inference system to measure how many requests it can complete, or how many tokens it can generate, while meeting the application’s quality and latency targets. Throughput complements latency: it shows how much work the system can complete while still meeting the latency target. Latency refers to how quickly an AI system can respond to an individual request, while throughput measures how much work it can handle within a given period, often expressed as requests or tokens per second across multiple users. For LLMs, time to first token is heavily shaped by prompt processing/prefill; inter-token latency or output tokens per second reflects generation/decode performance. An inference system has to handle incoming requests while balancing response time and hardware utilisation. For example, increasing batch sizes can improve throughput, but larger batches can also increase latency. Keeping latency very low may require extra capacity, which can lead to lower hardware utilisation when demand is low. Continuous or dynamic batching can improve utilisation while managing latency, but its effectiveness depends on factors such as prompt length, output variance, scheduler design and model architecture. This makes GPU utilisation a financial variable. But higher utilisation is not automatically better if it produces queueing and violates response-time targets. If a GPU costs the same whether it is processing requests or sitting idle, the effective cost of useful output reflects both accelerator spend and the supporting capacity required to meet the SLA. That includes CPU, memory, storage, network egress, software, support and reserve capacity. However, hardware is only one part of the equation. The serving engine, batching strategy, workload shape, model architecture, networking, memory utilisation, prefill/decode disaggregation, request scheduling and queueing, and traffic profile all influence how much useful inference can be extracted from that hardware. Latency is Not Just a User-Experience Metric For an AI system in production, latency is both an experience metric and a capacity and cost metric. Longer requests occupy serving capacity for longer, which can increase queueing and reduce the infrastructure’s ability to handle subsequent requests. That makes latency a key factor in both user experience and inference economics. Latency matters differently depending on how an AI system is used. In a voice agent, for example, performance is not just about generating tokens per second. Time to first audio and end-of-turn latency shape how responsive the interaction feels, while ASR, TTS, turn detection and network delays add to the overall latency. Model inference is only one part of the equation. This makes it important to look at latency as more than a single metric. Platforms such as Artificial Analysis illustrate the industry’s growing use of multiple metrics to evaluate inference performance. These include time to first token (TTFT), which measures how long a request takes to begin generating output; output tokens per second, which measures generation speed after the first token; and end-to-end latency, which captures the total time from request to completion. Performance can also be measured at different concurrency levels to show how latency changes as more requests are served simultaneously. This distinction becomes even more important as applications move from simple chat towards agents. Agent workflows create serial latency, while parallel tool calls can reduce wall-clock time but increase instantaneous compute demand. In a serial agent workflow, latency accumulates across planning, retrieval, tool execution, validation and final generation. Each additional inference step adds to the time required to complete a task. The economics therefore depend on both the cost of inference and the time required to complete the task. A higher-priced model can have lower task cost if it improves first-pass completion, reduces retries or allows more requests per unit of infrastructure while meeting the same SLA. Availability Changes the Capacity Equation and Tail Latency Inference also behaves differently from training. While some training workloads are deadline-sensitive, the key distinction is that many inference workloads are user-facing and must be served synchronously. A training job can often wait for capacity or be scheduled for a later period. A customer-facing AI service cannot simply tell users that the GPUs are busy. Systems can degrade gracefully by routing requests to a smaller model, reducing context, queuing non-urgent work for asynchronous processing, or falling back to a non-AI flow. These mechanisms make reliability a function of both capacity planning and how the system responds when capacity is constrained. This creates a capacity-planning problem. The system needs enough compute for normal demand, enough headroom for traffic spikes and enough resilience to handle hardware or infrastructure failures. Redundancy protects availability when hardware or infrastructure fails, while spare capacity protects performance during demand spikes. The two serve different purposes and have different economics. Provisioning too little creates latency and availability problems, while provisioning too much creates idle capacity. Both have costs, but the optimal trade-off depends on the value and criticality of the workload. This is particularly difficult because AI workloads are often bursty. A product launch, a popular feature or a sudden increase in agent activity can change the required capacity far faster than conventional enterprise workloads. The implication is that production inference should be planned around demand distributions, service-level objectives and failure scenarios, rather than average utilisation alone. Reasoning and Agents Change the Demand Curve The next phase of inference makes this problem more pronounced. Reasoning models use longer generated traces, test-time search, verification or multiple candidate paths at inference time, while agentic systems introduce multiple rounds of model execution into a single user task. Microsoft Research estimates that a typical frontier-scale query running on an H100 under realistic production assumptions consumes a median of 0.34 Wh, but when test-time scaling increases token generation by 15 times, median energy consumption rises to 4.32 Wh—roughly 13 times higher. That said, the actual energy use can vary significantly depending on the model, hardware, batch size, utilisation, token count and data-centre overhead. Longer reasoning means more tokens. More generated tokens generally consume more accelerator time under concurrent load, which raises queueing risk and the amount of capacity required to maintain an SLA. Agentic workloads introduce another multiplier.NVIDIA notes that agentic requests can trigger multiple tool calls and reasoning steps, making the cost of completing a task a more useful measure alongside cost per token. That changes how infrastructure teams need to forecast demand. That said, there is no single optimisation that solves inference economics. The broader point is that as AI workflows become more complex, infrastructure costs increasingly depend on how much compute is required to complete an outcome, not just how many tokens are generated. In multi-agent systems, a single user action can also start multiple model sessions at the same time. Architecture therefore matters: sequential agents increase elapsed time, concurrent agents increase peak resource demand, while smaller specialist models can sometimes reduce both latency and compute requirements. Hardware matters. But so do memory bandwidth, networking, model quantisation, batching, KV-cache management, prompt caching, speculative decoding, model routing and workload orchestration. Choose Infrastructure by Workload, Not Category General-purpose cloud infrastructure remains attractive for experimentation, variable workloads and organisations that value elasticity. That elasticity comes at a price, while availability can vary by region and accelerator type. For experimentation, cloud platforms also offer managed services, model catalogues and rapid provisioning, alongside broader platform capabilities. Dedicated infrastructure can make more sense when utilisation is high and sustained enough to justify committed capacity, particularly when applications require tighter control over latency, performance, resilience or data residency. The decision also depends on the commitment horizon, regional requirements, operational capability and the opportunity cost of tying up capital in dedicated capacity. The choice is therefore less about declaring one infrastructure model universally superior and more about matching the deployment model to the workload. Evaluate the workload against utilisation profile, latency SLO, data locality, expected growth, commitment period, operational maturity and recovery requirements. That is where the emerging Neocloud category becomes relevant. AI-focused clouds typically differentiate through accelerator availability, cluster/netwrok design,m scheduling, managed serving and support for AI frameworks. AI-focused infrastructure providers seek to optimise across the layers: compute, high-speed fabric, storage, orchestration, inference runtime, observability and support. Neysa, for example, positions Velocis as a full-stack AI Acceleration Cloud that combines GPU infrastructure with orchestration, observability, governance and managed inference capabilities. The significance of that approach is not that every AI workload needs a specialised cloud. It is that as inference becomes a permanent production workload, the infrastructure layer must expose and manage AI-specific workload characteristics, rather than merely providing raw accelerators. The Next AI Infrastructure Metric May Be Cost Per Outcome The first widely visible phase of generative AI emphasised model parameters, benchmark scores and token pricing. Production changes the equation, and application-level economics matter more than ever. Organisations need to know how many requests their infrastructure can serve, how quickly it can respond, how much capacity must remain available, how much compute each task consumes, the quality it delivers, and how often those tasks fail. The falling price of inference is good news. But cheaper tokens do not automatically mean cheaper AI applications. The real economic advantage will come from getting more useful work out of every unit of compute while maintaining the latency and availability that the application requires. That is why the economics of continuous inference ultimately come down to throughput, latency, availability, utilisation, and capacity planning—with token price only one variable in a much larger equation. And as AI moves from chatbots towards reasoning systems and agents that continuously execute work, that equation is likely to become one of the central infrastructure problems for production generative AI.