Agents 2.0 is here!

retained text

In today’s newsletter:
- This inference engine runs LLMs 4x faster than llama.cpp.
- Agents 2.0 is here!
- Continuous batching in LLMs, explained visually.
This inference engine runs LLMs 4x faster than llama.cpp:
We ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.
- llama.cpp reached 22.0 tokens/s
- MLX reached 25.1 tokens/s
- Uzu reached 92.1 tokens/s
The video shows all three runs side by side.
The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.
Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.
A base M5 provides about 153 GB/s of memory bandwidth.
At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.
Uzu gets past that by optimizing speculative decoding for Apple Silicon.
Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.
Uzu pushes this further in two ways.
→ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.
→ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.
The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.
In our run, every pass of the 9B model produced 7.5 output tokens on average.
Ordinary decoding produces exactly 1.
That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.
The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.
It is an open-source inference engine with bindings for Rust, Swift, Python, and TypeScript.
The repository is here: https://github.com/trymirai/uzu
(don’t forget to star 🌟)
Agents 2.0 is here!
Anthropic. OpenAI. xAI.
Are all racing to build an AI coworker.
These are agents that can work alongside humans, talk to other agents, and coordinate tasks in a shared workspace.
But the catch is that each product ties the model, runtime, memory, and orchestration to one company’s stack.
Most teams won’t use one provider for every job. They may use Claude Code, Codex, OpenCode running locally, and custom agents for internal systems. Making them collaborate with each other and your team usually means building the glue yourself.
That is vendor lock-in.
In the video at the top, we have shown how to bring humans and agents from different providers into one shared workspace using an open-source coordination layer you can host yourself.
Humans can talk directly to agents. Agents can talk to each other. And people can join the conversation, add context, or give further instructions as the work unfolds.
The project is called Switch, and it brings this collaboration into Slack, Teams, Discord, Telegram, or Mattermost.
You give each agent a role and can work through a Manager. The Manager creates rooms, brings in the relevant agents, carries context between them, and returns the result.
In this video, we gave the Manager two incidents.
→ The inference API started returning 500s after a morning deploy
→ A recommendation-model training job had been frozen for six hours
From one message, it created two linked rooms, brought in a Software Engineer, Architect, and QA Lead, and ran both investigations in parallel.
Then the engineer investigating the training failure checked the linked inference room and spotted the same config push behind both failures.
The Manager returned one consolidated RCA to the original channel.
Throughout the run, we could follow the conversations, give additional instructions, and inspect the underlying agent sessions in Switch Console.
That is what we mean by Agents 2.0. Humans and agents working together, sharing context, and contributing as the work unfolds.
We build everything locally with Docker and Mattermost and follow the run through to the final handoff.
Models will keep changing, so the coordination layer should be able to run separately from any one provider.
GitHub: http://github.com/sandbox-quantum/switch
(don’t forget to star 🌟)
Thanks to Switch for partnering today!
Continuous batching in LLMs, explained visually:
In traditional ML inference, a batch is a matrix.
Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.
LLM decoding does not work that way.
One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.
Under static batching, membership is fixed when the batch starts.
A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400.
The GPU keeps paying the full weight read for a batch that is mostly empty.
Loading model weights out of HBM costs the same whether four slots are producing tokens or one.
Continuous batching moves the decision boundary.
Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.
A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done.
Anyscale benchmarked both OPT-13B on a single A100.
With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving.
Variance drives the entire gap.
Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.
None of this alters the model.
vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA implements the same mechanism under the name in-flight batching.
The animation below depicts the continuous batching process:
To dive deeper into continuous batching specifically, we wrote a full breakdown of the scheduler underneath it.
It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation.
Good day!

document fetched 2026-10-24 11:26