loading
loading
After MCP, A2A, & AG-UI, there’s another agent protocol: And this one is built for Agent harnesses: UHP (Unified Harness Protocol) is a standardized interface for running agent tasks across any agent harness, regardless of which one you use. HarnessRouter is its open-source server implementation. Here’s how it works: - Your product backend sends a task + harness_id to HarnessRouter. - HarnessRouter routes it to whichever configured harness should run it. - The harness streams progress and artifacts back through HarnessRouter. - HarnessRouter returns the output to your product. Three core components handle this: - Gateway: Responses API, harness config, your model keys. - Runner: Isolated workspaces, one event stream, turn checkpoints. - Built-in tools: MCP servers, browser, database, GitHub, media. This means your agent infrastructure becomes fully interchangeable. - Swap Codex for Claude Code without touching your product code. - Run the same task across different harnesses and compare outputs. - Add new harnesses as they emerge without any rewrites. Here’s the repo: github.com/HarnessRouter/harnessrouter (don’t forget to star it ⭐) A Full Walkthrough on Multimodal Jev Jev can see now! We recorded a full video showcasing how to use it: Chapters: - 00:00 - Intro - 00:28 - Multimodal Jev demo - 02:13 - How Jev works - 03:49 - Jev with images - 06:51 - Demo app walkthrough - 10:21 - Using multiple images - 11:16 - Parallel questions - 12:32 - Caveats and danger zone headers - 12:59 - Outro If you’re new to JEV and want to understand how decision models work, we’ve written a detailed article about it. A tricky LLM interview question: You move your LLM to a faster GPU. Profiling confirms its kernels now finish sooner, but the time between generated tokens barely changes. The model, precision, and batch size are identical. Where did the expected speedup go? (answer below) An LLM forward pass is not one giant GPU operation. It launches a sequence of operations for attention, matmul, normalization, sampling, data movement, and other work. Without CUDA Graphs, the CPU prepares and submits these operations individually. Each launch adds only a small amount of overhead, but token-by-token decoding repeats a similar sequence for every generated token. This creates a strange bottleneck. Some kernels finish in only a few microseconds. The GPU can complete one kernel before the CPU has submitted the next one. That leaves empty spaces in a profiler trace. These gaps are not model computation. They show the GPU waiting for its next operation. CUDA Graphs change how this work reaches the GPU. The runtime captures a repeatable sequence of operations once, including their dependencies. Later iterations replay the captured sequence through one graph launch instead of submitting every operation separately. To be clear, the kernels themselves do not become substantially faster. Also, the model does not get smaller, and the forward pass performs almost the same computation. Most of the gain comes from removing CPU launch work between kernels. This is especially useful during LLM decoding. Each decoding step processes one new token per sequence. This produces many short, repeated GPU operations where launch overhead can become a meaningful part of total latency. Prefill often behaves differently. A large prompt gives the GPU more parallel work, producing longer-running kernels. When those kernels already keep the GPU busy, reducing launch overhead matters less. There is also a tradeoff. A captured graph expects the same kernel sequence, stable arguments, fixed memory addresses, and a compatible execution structure on every replay. Inference servers handle varying batch sizes and different prefill, mixed, and decode workloads. One captured graph cannot cover every execution path efficiently. Engines such as vLLM and TensorRT-LLM therefore capture graphs for several common execution cases and batch sizes. When a live batch does not match a captured size, the engine may use eager execution or pad the batch to a captured size. Capturing more sizes improves graph coverage, but it also consumes more GPU memory and increases startup time. So CUDA Graphs are not a universal inference speed switch. They trade flexibility and setup cost for lower steady-state launch overhead. The visual below shows the core mechanism. - Without a graph, the CPU submits each kernel independently and may leave idle gaps on the GPU timeline. - With a graph, the sequence is captured once and replayed as a unit. CUDA Graphs are one example of why LLM inference needs a specialized serving engine rather than a standard prediction API. Good day!
document fetched 2026-10-24 10:04