loading
loading
StartLux, the young Chinese team behind the StartLux-Decision model family, has open-sourced Prism, an inference engine written specifically for decision models. The code was released on GitHub on October 22, and the company describes it as the first openly available engine built for this class of model rather than adapted from a chat-oriented framework. Decision models read material and return a judgment rather than long text: yes or no, an option choice, or a risk score with probabilities. StartLux-Decision v1.1, released on October 16, previously ran on a general framework based on vLLM. According to StartLux, engines such as vLLM and SGLang are designed for token-by-token generation, so every question becomes a separate request and the large output layer is computed in full each time. Prism makes four changes. First, it narrows the output layer. For the 27B StartLux-Decision model, a general engine scores about 248,000 candidate tokens, while a decision task needs only the letters A to Z, so Prism cuts that computation to about 0.01 percent of the original. Second, it uses what the team calls forest-style batching: shared instructions form the root, shared reference material forms the trunk and each question is a branch, so common work runs only once. Third, matrix operations run in int8 with int32 accumulation, which makes results independent of batch size and scheduling order and keeps outputs bit-identical. Fourth, a mixed scheduler lets up to 64 longer reasoning requests share the GPU with direct-answer requests, with decoding steps precompiled as CUDA graphs so new short requests wait only one or two steps. StartLux's own tests on a single H200 show a short single question finishing in 19 milliseconds versus 53.1 milliseconds before, a 2.8x gain, and up to 5.3x faster when three questions are asked about the same material. On 231 JevBench questions, Prism averaged 41 milliseconds per question, against 75.6 milliseconds for SGLang, 112.7 for vLLM and 589 for llama.cpp. Four H200 cards in data-parallel mode handled 121 requests per second, while tensor parallelism cut single-question latency to 29 milliseconds. Across seven evaluation sets and 12,351 decision samples, the int8 Prism build averaged 92.94 percent accuracy, compared with 92.97 percent for the unquantized BF16 model. A 16.55GB compressed version runs on a single 32GB RTX 5090. All figures are StartLux's own measurements, and the test configurations are published in the repository. SEE ALSO: StartLux-Decision v1.1 Tops Decision Index 0.3.2 · Ant Digital Technologies Releases LingDT-Decision-1.0
document fetched 2026-10-27 09:04