laya.cpp: Optimized laya near-instant decision making
retained text
After seeing [u/Nandakishor_ml’s post introducing Laya](https://www.reddit.com/r/LocalLLaMA/comments/1wjieap/made_the_horizontal_opensource_model_for_jev_with/), I wanted to see how fast it could run in a standalone C++ implementation. Credit to u/Nandakishor_ml for the architecture, training and open-source release. My contribution is the inference implementation: **[laya.cpp](https://github.com/lkarlslund/laya.cpp)**, built on ggml with custom CUDA kernels. It supports all three checkpoints—English, multilingual and typed-decisions—with native tokenization, model execution and output formatting. There’s also an HTTP server with a JEV-compatible endpoint. No Python or PyTorch is required for inference. Some English-model results on an **RTX PRO 6000 Blackwell, capped at 450 W**: | Batch | Python BF16 | C++ BF16 | Python FP32 | C++ FP32 | |---|---:|---:|---:|---:| | 1 | 149 | 366 | 148 | 342 | | 2 | 268 | 586 | 202 | 421 | | 4 | 460 | 761 | 233 | 437 | | 8 | 663 | 810 | 232 | 386 | These are **questions per second** over a fixed 250-question corpus containing choices, scores and booleans. Each precision has paired Python/C++ timings with alternating execution order. Loading and JSON transport are excluded. The [README has the full three-model results](https://github.com/lkarlslund/laya.cpp#performance). Most of the optimization came from removing unnecessary conversions and copies, fusing operations while preserving rounding, and improving attention memory access. The code is MIT-licensed. BF16 currently needs the documented CUDA 13.0/cuBLAS 13.1.0 build profile. Implemented using Codex Astra.
document fetched 2026-09-21 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1wlmkm9/layacpp_optimized_laya_nearinstant_decision_making