← sources

laya.cpp: Optimized laya near-instant decision making

source: reddit (reddit:LocalLLaMA)published 2026-09-20 17:06ingested 2026-09-21 02:38

retained text

After seeing [u/Nandakishor_ml’s post introducing Laya](https://www.reddit.com/r/LocalLLaMA/comments/1wjieap/made_the_horizontal_opensource_model_for_jev_with/), I wanted to see how fast it could run in a standalone C++ implementation.

Credit to u/Nandakishor_ml for the architecture, training and open-source release. My contribution is the inference implementation: **[laya.cpp](https://github.com/lkarlslund/laya.cpp)**, built on ggml with custom CUDA kernels.

It supports all three checkpoints—English, multilingual and typed-decisions—with native tokenization, model execution and output formatting. There’s also an HTTP server with a JEV-compatible endpoint. No Python or PyTorch is required for inference.

Some English-model results on an **RTX PRO 6000 Blackwell, capped at 450 W**:

| Batch | Python BF16 | C++ BF16 | Python FP32 | C++ FP32 |
|---|---:|---:|---:|---:|
| 1 | 149 | 366 | 148 | 342 |
| 2 | 268 | 586 | 202 | 421 |
| 4 | 460 | 761 | 233 | 437 |
| 8 | 663 | 810 | 232 | 386 |

These are **questions per second** over a fixed 250-question corpus containing choices, scores and booleans. Each precision has paired Python/C++ timings with alternating execution order. Loading and JSON transport are excluded. The [README has the full three-model results](https://github.com/lkarlslund/laya.cpp#performance).

Most of the optimization came from removing unnecessary conversions and copies, fusing operations while preserving rounding, and improving attention memory access.

The code is MIT-licensed. BF16 currently needs the documented CUDA 13.0/cuBLAS 13.1.0 build profile.

Implemented using Codex Astra.

document fetched 2026-09-21 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1wlmkm9/layacpp_optimized_laya_nearinstant_decision_making