# I built an RK3588 NPU runtime without any proprietary inference libraries: Qwen3.5-4B at ~10 tok/s, ~2x faster than vendor SDK. — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/1df6a553e77f68e8

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-20 21:03 ingested 2026-10-21 02:41

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xb45bq/i_built_an_rk3588_npu_runtime_without_any/)

## retained text



```
If you have an RK3588 board, here’s another way to put its NPU to work.

I made open\_rknpu2\_lm\_infer, a local LLM inference engine for the Rockchip RK3588 NPU.

It runs without Rockchip’s proprietary userspace inference libraries. The upper part is written in Rust and the NPU operators are coded with C++. No black-box, you can actually read, modify, and hack the host inference code and NPU operator execution.

And now supports Qwen3.5-4B with image input and optional MTP speculative decoding. And it's fast!



Github: [https://github.com/happyme531/open\_rknpu2\_lm\_infer](https://github.com/happyme531/open_rknpu2_lm_infer)



* Quick comparison against RKLLM 1.3.1

|Qwen3.5-4B test|RKLLM 1.3.1|open\_rknpu2\_lm\_infer  MTP1|
|:-|:-|:-|
|Prefill, 512-token prompt|58.35 tok/s|147.83 tok/s|
|Prefill, 3,800-token prompt|54.03 tok/s|139.92 tok/s|
|Decode after 512 tokens|5.54 tok/s|10.61 tok/s|
|Decode after 3,800 tokens|4.57 tok/s|9.90 tok/s|

Current features:

* Terminal chat or an OpenAI-compatible API, with streaming, tool calling, and prefix caching.
* Qwen3.5-4B image input, alongside text inference.
* Validated checkpoints for Qwen3-4B, Qwen3.5-4B, Granite-4.2-3B, and MiniCPM5-2B.
* Prebuilt ARM64 Linux binaries and links to quantized models.

Incoming features:

* DFlash and DSpark speculation
* MiniCPM-V 4.7 35a3 with expert prefetching

If you have an RK3588 board, I’d love to compare results across boards and RAM configurations. Share your board, driver version, model, and speeds—or open an issue if something breaks.

If you try it and find it useful, I’d appreciate a GitHub star!


```



document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xb45bq/i_built_an_rk3588_npu_runtime_without_any
