KRAFTON/Raon-1-nano · Hugging Face
retained text
Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by [KRAFTON AI](https://krafton.ai/en/). It is designed to **follow instructions, maintain conversational context, and call the right tools in both Korean and English**. As part of the Raon open model family, it is released under the **Apache 2.0 license**. # [](https://huggingface.co/KRAFTON/Raon-1-nano#highlights)Highlights * **Faithful, natural conversation**: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below. * **Korean and English**: Trained on Korean and English, with solid English capabilities alongside its Korean strengths. * **Tool calling**: Tool calling within conversations, for everyday tasks such as office work and mobile control. * **On-device deployment**: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon. * **Long context**: Natively supports up to 262,144 tokens of context. # Model Overview Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers. * **Raon Delta Attention (RDA):** Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state. * **Raon-mHC:** Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training. * **Per-layer embedding:** Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on. * **Hybrid layout:** 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16. * **Tied embeddings:** The LM head shares the input embedding. |Field|Value| |:-|:-| |Architecture|Dense| |Total Parameters|3.1B (including the 1.5B per-layer embedding table)| |Layers|24| |Hidden Size|1,536| |Attention|Hybrid (RDA 3 : Gated MLA 1)| |Position Encoding|NoPE (no positional encoding)| |Number of Attention Heads|16 for RDA; 32 for Gated MLA| |Attention Head Dimension|128 for RDA; 192 QK / 128 V, for Gated MLA| |Query / KV LoRA Rank|512 / 512, for Gated MLA| |FFN Intermediate Size|6,144| |Activation Function|SwiGLU| |Residual Connection|Raon-mHC w/ 4 streams| |Per-Layer Embedding|256 per layer| |Vocabulary Size|259,840| |LM Head|Tied| |MTP|1 layer, 55M| |Context Length|262,144 tokens| The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module. # [](https://huggingface.co/KRAFTON/Raon-1-nano#training)Training * **MoE-to-Dense:** Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through [MoE-to-Dense](https://arxiv.org/abs/2605.28207) pruning and knowledge distillation. * **Continued pre-training:** 3T tokens. * **Post-training:** supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL). # Quantizations: * [https://huggingface.co/KRAFTON/Raon-1-nano-gguf](https://huggingface.co/KRAFTON/Raon-1-nano-gguf) * [https://huggingface.co/KRAFTON/Raon-1-nano-mlx](https://huggingface.co/KRAFTON/Raon-1-nano-mlx) * [https://huggingface.co/KRAFTON/Raon-1-nano-litert-lm](https://huggingface.co/KRAFTON/Raon-1-nano-litert-lm)
document fetched 2026-11-12 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xtmbkn/kraftonraon1nano_hugging_face