github

<details open> Only index by compile times + always multiply/add (#25445) The first one avoids relying on compile to optimize local memory away, and the second is cheaper than issuing control flow statements </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://gi

<details open> llama-bench : init params.offline (#25476) Signed-off-by: Adrien Gallouët <angt@huggingface.co> </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9940/llama-b9940-bin-macos-arm64.tar.gz) - macOS A

<details open> metal : add CONV_2D_DW (depthwise convolution) support (#21565) * metal : add CONV_2D_DW (depthwise 2D convolution) support * test : add perf cases for CONV_2D_DW * metal : use 3D dispatch for CONV_2D_DW kernel * metal : add channel-tiled CONV_2D_DW kernel for

<details open> ggml-hip: enable -funsafe-math-optimizations (#24668) CUDA is compiled with fast math and AMD/HIP is not — this flag lets AMD use fast math too. We can't use -ffast-math: it implies -ffinite-math-only, which won't compile (ggml uses INFINITY for masking) and pro

<details open> cuda: align snake fusion matcher with the other backends (#25460) * cuda: fix snake fusion type predicate, a and inv_b are F32 The matcher required a->type == x->type while launch_snake reads both as const float *, matching the CPU and Metal contract where a and

<details open> server : respect min-step when splitting prompt batches (#25420) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9936/llama-b9936-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI en

<details open> hexagon: add VISION RoPE support (#25216) * hexagon: add VISION RoPE support * hexagon: support RoPE on strided half-dim views for all modes * hex-rope: decouple src0 DMA copy size from row stride * hex-rope: support non-contiguous dst for RoPE * hex-rope: fi

<details open> ggml-webgpu: tune subgroup split (d_split) in flash_attn_vec (#25418) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9934/llama-b9934-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, Kleidi

<details open> opencl: Q6_K GEMM/GEMV fix for ne01 of weights that are not multiples of 128. (#25464) * opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number. Route Q6_K dense mul

<details open> vulkan: disable FA mask_opt on GCN to improve performance (#24362) * vulkan: disable FA mask_opt on GCN to improve performance * reenable mask opt over attention head size 256 </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-or

<details open> opencl: ragged-tile MoE prefill FP16 GEMM optimization (skip padded expert tiles) (#25433) * opencl: ragged-tile MoE prefill GEMM (skip padded expert tiles) The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low tokens-per-expert most til

<details open> llama-batch: fix allowed decreasing pos in a seq (#25449) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9930/llama-b9930-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled)

<details open> vulkan: for small AMD GPUs, reduce submission threshold based on CU count (#25240) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9929/llama-b9929-bin-macos-arm64.tar.gz) - macOS Apple Silicon (

<details open> hexagon: new vtcm layouts and improved pipelines for MUL_MAT, MUL_MAT_ID and FLASH_ATTN_EXT (#25425) * hex-fa: refactor kernel param compute to use common layout builder * hmx: add explicit compiler barriers to make hmx funcs more robust * hex-vtcm: more generi

# PyTorch 2.13.0 Release Notes - [Highlights](#highlights) - [Backwards Incompatible Changes](#backwards-incompatible-changes) - [Deprecations](#deprecations) - [New Features](#new-features) - [Improvements](#improvements) - [Bug fixes](#bug-fixes) - [Performance](#perfo

<details open> spec : fix naming, spacing (#25410) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9912/llama-b9912-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https://git

<details open> CUDA: Fuse MMVQ post-scale for NVFP4 (#24481) * CUDA: Fuse MMVQ for NVFP4 and BS 1 TODO: 1. Add tests to test-backend-ops (did verify correctness manually for one model) 2. Reorder bias/scale once PRs for NVFP4 are merged/landed * Add dense MMVQ fusion as we

<details open> server : fix draft model fit vs load inconsistency (#25056) * fix: draft model fit vs load inconsistency * refactor(server): unify draft/mtp parameter initialization, model, and context load - moves speculative init to speculative.cpp - changes server_context_im

### CuTe DSL * Bug fixing and improvements - Fixed a compilation time regression issue in 4.5.0. Compilation times now match those in the 4.4 and 4.6 branches.

<details open> server : add timings and progress to /responses API stream (#25348) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9909/llama-b9909-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI

<details open> server: enforce prompt cache RAM limit (#25070) Before this commit, --cache-ram was not a hard limit: - The cache always kept at least one entry, even if that entry exceeded the RAM/token limits. - Old entries were only evicted for the RAM/token limits after s

<details open> common : add missing <fstream> include in common.h (#25220) Signed-off-by: zhangrunda <zhangrunda1234@outlook.com> </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9907/llama-b9907-bin-macos-arm6

<details open> ggml-hip : add -fno-finite-math-only alongside -ffast-math (#25373) -ffast-math implies -ffinite-math-only under ROCm/clang 22, which disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors under -Werror in CI). Re-enable infinity handling without drop

<details open> llama: fix quantized kv-cache for dsv4 (#25202) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9905/llama-b9905-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED]

<details open> [SYCL] fix unsupported UT cases of CONT & CPY (#25231) * fix unsupported UT cases of CONT & CPY * update ops.md * rm unused head file </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9904/llama

<details open> [SYCL] support OP cross_entropy_loss, cross_entropy_loss_back (#25236) * support OP cross_entropy_loss, cross_entropy_loss_back * correct format issue </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/down

<details open> vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support (#25351) * vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support * chore : get rid of else --------- Co-authored-by: Stani

<details open> opencl: general flash attention decode performance optimizations (#25366) * opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV * opencl: improve non FA KQ mv kernels * opencl: tweaks for multiquery FA * opencl: some tweaks for FA q1 kernels * ope

<details open> common: Set optimal default thread count for ppc ( linux as well as AIX) (#25237) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9892/llama-b9892-bin-macos-arm64.tar.gz) - macOS Apple Silicon (a

<details open> metal: add col2im_1d op (f32/f16/bf16) (#25176) * metal: add col2im_1d op (f32/f16/bf16) Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc) reads its ceil(K/s0) source columns with an F32 accumulator, a single write and no atomics. One thread per

<details open> CUDA: remove -sm row, refactor cuBLAS (#24216) * CUDA: remove -sm row, refactor cuBLAS * fix CDNA + BF16 logic * fix bad return * fix src0 strides, contiguous requirements * fix GGML_CUDA_FORCE_CUBLAS * fix casts to BF16 </details> **macOS/iOS:** - [macOS

<details open> CUDA: extend K-type validation to V-types for flash attention (#24403) * CUDA: extend K-type validation to V-types for flash attention * reorder </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b

<details open> ggml-cpu: use UE4M3 LUT in ARM NVFP4 dot product (#25331) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9886/llama-b9886-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled)

<details open> ggml-cpu: Enable tiled matmul on AIX (#25199) The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable. Performance Impact: ~

<details open> vulkan: fix 32-bit integer overflow in CEIL_DIV (#25245) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9884/llama-b9884-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [

<details open> scripts : use HF_TOKEN when downloading UI assets (#25280) Signed-off-by: Adrien Gallouët <angt@huggingface.co> </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9882/llama-b9882-bin-macos-arm64.t

<details open> ggml-hip: enable -ffast-math for HIP builds (#23862) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9881/llama-b9881-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISA

<details open> Fix stale tensor-split params for draft models (#24814) * meta: fix tensor split metadata for GQA attention * Tidied the code a bit to match existing style * Revert "Tidied the code a bit to match existing style" This reverts commit b90c6c6300091fe09e2350a3d4e

<details open> abort if we see a multi buffer (#25276) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9877/llama-b9877-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https:/

<details open> ggml : fix tensor-parallel + -ncmoe crash on MoE models (#25028) Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE experts) aborts during warm-up on MoE models with GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp. The faili

<details open> cuda : concat implementation for quantized types (#25303) * cuda : concat implementation for quantized types * chore : apply am17an clever suggestion to shorten the code --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> </details> **macOS/iOS

<details open> llama : add guard for K/V rotation input when buffer is unallocated (#25215) llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's b

<details open> ggml : fix broken CPU concat implementation for quantized types (#25247) * ggml : fix broken CPU concat implementation for quantized types * tests : concat tests for quantized types --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> </details>

<details open> chat: trim messages sent to StepFun parser (fixes long reasoning loops) (#25238) * chat: trim messages sent to StepFun parser (fixes long reasoning loops) * add regression test; remove duplicate template * chat: trim StepFun content parts before rendering The

# Release v5.13.0 ## New Model additions ### KimiK 2.5, 2.6, and 2.7 <img width="1097" height="400" alt="image" src="https://github.com/user-attachments/assets/c24d2232-a9b4-413b-a2c8-58d013b6dfbd" /> This release includes the architecture for Kimi 2.5 which is used

<details open> spec: support spec-draft-p-min in DFlash (#25246) * spec: support spec-draft-p-min in DFlash * dflash: add n_min guard * dflash: guard both n_min and n_max </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/release

<details open> cuda: enable topk-moe fusion for 288 experts (#25267) * cuda: enable topk-moe fusion for 288 experts The topk-moe fusion only accepted power-of-2 expert counts (or the special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash) fell back to the unfused

<details open> server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections (#25241) * server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections * server + ui: ss

<details open> Remove redundant CUDA copies after gated_delta_net. (#23940) * Remove redundant CUDA copies after gated_delta_net. Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With M

<details open> vendor : update cpp-httplib to 0.49.0 (#25218) </details> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b9861/llama-b9861-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](