← sources

I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

source: reddit (reddit:LocalLLaMA)published 2026-11-15 21:46ingested 2026-11-16 02:38

retained text

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

**What it shows**

* **Stream:** the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: *lockstep* (both models read the same text, so you get a clean per-position comparison) and *free* (each follows its own picks until they split).
* **Candidates:** the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
* **Gates:** the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
* **Timeline:** entropy of A and B and their divergence along the whole text, plus the tree of branches.
* **Branching:** click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
* **Logit lens** (`--lens`): what each layer would predict, as a 2D map and as a 3D relief.
* **Model picker:** choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

**How it's built**

* One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
* Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
* The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

**Run it**

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or `inside-llm -a A.gguf -b B.gguf`. Local only: listens on [127.0.0.1](http://127.0.0.1), no accounts, no telemetry. MIT. One click download, one click execute.

**Limitations (honest list)**

* A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B\_Q4\_K\_M and Qwen3.5-0.8B\_Q8\_0
* Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
* Text only. No split (multi-part) GGUFs, no embedding models.

**Links**

* Code + releases: [https://github.com/manuelgscom/inside-llm](https://github.com/manuelgscom/inside-llm)
* Project page: [https://manuelgs.com/en/inside-llm/](https://manuelgs.com/en/inside-llm/)

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?

document fetched 2026-11-16 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1xx7qbf/i_have_created_a_tool_to_observe_what_happens