A primer on key AI LLM concepts — KV cache, weights, quantization, and how inference actually works

extract

A primer on key AI LLM concepts — KV cache, weights, quantization, and how inference actually works Everyone knows about hallucination, but a handful of topics are often unclear. We'll fix that. As hardware for running local AI models continues to move to the forefront, such as Nvidia's RTX Spark, a new set of acronyms, definitions, and terminology is popping up. Given the pace of releases, we've put together a list of AI's new nomenclature to familiarize you with new concepts and pieces of terminology that you'll likely see in Tom's Hardware Premium articles. This primer should get you started on the world of AI and be handy the next time news on the topic comes around, and possibly when you're downloading some models to try out and tweak for yourself. For the context of this article, we're going to stick with notions related to text LLMs, or in common parlance, chat, programming, and most agentic bots. Model This is probably the easiest one: a model is the LLM itself (a Large Language Model), computed from training sets of mostly text data on the scale of terabytes to petabytes. That training information comes from the internet, books, video and audio transcripts, and other sources. The model is roughly analogous to a set of highly advanced layers of statistical analysis of relationships between tokens — words or parts of words, computer code, punctuation, and other symbols. The tokens themselves aren't part of the model, but they are packaged along with it. An AI model generates tokens in its outputs. Weights Those computing layers within an AI model would be useless without corresponding weights, which represent how strong the connection is between sections of tokens, sort of like how a pathway between neurons is more traveled than others. When the model runs, these values end up establishing how tokens end up relating to each other. Example: the weights of the connections between "na" and each next part for "naval", "nachos", or "naruto" will all be different, and the higher the weight, the more often that word appears in training data. Importantly, weights cannot change without re-training the model, or parts of it. When you see LLM specs online, you'll see something like "4B", "8B+", "20B", and so on and so forth. That shows how many parameters, or weights, the model uses, and it's generally measured in billions for consumer-grade models, and trillions for frontier models like GPT, Claude, and Kimi. Note that a model having more weights than another doesn't automatically mean it's smarter. Just like with CPU clock speeds, it's not just about getting a higher number — bigger is usually better, yes, but you want those weights to be meaningful for the output. You may also have heard about "open-weight" models versus "closed" models. Those refer to models like Llama, DeepSeek, and Kimi, which are effectively free-to-use, since the entities behind them freely publish the weights online. You can tweak the weights to your heart's content, if you wish, and come up with your own spin on things, as long as you have the hardware to run it. Note that downloading a free model doesn't reveal anything about how it was trained or how the weights were calculated — think downloading a free customizable application, not open-source software. Data types and quantization FP16, INT8, FP4, all sound like acronyms in some military handbook, but they're simple to interpret. All programs use numbers of varying width and precision, depending on their purpose: a simple money counter in your favorite game might be fine with a 32-bit integer that can represent 0 to 2^32 (INT32), while a value held in memory for a highly precise scientific measurement may need floating-point 64-bit arithmetic (FP64). As memory is an exceedingly finite resource, good developers try to use the smallest units they can get away with, a notion that's particularly important in the VRAM-limited world of GPUs. Predictably, these chips became particularly adept at handling lower-precision values as a tradeoff between processing speed and numerical accuracy. A particular datatype isn't better or worse, as that assessment will depend on what it's used for. When training an AI model, the labs will want to use as high a precision as they can for the weights, but keeping that in the final version would make for a prohibitively large and slow LLM. That's where the concept of quantization comes in, which is a fancy word for "clever rounding." As one of the steps after training, several passes are made to transform highly precise weights. Each AI lab has its own secret sauce about how to do this, as it's a critical step. Frontier models predominantly use relatively high precision for their weights (BF16), in order to maintain response quality, nuance, and reasoning ability (an analogy would be deciding how accurate you want a GPS location to be — miles or inches). Open-weight models often have several versions with different quantization levels depending on what your hardware can bear — for example, Llama 3.3 70B can be run in full BF16 or quantized INT4 flavors. The lower the precision, the more likely it can be run on consumer hardware. AI models listed on Hugging Face may have several quantization levels to fit certain levels of hardware, trading response accuracy for size. Consumer-grade models use smaller data types so they can fit on your Mac Studio, RTX Spark, or graphics card. Context window "Context window" is another term you've heard a lot and probably seen firsthand, likely in Claude Code's UI as a broad indicator of how long your session can go before the bot starts acting a little dumb. It's measured in tokens, with one million being a common number for frontier models, and it's almost directly analogous to RAM in a computer. The context window is the LLM's thinking space, and it needs to contain everything associated with your session, not just the current question. Along with all your questions and its replies, it contains system-level prompts, all files you uploaded, the content of links it looked up, plus any associated data. Crucially, while for you all these things are distinct concepts, for the LLM, they get reduced to tokens in one large pile: the context window. This is also why separating "data" from "code" is technically impossible — the best that modern models do is an approximation of that distinction. LLMs don't think — when answering a new question, they re-run a statistics program over all the tokens and predict the next handful, so having a larger context window is beneficial (at the cost of VRAM), hence the one-million figure that's currently popular. But even statistical algorithms have their limits. Even if you increase the window to, say, three million tokens, you may find that your LLM's prediction starts breaking down even though it hasn't run out. This is amusingly called the "lost in the middle" effect, where the bot will pay attention to the start and end of the conversation but will forget things down the middle. Thus, each model is finely tuned to the size of its context window. KV Cache If you've looked into using LLMs locally, or by way of running a remote GPU, you may have heard about the "KV cache" time and again, and that you need to reserve VRAM for it, or store it on a fast SSD if you absolutely must. This is a set of data that's created when the model runs, but that's actually reusable between queries in the same session. The data that's saved and reused can be described as "the relations between tokens for this session." For example, the ever-popular sentence "you're right to push back" can be tokenized as "you-'re-ri-ght-to-pu-sh-back". The dashes between each token represent a vector (or array), called KV for key-value, that is used for determining which token gets picked next. Since the text generation is linear, the math for calculating each new token needs every KV preceding it. Thus, if you're asking repeated questions in the same session, you'll want the KVs reused to avoid the expense of recalculating them over and over again not just with each round of conversation, but with each additional token. Remember, an LLM doesn't simply answer the question you just posted — it always analyzes the entire conversation from the top to eventually generate tokens for your response. In a long conversation with 300,000 tokens, not having the cache would mean recalculating the KVs for all of them, a slow operation that gets disproportionately slower still as the interactions grow. This excellent blog post has a more detailed example. Remember quantization? That's also applied here — modern models reduce the precision of the KV cache's content compared to their initial values, trading a minor accuracy loss for all new tokens in the session, in exchange for keeping VRAM usage down. In fact, with large context windows and conversations, the KV cache may well eat more space than the model and weights themselves. Mixture-of-Experts (MoE) Previously, AI firms touted how many more parameters their newest model had. Now, the race seems to have shifted to who can include more experts in any given model. That description may conjure a mental image of many tiny scientists living inside a knowledge laboratory, each handling a specific topic, but that's not quite how it works. Mixture-of-experts, or MoE, is essentially a set of sub-models within the main model. When training starts, all sub-models get the same dataset, but the choice of weights is randomized ever so slightly for each one. Each sub-model will value tokens a bit differently from the rest, and due to how the training process reinforces patterns (called the "rich-get-richer" effect), related tokens will tend to converge together, so you end up with "experts" on sets of tokens. A common misconception is to think that these experts are related to topics (like programming), but they're only establishing relations between tokens, meaning they'll be more accurate for certain words and text patterns, but devoid of any concept of what they're about. The fact that an expert might offer pretty good responses in regard to landscape painting is a happy accident, not a feature. You may be thinking that this makes the models larger, and that maybe just the one generalist expert may prove a wiser choice. That's correct, but the trick is that the models only activate certain experts, meaning that only a portion of the model is actually running, thus at higher speed than a generic LLM. This model selection happens on a per-token basis, too, as a router re-selects experts for each incoming token. For visualizing this, think of brain scan patterns where different areas light up depending on words heard. In practical terms, it means the model can answer more accurately in certain sections, while answering the question faster — albeit at the cost of higher VRAM usage. Retrieval-Augmented Generation (RAG) RAG is a concept that's pretty easy to describe, and yet one of the most powerful ones when used with any given model. In short, it can be described as: your chat application runs a few Google searches, feeds them to the model, and voila, there's now much more and fresher information to answer your question. This additional information can come in most any form: company documents, reports, emails, or spreadsheets — anything that you might supply to the model yourself, directly tokenized into its context window. While broadly speaking most contemporary LLM chat applications run searches, activating RAG, usually triggered by selecting a "web search" switch or sometimes enabling "thinking" or "high effort" mode, will explicitly instruct the application to find as much data as it believes necessary to answer a query. Given that pure context is often key to answering many questions, and do so in an informational manner, RAG is among the most powerful tools available. Fun fact: the search queries the application will search for are usually generated by a smaller, separate model that will, for example, say what web searches are bound to give the best results, or what to look for in a linked Office 365 account. The main drawback is that it can be extremely computing-intensive, as the text for your questions and responses may consume, say, 10,000 tokens, yet the information in the retrieved webpages can take hundreds of thousands of tokens all on their own. For example, the Perplexity service was built with an emphasis on RAG usage, and judging from personal experience and user reports, it's usually much more competent and context-aware when answering questions, even though it's underpinned by the same models as everyone else. Inference, Prefill and Decode Inference is the act of using a trained model to generate outputs or tokens. It happens every time you send a prompt, and it's where all of the elements that we've discussed above come into play. Inference can be split into two primary parts: Prefill and Decode. Prefill is the first phase of inference. Essentially, all of the input history (or your prompt) is tokenized and then processed by an AI model. This can include documents, code, and more. These inputs also dictate what gets put into your KV cache. Prefill is processed all at once in parallel, which often means that prefill times are often faster than overall outputs. Prefill is a large part of the overall workload when measuring how long it takes for a specific chip or piece of hardware can take to output its first token, also known as "time to first token" (TTFT), a common AI benchmark, usually reported alongside prefill tokens per second. Longer prompts take longer to prefill, and context, KV cache, and everything else we've discussed are taken into account. Decode is the second phase of inference. After a prefill has processed your input and has built the KV cache, the AI model then generates its response one token at a time, which must be done sequentially in order to determine the ones that come after, which means that the step cannot be performed in parallel, unlike prefill. Decode is usually also dictated by context, and reads from a model's KV cache during this step. Decode is usually limited by memory bandwidth, instead of raw compute, as streaming single tokens from weights and reading the KV cache is a bandwidth-intensive task, meaning memory speeds matter more. You'll usually see Decode speeds referred to as tokens-per-second (TPS). Speculative Decode is a technique that makes those single tokens in the decode step stream faster, without changing the overall output. A drafting mechanism guesses several upcoming tokens, and then the model attempts to verify them in a parallel pass (almost like an additional prefill step). The model then verifies the pass itself; if the guesses are good, then the model's outputs are indeed correct, you get tokens for less of a computational cost during a given decode pass, and when they're bad, they are rejected by the main model during verification, since the main model's output is always what's kept. Multi-token prediction (MTP) attempts to predict the next set of tokens at once, rather than only serving the next token. This is often built into AI models when they are trained with multiple steps. This can serve as the built-in drafting mechanism required for speculative decode. This is a non-exhaustive list of concepts critical to understanding how AI models and LLMs work, but it is by no means complete. We will continue to update and refine this article as we add more. Bruno Ferreira is a contributing writer for Tom's Hardware. He has decades of experience with PC hardware and assorted sundries, alongside a career as a developer. He's obsessed with detail and has a tendency to ramble on the topics he loves. When not doing that, he's usually playing games, or at live music shows and festivals.