How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences

extract

The 2026 monitoring platform planning framework (Sponsored) Uncover the 11 requirements for evaluating enterprise monitoring platforms anchored to operational outcomes. Use it to identify the capabilities that actually matter, eliminate overlap with tools you already have, and build a monitoring stack that reduces risk and improves collaboration across your DevOps teams. You’ll learn how to: - Evaluate monitoring platforms against 11 operational requirements tied to real DevOps outcomes - Identify and eliminate redundant visibility across your existing toolchain - Architect a monitoring solution that reduces toil, improves collaboration, and cuts MTTR In 1822, Joseph Fourier published his work on how heat spreads through materials. One of his key mathematical tools was a way to represent complicated temperature patterns using combinations of sine and cosine functions. Nearly two centuries later, those same functions appear inside a very different technology: the transformer architecture behind modern language models. The applications seem to be quite far apart. Fourier studies temperature across a physical object. On the other hand, a language model processes tokens arranged in a sequence. Yet, sine and cosine functions provide a useful structure in both settings. This is because sine and cosine functions can turn positions into structured numerical patterns. A transformer model can then use those patterns to learn how the order of words affects their meaning. The waves supply position information. The model’s training teaches it how to use that information. In this article, we are going to understand how this connection works and why a model needs help understanding the order of the words even when the words already arrive in an ordered list. Here’s what we will cover: - The common physics behind heat measurement and LLMs - How word order carries information - How does a model understand - How the attention mechanism makes sense of the order - How the location of the world is made available to the model - The use of sine and cosine functions - How training uses the position information - How RoPE works Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies. The Physics Behind Heat Measurement and LLMs In the early nineteenth century, Joseph Fourier studied how heat spreads through materials. Consider a metal bar with some sections hotter than others. In order to predict how its temperature changes, we need a way to describe that temperature pattern mathematically. Fourier developed methods to represent complicated patterns as combinations of simpler sine and cosine functions. Eventually, these functions became the key tools for studying heat, vibrations, sound, and many other physical phenomena. For reference, a sine wave is a smooth, repeating pattern. Combining waves that repeat at different rates creates a way to describe much more complicated patterns. For example, a broad temperature variation might require slowly changing components, while smaller variations require faster ones. The temperature itself doesn’t have to oscillate over time. A wave can describe how temperature varies across space at a particular moment. Similarly, in the field of LLMs, the mathematical input doesn’t have to represent time or physical distance. It can represent a token’s position in a sequence. Though sine and cosine functions predate Fourier, the 19th century saw their greater use in mathematical physics. Positional encoding, which is a critical way in which LLMs interpret word order, also belongs to the same mathematical family. How Word Order Carries Information Word order carries information that the words alone cannot provide. Consider these two sentences: Alice trusts Bob. Bob trusts Alice. Both contain the same words. Their meanings are different because the positions of the names determine who trusts whom. Software has the same requirement. The expressions income - expenses and expenses - income contain the same variable names and operator, but generally produce different results. Therefore, a language model needs access to two kinds of information: which tokens appear, and how those tokens are arranged. Just recognizing the vocabulary only provides part of what is needed to interpret the sequence. This becomes especially clear with repeated words. In “very, very difficult,” both occurrences of “very” have the same identity, but they occupy different places. A useful representation of this sentence must allow the model to distinguish those two occurrences of the same word. How Does a Model Understand Words A model starts by turning text into numbers. Since a neural network performs calculations on numbers, the text has to first pass through a process called tokenization. A token can be a word, part of a word, punctuation, or another text fragment. For the examples we will see, let’s assume that each word is one token. However, actual token boundaries depend on the tokenizer. Each token receives an identifier. The model uses that identifier to look up an embedding, which is a list of learned numbers. Another name for such a list is a vector. An embedding might look like this: [0.20, -0.40, 0.70, 0.10] Actual embeddings usually contain many more numbers. Their values are adjusted during training so that they become useful for the model’s tasks. Individual numbers generally do not have tidy meanings such as “this is a noun” or “this refers to a person.” At the initial lookup stage, the same token identifier retrieves the same embedding wherever it appears. The embedding for a token doesn’t automatically change just because that token occupies position 2 instead of position 20. Later layers build representations that incorporate context. But at the beginning, the model still needs a way to connect token identity with location. How the Attention Mechanism Makes Sense of the Order Basic attention doesn’t automatically use the order of the list. Transformers rely heavily on a mechanism called attention. Attention lets each token gather information from other tokens by calculating how strongly to weight their contributions. For example, processing a pronoun may require information from a name that appeared earlier. Attention provides a way to connect those pieces of text. Let’s now consider the simplest case. Attention can examine every token, and there is no positional information or other mechanism that indicates order. For the token “trusts,” both example sentences provide the same surrounding token identities: “Alice” and “Bob.” Swapping those names changes their locations, but the basic attention calculation has no position feature telling it which name comes first. More precisely, reordering the input reorders the corresponding output representations. But the reordering doesn’t independently recognize the new grammatical relationships created by the rearrangement. This is the order problem that positional representations address. One objection is that the tokens already occupy different array indexes. Why cannot the model simply use those indexes? This is because an array’s layout doesn’t automatically become an input feature. A program that sums an array receives an ordered structure, but its calculation produces the same answer even after the elements are rearranged. Order can make a difference only when the computation actually uses it. In other words, basic attention needs a way to make the sequence structure affect its calculations. How Location of the Words is Made Available to the Model Positional encoding is the technique that is used to make the location of the words available to the model. A positional encoding converts a location in a sequence into a numerical representation. For example, in the first sentence, Alice is at position 0, trusts at position 1, and Bob at position 2. On the other hand, in the second sentence, Bob is at position 0, trusts is at position 1, and Alice is at position 2. The word “Alice” retains its token identity, even while its position changes. The same for “Bob”. The model can now receive different combinations of content and position for the two sentences. The immediate question is how to represent those positions. We can give the model integers such as 0, 1, and 2 as input. Those numbers contain the necessary ordering information. Sine waves don’t add information that the position number lacks. Their purpose is to express that information in a form with useful numerical structure. For instance, naively adding the number 10000 to every component of an embedding could overwhelm the token’s existing values. More generally, a single scalar requires the model to construct whatever positional features it needs from that one number. A collection of sine and cosine values supplies several structured features directly. Their values remain bounded, and different components change at different rates as positions increase. The Use of Sine and Cosine Functions Sine and cosine describe movement around a circle. A simple explanation involves a hand rotating around a circle. Imagine a circle centered at zero with a radius of one. A hand starts by pointing to the right and turns counterclockwise. The horizontal coordinate of its tip is the cosine of its angle. The vertical coordinate is the sine. As the hand turns, its height rises, falls, becomes negative, and returns to its starting point. Plotting that height against the angle produces a sine wave. The horizontal coordinate produces a cosine wave. Both coordinates remain between −1 and 1. To use this idea for positions, let each step forward in the token sequence rotate the hand by a fixed amount. Position 0 has one pair of coordinates, position 1 has another, and so on. Why do we use both coordinates? This is because a height alone can correspond to two different places on the circle. A point on the upper-left side and a point on the upper-right side can have the same height. Their horizontal coordinates distinguish them. Together, sine and cosine describe the hand’s orientation within a complete turn. However, after one full turn, both readings repeat. This creates the next problem that has to be solved. Why Multiple Rotation Speeds Are A Better Choice Multiple rotation speeds provide a richer position code. One rotating hand cannot distinguish positions separated by a complete cycle. If it returns to its starting point every eight tokens, positions 0 and 8 have identical readings. Now imagine three hands with different speeds. The first completes a turn every eight token positions, the second every 32, and the third every 128. At position 8, the fastest hand returns to its starting orientation. The other two have different readings from their readings at position 0. Their combined coordinates still distinguish those positions. The example uses these speeds to show the point. Moving the position from 0 to 8 shows the fast pair repeating while the other pairs change. Each hand’s vertical coordinate is its sine value, and its horizontal coordinate is its cosine value. The combined position code consists of all six numbers: a sine and cosine value for each speed. More pairs provide more components that change at different rates. The original transformer uses this general construction, with a particular mathematical schedule of frequencies rather than the three example periods above. Faster components change substantially over short distances. However, slower components vary across longer stretches. Therefore, the encoding represents each position through several scales of variation. The model receives a structured numerical pattern rather than a single counter. To be clear, this doesn’t establish unlimited distinguishability. Our three-hand system repeats completely after 128 positions. Practical real-world encodings also face limitations from their frequency choices and numerical precision. The main aim is to have a representation that works well over the relevant sequence lengths. How is the Position Code Used The original transformer adds the position code to the token embedding. In the 2017 paper Attention Is All You Need, the position encoding has the same number of components as the token embedding. Corresponding components are added together before the sequence enters the transformer layers. The central operation is: InputRepresentation = TokenEmbedding + PositionEmbedding For a small illustrative embedding, the addition at position zero could look like this: Token embedding: [0.20, -0.40, 0.70, 0.10] Position-zero encoding:[0.00, 1.00, 0.00, 1.00] Combined input: [0.20, 0.60, 0.70, 1.10] The alternating zeros and ones arise because sine is zero and cosine is one at angle zero. Moving the same token to another position changes the position encoding, which changes the combined input. Placing a different token at the same position changes the token embedding. This gives the transformer a representation influenced by both identity and location. Adding the vectors mixes their components. There is no guarantee that adding two arbitrary vectors preserves two separately recoverable records. Instead, the model trains with this combination throughout learning. Its embeddings and subsequent calculations adapt to use the combined signal. The model also doesn’t need to explicitly say “this is token 17.” Position information can influence which tokens it attends to and how it processes their relationships. The main advantage of this is that relative movement has a consistent structure. Absolute position describes where a token sits in the whole sequence. Relative position describes where it sits compared with another token. For example, positions 5 and 8 are three steps apart. Positions 105 and 108 are also three steps apart. Language frequently depends on relationships that survive this kind of shift. Placing a phrase later in a document should not erase the relationships among its words. Sine and cosine pairs have a useful property here. Advancing by a fixed number of positions always rotates each pair by a fixed angle for that frequency. The rotation depends on the offset, rather than the starting position. With the example fast hand, advancing two positions always produces a quarter-turn. That remains true whether the movement starts at position 0, 7, or 20. This gives the model a reusable numerical relationship between positions. Mathematically, a fixed linear transformation can map a sinusoidal position code to the code at a fixed offset. Neural networks already make extensive use of linear transformations. This explains part of the answer to “why sine waves?” Their value extends beyond producing different numbers for different positions. They provide a regular way to represent movement between positions. The model still has to learn how to exploit that structure. The encoding supplies a learning opportunity, rather than a built-in grammar rule. How Training Uses the Position Information Training turns position information into useful language behavior. Knowing that a name appears before a verb does not, by itself, explain what the sentence means. Consider these examples: Alice trusts Bob. Alice is trusted by Bob. Alice appears first in both, but the relationship changes. An adequate model must consider the verb form and the surrounding words as well as their positions. During language-model training, the system repeatedly tries to predict text. Its parameters are adjusted when its predictions are poor. Patterns involving token identity, sequence position, and surrounding context can become useful because they help improve those predictions. Over many examples, a model can learn relationships involving subjects and verbs, modifiers and nouns, negation and statements, or opening and closing brackets in code. Those relationships may involve adjacent tokens or tokens separated by substantial text. Positional encoding makes order available to this learning process. Attention helps connect relevant tokens. Other learned computations transform the resulting information. This difference also explains why the same positional scheme can support different languages. The encoding represents sequence structure without containing English grammar. Training supplies the language-specific patterns. How Rotary Position Embeddings Work? Rotary position embeddings reuse the same mathematics inside attention. The original method adds a position vector to the input embedding. Later architectures introduce position in other ways. One influential method is Rotary Position Embedding (RoPE). Attention compares numerical representations called queries and keys. A query helps determine what information a token gathers, while keys participate in calculating how strongly other tokens match. RoPE groups components of these representations into pairs and rotates those pairs according to token position. The rotations use sine and cosine functions. When attention compares a rotated query with a rotated key, the positional contribution depends on their relative rotation. That rotation difference corresponds to their relative position. Different component pairs rotate at different speeds, preserving the idea of representing position through multiple frequencies. Other Methods Apart from Sine and Cosine Functions Sine and cosine are useful choices, but transformers have several ways to obtain position information. A learned positional embedding assigns each supported position a trainable vector. Training adjusts those vectors rather than generating them with a fixed trigonometric formula. Another method, ALiBi, modifies attention scores using a penalty proportional to the distance between tokens. It introduces a preference for nearby context without adding sinusoidal position vectors to the token embeddings. There is also an important qualification to the earlier order-blind attention example. Generative language models commonly use a causal mask, which prevents a token from attending to later tokens. Different positions consequently have access to different prefixes. That pattern itself provides structural information. Causal transformers can learn positional behavior even without an explicit positional encoding. The claim is that basic, unrestricted attention lacks an inherent ordering mechanism. A complete model may receive order information through several architectural choices Conclusion Ultimately, being able to calculate a position encoding for a very large position doesn’t mean a model can reliably understand an arbitrarily long document. A formula can produce a code for position one million, while the model may have little experience using such positions or relating tokens across such large distances. Memory and computation impose additional constraints. However, mathematics developed long ago for describing physical patterns provides an effective numerical language for representing sequence position. Sine and cosine give that representation bounded values, multiple scales, and consistent relationships under shifts. Training then teaches the model how those positional relationships interact with the content of language. References: