How Large Language Models Actually Work (Without the Math Degree)

Here is a strange thought: the chatbot that just wrote your email, explained quantum physics, and helped debug your code does all of it with one basic trick, repeated billions of times — guessing the next small piece of text.

That is not a dismissal — it is the actual mechanism. Understand it, and almost everything about large language models (LLMs) makes sense: why they are so fluent, why they invent facts with total confidence, why they “forget” the start of a long conversation, and why making them bigger changed everything. No maths degree required.

Our running example is this sentence beginning:

“The cat sat on the…”

By the end of this article, you will know exactly what happens inside a model between reading those words and answering “mat.”

Key takeaways

• An LLM never reads words the way you do. It reads tokens — small chunks of text — converted into numbers.
• Each token is turned into an embedding: a list of numbers that places it on a “meaning map,” where tokens used in similar ways sit close together.
• Attention lets every token look at the others and adjust its meaning in context — the breakthrough of the 2017 Transformer paper.
• Training teaches the model one skill at enormous scale: predict the next token. Answering you (inference) is that same skill, run one token at a time.
• Because the model predicts what sounds right rather than checking what is right, it can hallucinate — confidently inventing facts is a side effect of the design, not a mysterious glitch.

Step 1: Words become tokens

A language model cannot work with raw text. As IBM’s explainers put it, machine learning models require numbers as inputs — they cannot process plain text in its raw form. So the first step is translation, and the unit of translation is the token.

A token is a chunk of text: sometimes a whole common word, sometimes part of a word, sometimes a single character or a punctuation mark. One widely used explainer example splits “unbelievable” into three tokens — “un”, “believ”, “able” — and OpenAI’s own documentation famously splits “tokenization” into “token” + “ization”. Short, common words like “the” or “cat” are usually a single token each.

For our sentence, a tokenizer might produce something like:

• “The” · ” cat” · ” sat” · ” on” · ” the” · “…”

(For example — the exact split depends on the model’s tokenizer.)

A useful rule of thumb, published by OpenAI: one token is roughly four characters of English text, or about three-quarters of a word — so 100 tokens is about 75 words. This is why AI services talk about tokens constantly: tokens are the model’s unit of reading, writing, memory and cost, all at once.

Each token in the model’s vocabulary has an ID number. From here on, the model works only with numbers. The words are gone.

Step 2: Tokens become embeddings — a map of meaning

A token ID by itself says nothing. Token #4521 is not inherently closer to token #4522 than to token #9. So the model converts each token into an embedding: a long list of numbers (a vector) that represents that token’s meaning and usage.

IBM describes the idea well: embeddings represent words so that their positions capture semantic relationships — tokens with similar meanings end up positioned close to each other, and the distance and direction between them encode how similar they are.

Picture a map. On a real map, coordinates place cities near each other when they are geographically close. On the model’s meaning map, coordinates place “cat” near “kitten” and “dog,” far from “democracy” or “carburettor.” No human drew this map by hand. The positions were learned during training, from seeing how tokens are actually used in text, billions of times.

A crucial subtlety: the starting embedding is only the beginning. In our sentence, “sat” means one thing in “the cat sat on the mat” and another in “she sat the vase on the table.” The model’s next mechanism exists precisely to adjust each token’s representation using its neighbours. That mechanism is attention.

Step 3: Attention — every token looks at every other token

In 2017, researchers from Google and the University of Toronto published a paper with an unusually confident title: Attention Is All You Need. It introduced the Transformer architecture — the “T” in ChatGPT — and modern LLMs are built on it.

The paper’s core move, in plain language: instead of reading a sentence strictly one word at a time and trying to remember everything (the older approach), let every token look directly at every other token and decide which ones matter to its meaning.

In “The cat sat on the…”, attention is what lets the model connect “sat” back to “cat” (the sitter) and forward to “on the…” (sitting usually happens on something). It is how “it” in a later sentence gets linked to the cat rather than the mat. Each token’s embedding gets updated, layer after layer, into a context-aware representation: not just “what this token means in general,” but “what this token means here.”

Two honest simplifications: real attention uses multiple “heads” checking different relationships at once, across dozens of layers. But the core idea really is that simple — context is gathered by looking, not by remembering sequentially — and that parallelism is a big reason Transformers could scale where older designs could not.

Step 4: Training — one lesson, repeated at planetary scale

Before an LLM can answer you, it is trained, and training is where the “large” really comes from. Here is the entire lesson plan:

Show the model an enormous amount of text. Hide the next token. Make it guess. Correct it. Repeat.

Given “The cat sat on the”, the model assigns probabilities to possible next tokens — “mat” high, “roof” lower, “banana” tiny — then finds out the real next token and nudges its billions of internal settings (its weights) slightly, so next time it guesses a little better. Done across trillions of tokens of books, articles, code and conversation, this humble exercise forces the model to absorb grammar, facts, styles, reasoning patterns and the shape of arguments — because all of those help it predict what comes next.

Notice what training is not: nobody programmes in rules of English, and nobody types facts into a database. The knowledge ends up stored implicitly in the weights — more like a skill than like a library.

Step 5: Inference — answering you, one token at a time

When you chat with the finished model, you are seeing inference: the trained weights, frozen, put to work. Your prompt is tokenised and embedded, attention weaves the context together, and the model produces a probability ranking for the next token. One token is chosen, added to the text, and the whole process runs again — and again — until a complete answer exists, one token at a time. That is why chatbot answers stream onto your screen word by word: they are literally being generated piece by piece.

Two consequences follow. First, the model does not plan the whole answer before starting; it commits token by token, which is part of why answers can drift or contradict themselves. Second, everything the model “knows” about your conversation must fit into its working memory — which brings us to the context window.

The context window: the model’s desk

The context window is the maximum amount of text — measured in tokens — that a model can consider at one time: your prompt, the conversation so far, and its own reply all share the same space. A simple illustration from The Hindu: a model with a context window of 8,000 tokens can handle roughly 6,000 words of text at once.

Think of it as the size of the model’s desk. Everything on the desk can be attended to; anything that falls off the desk effectively does not exist for that answer. In a very long chat, the earliest messages fall off — which is why a chatbot can “forget” something you told it an hour ago. It is not being careless; the text is literally no longer in front of it. And between separate conversations, the model has no memory at all unless the product saves and re-sends the history.

Why scale changed everything

If next-token prediction is so simple, why did it only recently produce systems this capable? Because prediction quality improves relentlessly with three ingredients multiplied together: more text, more weights, more computing power. A small model learns that “sat on the” is often followed by “mat.” A vastly larger one absorbs the patterns behind law, poetry and Python errors — the same single skill, at a scale where quantity starts to look like quality. The Transformer mattered precisely because it could scale this way: attention processes whole passages in parallel, so bigger training runs stay practical.

Why LLMs sound confident when they are wrong

Now the famous failure mode makes sense. A hallucination is when a model produces output that sounds plausible but is factually wrong or entirely fabricated — an invented study, a confident wrong date, a legal case that never existed. IBM’s definition adds the key: LLMs are trained to predict the next token, and “when information is sparse or ambiguous, these models might create confabulations.”

Follow the mechanism and you can see why. The model has no truth-checking step. It does not pause to ask “do I actually know this?” — it continues the pattern. Fluent, confident style is itself a pattern in its training data, so a guessed answer comes out wearing the same confident tone as a known one. And as IBM’s researchers point out, the model “isn’t pulling facts from a database but making educated guesses” — there is no database underneath to look anything up in.

This is not a bug that a patch will eliminate; IBM notes hallucinations can be reduced but not fully removed, because they grow out of the same prediction engine that makes the model useful. The practical rule stands: enjoy the fluency, verify the facts — especially names, numbers, quotes and citations.

What an LLM is not doing

To finish, four honest corrections to common impressions:

• Not searching a database. Unless a product deliberately adds a search or retrieval tool, the model answers from its weights alone.
• Not understanding the way you do. It manipulates patterns in language with astonishing skill — not human-like comprehension backed by lived experience.
• Not remembering you between chats (on its own). Any memory is the app re-supplying text into the context window.
• Not trying to deceive you when it errs. A hallucination is a statistical stumble, not a lie. The model has no beliefs to lie about.

If this leaves you curious where LLMs sit in the bigger picture, our beginner’s guide to what artificial intelligence actually is gives the full landscape, and everything sits in the Artificial Intelligence section.

Frequently asked questions

What is the difference between training and inference?
Training is the long, expensive learning phase: the model reads huge amounts of text and adjusts its weights by predicting hidden next tokens. Inference is using the finished model: weights stay frozen while it generates answers, one token at a time, for each user request.

Is an LLM just a fancy autocomplete?
Mechanically, prediction of the next token is its engine — but “just autocomplete” undersells what that engine must learn to predict well at scale: grammar, facts, logic patterns, code structure and conversational flow all become necessary. The honest summary: autocomplete is the mechanism; the capabilities are what emerged from scaling it.

Why do LLMs need so much text to train on?
Because their knowledge lives in statistical patterns, and reliable patterns need many examples. Predicting the next token well across every topic requires seeing language in an enormous range of situations — which is why training corpora are measured in trillions of tokens.

Can an LLM know that it is wrong?
Not reliably. The model generates by probability, not by checking a fact store, so it has no built-in alarm that separates “I know this” from “this sounds right.” Some systems are trained to say “I don’t know” more often, but the underlying mechanism cannot guarantee it — which is why verification remains your job.

What is a context window in one line?
The maximum number of tokens a model can look at in a single request — its working memory for your prompt, the conversation history, and its own answer combined.

Sources

1. Vaswani, A. et al. — Attention Is All You Need, arXiv:1706.03762 (2017).
2. IBM — What Are Word Embeddings? (living page, accessed 21 October 2026).
3. IBM — What Are AI Hallucinations? (published 1 September 2023; updated 31 July 2026).
4. IBM — Smarter Memory Could Help AI Stop Hallucinating (living page, accessed 21 October 2026).
5. IBM Research — What is generative AI? (living page, accessed 21 October 2026).
6. The Hindu — What is the context window? (12 January 2026).

Frequently Asked Questions

What is a token in a large language model?

A token is a chunk of text a language model reads and writes: sometimes a whole common word, sometimes part of a word, a character or punctuation. One token is roughly four characters of English text, or about three-quarters of a word, so 100 tokens are about 75 words. Tokens are the model’s unit of reading, writing, memory and cost.

How does attention work in a Transformer model?

Attention lets every token look directly at every other token and decide which ones matter to its meaning. In “The cat sat on the…”, attention links sat back to cat and forward to the surface being sat on. Each token’s representation is updated layer by layer, turning a general meaning into a context-aware meaning for that sentence.

What is the difference between training and inference?

Training is the long learning phase. The model reads enormous amounts of text, guesses each hidden next token, is corrected and adjusts its weights slightly, repeated at huge scale. Inference is using the finished model: the weights stay frozen while it generates an answer for a request, choosing and adding one token at a time.

What is a context window in one line?

A context window is the maximum amount of text, measured in tokens, that a model can consider at one time. The prompt, conversation history and the model’s own reply all share that space. A window of 8,000 tokens holds roughly 6,000 words, and in a very long chat the earliest messages can fall out and be forgotten.