How large language models work
Tokens, next-token prediction, training and fine-tuning, the context window, temperature and knowledge cut-offs: the few ideas that explain almost everything ChatGPT, Claude and Gemini do.
🎯 By the end of this lesson you will be able to:
- Explain what a token is and why it matters for cost and limits
- Describe how an LLM writes text one token at a time
- Use the ideas of context window, temperature and knowledge cut-off to predict how a model will behave
In this lesson
The big idea: predicting the next word
A large language model (LLM) is a deep learning model trained on a very large amount of text. At its heart it does one thing: given some text, it predicts what comes next. It writes a whole answer by predicting one small piece, adding it to the text, and predicting again.
That sounds too simple to explain an AI that writes code and essays. The surprise of the last few years is that, when a model is large enough and has learned from enough text, getting good at "what comes next" requires picking up grammar, facts, reasoning patterns and writing styles along the way.
Tokens: how models read
Models do not read letters or whole words. They read tokens: common chunks of text. A short common word is often one token; a long or rare word is split into several. In English, a token is very roughly three-quarters of a word on average, so 1,000 words is about 1,300 tokens.
How text might be split into tokens
Unbelievable results!
Un | believ | able | results | !
The exact split depends on the model. Other languages and code often use more tokens per word.
How an LLM is trained
- Pre-training: the model learns to predict the next token across a huge collection of text, such as books, websites and code. This gives it broad knowledge of language and the world, and it is the expensive part.
- Fine-tuning: the model is trained further on examples of helpful conversations, so it follows instructions instead of just continuing text.
- Learning from feedback: people (and sometimes other models) rate answers, and the model is tuned to prefer the better ones: helpful, honest and safe. This is often called reinforcement learning from human feedback (RLHF).
The context window: the model's working memory
Everything the model can consider for one answer must fit into its context window: instructions, the conversation so far, any documents you paste in, and the answer it is writing. Context windows have grown a lot, but they are still a limit, and very long inputs cost more and can make the model miss details.
Temperature: predictable or creative
When choosing the next token, the model can always pick the most likely one, or sometimes pick a less likely one. A setting called temperature controls this. Low temperature gives focused, repeatable answers, which is good for extracting data or classifying. Higher temperature gives more varied, creative answers, which is good for brainstorming. This is also why the same question can give different answers each time.
Knowledge cut-off
A model learns from text collected up to a certain date, its knowledge cut-off. It does not know about events after that unless the application gives it fresh information, for example through web search or your own documents. Lesson 9 shows how teams do this with retrieval-augmented generation (RAG).
| Behaviour you notice | Explained by |
|---|---|
| Different answer to the same question | Temperature and sampling |
| Forgets the start of a very long chat | Context window limits |
| Doesn't know last month's news | Knowledge cut-off |
| Costs more for long documents | Pricing per token |
| Sounds confident but is wrong | It predicts plausible text, not verified facts (Lesson 4) |
What this means for your role
Developer
Count tokens, set temperature on purpose (low for structured tasks), and send only the context the model needs. These three choices drive quality, speed and cost in every AI feature you build.
Tester / QA
Because output varies, run the same test several times and judge quality across runs. Test long inputs near the context limit and questions about recent events after the cut-off.
Business Analyst / PO
Write requirements that say what context the AI needs, such as which documents or data it may use, and what should happen when it does not know the answer.
Tech Lead / Architect
Context size, model choice and token usage are architecture decisions with direct cost impact. Estimate tokens per request early.
Project / Delivery Manager
AI costs scale with usage, measured in tokens. Ask for a cost-per-request estimate and expected volume before committing to a launch plan.
Scrum Master / Agile
Help the team agree conventions, such as when to start a fresh chat, how to share good prompts, and which tool to use for which kind of task.
Try it: see these ideas in action
- Ask any AI assistant the same creative question three times in new chats, for example "Suggest a name for a team newsletter". Notice the variation.
- Ask it about a very recent event, then ask what its knowledge cut-off is.
- Paste a long document and ask about a detail near the start. Then ask the same in a fresh chat with only the relevant paragraph. Compare the answers.
Key takeaways
- An LLM writes by predicting the next token again and again.
- Tokens are the unit of limits and cost: roughly three-quarters of an English word each.
- Everything for one answer must fit in the context window; models do not remember between chats by default.
- Temperature controls variety, and the knowledge cut-off limits what the model knows by itself.
Key terms
- LLM
- Large language model: a deep learning model trained on large amounts of text to predict the next token.
- Token
- A chunk of text that a model reads and writes; the unit for limits and pricing.
- Context window
- The maximum amount of text (in tokens) a model can consider at once, including its answer.
- Temperature
- A setting that controls how varied or predictable the model's choices are.
- Knowledge cut-off
- The date after which a model has no training data.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. How does an LLM produce a long answer?
2. Roughly how many tokens is 1,000 English words?
3. You need the model to extract invoice numbers the same way every time. Which temperature suits this?
4. Why does a chatbot forget a previous conversation in a new chat?
5. A model doesn't know about an event from last month. What is the most likely reason?