Ask Nicely, Then Stuff the Context

Deep down, at the lowest layer of the architecture, a language model is a machine that guesses the next word. For a while, the whole industry fooled itself into believing that the craft of building with AI was simply the art of arranging the words in front of that machine cleverly enough.

That misconception held back system design for a long time, until the cracks that opened up in production forced us to wake up.

I. The illusion of learning without touching the weights

In May 2020, Tom Brown's team published GPT-3.¹ It carried 175 billion parameters, was pre-trained on roughly 300 billion tokens, and had a context window of 2,048 tokens. The paper, Language Models are Few-Shot Learners, wasn't selling a chat box. It was selling an observation that sent a whole industry scrambling to change careers: a large enough model can do a new task without fine-tuning.

All you had to do was put a few examples in front of the question (zero-shot means none, one-shot means one, and few-shot usually means 10 to 100, as many as fit in 2,048 tokens), and the model would imitate the pattern. On LAMBADA, a test of predicting the last word of a passage, GPT-3 scored 76.2% with no examples and jumped to 86.4% with a full set.

Hand-drawn diagram comparing fine-tuning with GPT-3 zero-shot, one-shot and few-shot prompting, with LAMBADA scores — In-context learning, adapted from Brown et al. (2020)

Read the figure left to right. On the left is the old way: each example pair goes through the model, produces a gradient, and nudges the weights $\theta$ a little, thousands of times over. On the right is what GPT-3 does: the same 175-billion-parameter model, untouched. The only difference between the zero-shot, one-shot and few-shot rows is how many examples sit in the context ahead of the question. Look at the middle row: adding exactly one example drops the score to 72.5%. How you lay the text out in front of the model weighs as much as what the text says.

What made this technology both frightening and seductive wasn't the score. It was that throughout the whole "jump", not a single gradient update took place. The model "learned" inside one forward pass. The weights $\theta$ stood still like a wall; the only thing that changed was the text in front of the question.

This is where prompt engineering was born. Nobody touched the code or the parameters; people only manipulated what the model got to see. And with it came a very dangerous belief: engineers began to mistake the context window, which is only temporary working memory, for the long-term memory of the whole system.

II. A machine that tells you what you want to hear, and the truth about hallucination

An autoregressive model is brutally simple. Given a sequence $x_1 \ldots x_t$ , it performs exactly one operation: it estimates the probability $P(x_{t+1} \mid x_{\le t})$ .

It has no physical slot, no database, in which to store "last month's warehouse revenue". Unless that number happened to be in the internet scrape it was pre-trained on, it simply does not exist in its weights. And the fatal part is that when it doesn't know, it never returns an empty cell. Trained by maximum likelihood, it only knows how to assign high probability to sequences that sound plausible; the decoding step then picks out whichever token sounds most natural.

Ask it for an internal figure and it will recite an invented number without blinking, because on the internet the phrase "last month's revenue was" is usually followed by a number. It is completing a statistical pattern, not opening the ledger. Scolding the model for "hallucinating" is really being angry at a loss function that was never built to understand what "correct books" means.

In 2025, Kalai and colleagues laid this bare with math:² hallucinations arise from the ordinary statistical pressure of pretraining, and are then amplified by the way we grade benchmarks. Because most tests give zero points for "I don't know", guessing always has a better expected score than honesty. The model is optimized to sit exams, and a bold exam-taker never hands in a blank sheet.

III. InstructGPT: when the engineers were steered by the loss function

GPT-3 was very good at completing text and very bad at doing what it was told. Give it an instruction and it would happily continue it as if it were the opening line of an essay. In early 2022, Long Ouyang's team released InstructGPT and its three-layer pipeline,³ which officially turned LLMs into machines that tell you what you want to hear rather than machines that carry out logic.

Layer 1: SFT (supervised fine-tuning). About 13,000 prompts paired with human-written answers, trained for 16 epochs. The most unsettling detail: after the very first epoch, validation loss had already started to rise, the classic sign of overfitting. They kept training on purpose. The paper states the reason plainly: the more they trained, the higher both the reward-model score and human ratings went. I read that as a confession: they accepted a model that fit the data worse, as long as it was pushed into the distribution of an "assistant" that people liked to listen to.

Layer 2: the reward model. For each prompt, the model generates 4 to 9 answers for labelers to rank. The judge is not the 175-billion-parameter GPT-3 but a 6-billion-parameter model, initialized from SFT, with its unembedding layer removed. It does exactly one thing: output a single scalar standing for how pleased a human would be.

Layer 3: PPO (proximal policy optimization). The SFT model becomes the policy. Their objective is defined as:

\begin{aligned} \text{objective}(\phi) = {} & \mathbb{E}\left[ r_\theta(x, y) - \beta \log \frac{\pi^{RL}_\phi(y \mid x)}{\pi^{SFT}(y \mid x)} \right] \\ & + \gamma\, \mathbb{E}_{x \sim D_{\text{pretrain}}}\left[ \log \pi^{RL}_\phi(x) \right] \end{aligned}

The reward $r_\theta$ is the reward model's score. The KL penalty (weighted by $\beta$ ) is the rein that holds the model back, so it can't drift far from the original SFT model just to squeeze out more points. The $\gamma$ term is added to reduce the alignment tax: the price paid when a model becomes better behaved but worse on public NLP benchmarks.

Hand-drawn diagram of the three InstructGPT training stages: SFT, reward model and PPO with a KL penalty — The three layers of InstructGPT, adapted from Ouyang et al. (2022)

Each column in the figure is one layer. In the first column, the loss curve is an illustration rather than the paper's data, but its shape is exactly what the paper admits: it bottoms out after the first epoch and climbs, and training carries on anyway. In the last column, the rope tying the policy to the frozen SFT copy is the KL penalty from the formula.

So what do these three layers add up to? The whole architecture revolves around words and optimizes the reader's sense of comfort. It does not optimize calling an API, it doesn't teach the model to read an error log from a terminal, and it doesn't teach it to fix a broken line of code. When we put this system into production and expect it to work like a platform engineer, we are forcing it to solve a problem that does not exist anywhere in its objective.

IV. A passive encyclopedia, and the architectural trick called RAG

Once people saw that incantations, whether the worked reasoning examples of chain-of-thought⁴ or the line "Let's think step by step",⁵ could not update a live price list or restart a server cluster, the industry turned to stuffing the context. The RAG (retrieval-augmented generation) wave of 2023 convinced many people that plugging an internal database into the prompt was all it took.

But the RAG of 2023 enterprise meeting rooms and the RAG of Patrick Lewis's original 2020 paper⁶ are two entirely different species.

The original architecture draws a sharp line between two kinds of memory. Parametric memory, inside the weights, is BART-large with 400 million parameters. Non-parametric memory, outside the weights, is a Wikipedia index of 21 million passages, searched with DPR (dense passage retrieval). DPR doesn't use classic keyword counting like BM25; it runs two BERT networks side by side, one to encode the question and one to encode documents, then looks for the largest inner product (MIPS).

The clever part of RAG 2020 is that it is fine-tuned end to end. Once the generated answer is compared with the target, the gradient flows back through the BART generator and all the way into the question encoder. The retriever isn't allowed to sit still: it has to learn to place questions in vector space so that it fetches exactly the passages that help the generator get the answer right.

Hand-drawn diagram comparing the original 2020 RAG with a typical 2023 enterprise RAG pipeline — Original RAG and enterprise RAG, adapted from Lewis et al. (2020)

Read both columns top to bottom. The middle stretch is nearly identical on both sides: a store of pre-cut passages (the original also cut Wikipedia into 100-word chunks), searched by vector, returning the top few. The difference is in the dashed lines. On the left, the gradient flows back into BART and on into the question encoder; the two blocks with a flame are the ones being trained, while the document encoder stays frozen. On the right, every block is frozen, and the way back is crossed out.

So what do enterprise RAG systems do today? They borrowed the name and cut away that backward pass entirely. The pipeline is whittled down to: chunk the text mechanically (say 512 tokens with 50 overlapping), embed it with an off-the-shelf frozen model, take the top 4, paste them straight into the prompt. Nothing learns anything. The LLM is demoted to a passive reading-comprehension machine. When the chunking and the choice of embedding model are wrong from the start, stuffing dozens more documents into the context is just shovelling more garbage into the system.

V. Context rot: buried in a pile of paper

Even when the retriever does its job and fetches the right document, the model can still take a fatal fall in the way it reads the context.

In July 2023, Lost in the Middle (Liu et al., TACL 2024)⁷ measured this stuffing disease precisely. When they moved the passage containing the answer from the start of GPT-3.5-Turbo's context to the end, one position at a time (with 20 documents), the scores traced a merciless U-shaped curve.

Hand-drawn Lost in the Middle U-curve: GPT-3.5-Turbo accuracy by position of the answer document — The U-curve, redrawn from data in Liu et al. (TACL 2024, CC BY 4.0)

This figure is redrawn from the paper's own table. The dashed blue-grey line is the 56.1% the model scores closed-book, with no documents at all. The shaded area is where the curve dips below that line. The strip of squares at the bottom is the context window: the closer to the middle, the fainter a document becomes to the model.

The model only makes good use of information at the start of the context (primacy bias) and at the end (recency bias). Information stuck in the middle sees the score fall off a cliff. In the worst case, accuracy with the answer in the middle is lower than closed-book accuracy, with no documents at all (56.1%). Literally, stuffing in more documents makes the model answer worse.

A longer window doesn't save you either: GPT-3.5-Turbo 16K, with a window four times longer, produces numbers almost identical to the standard model at every position. And this is not a disease of old models. In July 2025, Chroma's researchers tested 18 leading models (including GPT-4.1, Claude 4 and Gemini 2.5) and concluded that every one of them degrades as inputs get longer.⁸ They called it context rot: the longer the context, the more it decays.

Suppose the passage with the right answer ranks 7th of 20 retrieved. If we blindly paste all 20 in order of embedding score, that precious passage lands right in the dead zone in the belly of the context. A high embedding score never rescues a bad position in the context.

VI. Beyond the space of language

Two disillusionments, one root.

The first: realizing the model is not an all-knowing teacher, only a closed memory with a habit of completing patterns.

The second: realizing that pumping in internal documents only gets you halfway, because the work of a company (writing to a database, sending an email, fixing a server) demands that the system act, while reading is entirely passive.

Every effort around the model's mouth, from polishing prompts to stuffing context, in the end only changes the "text so far" in $P(\text{next word} \mid \text{text so far})$ . None of it changes the action space, and none of it changes the loss function. Both trades are necessary, and neither will ever be enough.

An architecture only becomes whole when the words the model prints can break out of language, become a program the operating system reads as a command, and touch the physical world.

That door opens onto the era of ReAct, brittle regular expressions, and the collapse of the "virtual departments" in Part 02.

Tom B. Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020. LAMBADA results in Table 3.2; zero/one/few-shot in Figure 2.1. ↩
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang, Why Language Models Hallucinate, 2025. ↩
Long Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022. Announced on OpenAI's blog in January 2022; the arXiv version appeared on 4 March 2022. SFT and the reward model in §3.5, the objective in equation (2). ↩
Jason Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022. ↩
Takeshi Kojima et al., Large Language Models are Zero-Shot Reasoners, NeurIPS 2022. This is the paper that introduced "Let's think step by step". ↩
Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020. The fixed document encoder in §2.4; the BM25 comparison in Table 6. ↩
Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 12 (2024), pp. 157–173. Data from Tables 1 and 6. ↩
Kelly Hong, Anton Troynikov, Jeff Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, July 2025. ↩