In 2022, Google researchers discovered something strange: when they asked their largest AI models to solve math problems, the models performed barely better than random guessing. But when they changed the prompt to include a few examples showing step-by-step reasoning, accuracy jumped from near-zero to 58%. The models hadn't changed. The training data hadn't changed. Only the prompt had changed.

This technique, called chain-of-thought prompting, revealed that language models already possessed reasoning abilities that no one knew how to unlock. The discovery upended assumptions about what these systems could do and how they worked.

The Accidental Discovery of Hidden Abilities

Jason Wei and his colleagues at Google Research weren't trying to invent a new AI capability. They were trying to measure one they assumed didn't exist. Large language models had shown impressive performance on many tasks, but multi-step reasoning seemed beyond their reach. When asked to solve grade-school math problems, even models with 100 billion parameters struggled.

The team decided to try something simple: instead of just showing the model question-answer pairs, they included the reasoning steps in between. They manually wrote out how a person might think through problems like "Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?"

The results shocked them. Their 540-billion parameter model, PaLM, achieved 58% accuracy on the GSM8K benchmark of math word problems. This beat the previous state-of-the-art of 55%, which had required fine-tuning a smaller model on thousands of examples and using a separate verification system. PaLM needed just eight examples in its prompt.

Why Size Suddenly Mattered

Chain-of-thought prompting only works with sufficiently large models. Below roughly 100 billion parameters, showing reasoning steps produces almost no improvement. Above that threshold, performance takes off.

This pattern suggests something profound about how these models learn. Smaller models can memorize patterns and retrieve facts, but they lack whatever internal structure enables step-by-step reasoning. That structure emerges somewhere around 100 billion parameters, but it remains dormant until the right prompt activates it.

The researchers tested this across three model families: LaMDA (ranging from 422 million to 137 billion parameters), GPT-3 (up to 175 billion), and PaLM (8 billion to 540 billion). In every case, the smallest models showed flat performance whether you used chain-of-thought prompting or not. Only the largest models responded to the technique.

This wasn't gradual improvement. It was a phase transition. Models either had the capacity for prompted reasoning or they didn't.

The "Let's Think Step by Step" Shortcut

If manually writing reasoning chains for every problem seemed tedious, researchers soon found an even simpler approach. Adding the phrase "Let's think step by step" to the end of a question, with no examples at all, triggered similar reasoning behavior.

This "zero-shot" chain-of-thought prompting worked across arithmetic, commonsense, and symbolic reasoning tasks. The model would generate its own intermediate steps, then arrive at an answer. The phrase acted like a switch, changing the model's behavior from direct answer generation to deliberate reasoning.

The mechanism remains somewhat mysterious. These models weren't explicitly trained to respond to that particular phrase. They learned it from patterns in their training data, where step-by-step reasoning often followed similar cues. The models generalized from those patterns to understand that certain prompts call for certain types of responses.

What the Model Actually Learns

Chain-of-thought prompting doesn't teach models to reason. It reveals reasoning they've already learned but wouldn't normally express. The training process on vast text corpora apparently builds internal representations that support multi-step inference, even though the training objective is simply predicting the next word.

This creates an odd situation: the model possesses capabilities that emerge only under specific conditions. Without the right prompt, those capabilities remain inaccessible. It's like discovering that someone speaks a language fluently but only when addressed in that language first.

The intermediate steps the model generates aren't just for show. When researchers used "self-consistency" techniques—generating multiple reasoning paths and taking a majority vote—accuracy on GSM8K jumped to 74%. The model was actually using its reasoning chains to reach better answers, not just mimicking the format.

The Decomposition Advantage

Why does showing your work help? For the same reason it helps humans: complex problems become manageable when broken into pieces.

A question like "If a store sells 3 apples for $2, how much do 15 apples cost?" requires multiple steps: recognizing the ratio, determining how many groups of 3 are in 15, and multiplying. Standard prompting asks the model to leap directly to the answer. Chain-of-thought prompting lets it work through each step.

This decomposition also provides interpretability. You can see where the model's reasoning goes wrong. If it calculates 15 ÷ 3 = 6 instead of 5, the error is visible. With direct answer generation, you only see the wrong final result.

The language-based nature of these reasoning chains matters. Because the model thinks in words rather than opaque internal representations, any problem a human could solve through language becomes accessible to the same technique.

When Prompting Beats Training

Chain-of-thought prompting requires no fine-tuning, no additional training data, no modification to model weights. It works through in-context learning alone—the model's ability to adapt its behavior based on examples in the prompt.

This challenges the assumption that improving AI capabilities requires bigger datasets and longer training runs. Sometimes the capability already exists, waiting for the right way to elicit it. The researchers needed eight examples to match systems that had been fine-tuned on thousands.

The technique has since spawned variations: Auto-CoT automatically generates reasoning chains instead of requiring manual composition. Tree-of-thought prompting explores multiple reasoning branches. Self-taught reasoners use the model's own generated chains as training data.

But the core insight remains: these models contain latent abilities that only surface under the right conditions. We're still discovering what they can do, not by making them larger, but by learning how to ask.