How Is ChatGPT Made? Demystifying the Three-Stage Training Pipeline of Large Language Models

Every time you fire off a question to ChatGPT and it hands back something fluent and right on the money — what you're actually seeing is three completely different training stages stacked on top of each other. And here's the thing: each stage has its own personality, its own job. Professor Hung-yi Lee has this wonderful way of describing them — creating the world, showing the way, and surpassing oneself. So let's walk through all three. And along the way I want to plant one idea in your head, because it's the part most people miss: capability and alignment are two separate things, forged completely independently.

Let's start at the beginning. Stage one — creating the world. This is pre-training, and it's the stage where the model builds its knowledge. It's also, by a wide margin, the most computationally insane part of the entire pipeline.

The goal here is almost laughably simple to state: text completion. You give the model a span of text, and it predicts the next token. That's it. That's the whole game. And the beautiful part is the method — it's called self-supervised learning, and it runs on trillions of tokens of raw web text. No humans labeling anything. Why? Because the answer is already hidden inside the text itself. You just cover up the next word, let the model take a guess, and correct it when it's wrong. Do that across the entire internet, over and over, and something remarkable happens. By pushing this one tiny task — predict the next token — to its absolute extreme, the model is forced to pick up grammar, facts, common sense, even a real degree of reasoning. It learns the world in order to finish your sentences.

What you get out the other end is what we call a base model. And here's the twist that surprises people: this thing is incredibly knowledgeable, but it does not follow instructions. If you ask it a question, it might not answer you at all. Instead it might just continue with a whole list of other questions you could ask next — because, statistically, that's the kind of text that shows up most often on the web. It's brilliant, and it's useless as an assistant.

So hold onto this, because it's the key takeaway of the whole episode: almost all of the model's knowledge and capability is formed right here, in stage one. Everything that comes after is not about pouring in more knowledge. It's about taking this raw, sprawling skill set and shaping it into something you can actually use.

Which brings us to stage two — showing the way. This is instruction fine-tuning, sometimes called supervised fine-tuning, or SFT. The whole point is to take that text-completion machine and turn it into an assistant that actually answers you.

The method is much more hands-on. You bring in human annotators, and they write high-quality pairs — an instruction, and the ideal answer to it. "Write me a poem." Here's a good poem. "Summarize this article." Here's a good summary. You gather a pile of these examples and you fine-tune the model on them. And here's what's fascinating: the amount of data is tiny compared to pre-training. Microscopic, really. But its job isn't to teach — its job is to unlock. It takes all that broad capability the model already absorbed and steers it into a single behavior pattern: understand what a human is asking, and carry it out. You're not adding skills. You're flipping a switch that says, "now act like an assistant."

And then there's stage three — surpassing oneself. This is RLHF, reinforcement learning from human feedback, and it exists because stage two has a ceiling. Think about it: relying on humans to write perfect demonstrations only gets you so far, because humans simply can't write the ideal answer to every possible question. And honestly, a lot of the time we can't even put into words what makes one answer better than another. We just know it when we see it.

So RLHF comes at the problem from a much cleverer angle. Here's how it flows. First, you have the model generate several different answers to the same question. Then — and this is the smart bit — you ask humans not to write the perfect answer, but simply to rank the answers the model produced, from best to worst. And ranking is so much easier than writing from scratch. You don't need to be a poet to know which of four poems is the best one. Next, you take all those rankings and you use them to train a separate model called a reward model, whose entire job is to predict which answer a human would prefer. And finally, you turn that reward model loose as a kind of coach — using reinforcement learning, OpenAI used an algorithm called PPO, to nudge the original model toward producing the high-scoring kind of answer more and more often.

This is exactly the result OpenAI put on the table in their InstructGPT paper, and it's a stunner. A smaller model that had been properly aligned could beat a much, much larger model that hadn't been — beat it on what humans actually preferred. So let that sink in. What makes ChatGPT stand out isn't that it's bigger. It's that it's better aligned.

Now, there is a price tag here, and it's worth naming. It's sometimes called the alignment tax. When you align a model like this, it can actually slip backward a little on certain pure-capability benchmarks. That's the trade you're making for "safer and more obedient," and in practice engineers have to deliberately balance it — you don't get the good behavior entirely for free.

So let's pull the three stages together, because there's really one sentence worth carrying out of all this. Pre-training gives the model its soul — its knowledge and its raw capability. And then instruction fine-tuning and RLHF give it its personality and its sense of proportion — its usability, and its safety.

And once you see it that way, a whole bunch of confusing things suddenly make sense. It explains why a model can clearly know something but still not be willing to say it in the exact format you wanted — that's the alignment layer doing its job, for better or worse. It explains why you can take one single base model and align it into assistants with completely different styles and voices. And it explains why, in those last two stages, the quality of your data matters far, far more than the quantity — because you're not pouring in knowledge anymore. You're sculpting behavior.

So, three things to walk away with. One: nearly all of a model's actual knowledge comes from that first massive pre-training stage — the later stages don't make it smarter. Two: instruction fine-tuning and RLHF are about alignment, about turning raw capability into something obedient, useful, and safe — and that's a separate craft entirely. And three: the real lesson of ChatGPT is that being better aligned beat being bigger. Capability and alignment are forged apart — and it's the second one that turned a knowledgeable text predictor into something you'd actually want to talk to.