Unrolling the Codex Agent Loop: What Actually Happens Behind a Single Conversation Turn

Most people who use the Codex CLI experience it as something almost magical: you type a sentence, and code appears. But in the gap between your sentence and that code, something far busier is happening. The agent has quietly run dozens of inferences and tool calls. OpenAI's Michael Bolin wrote a piece called "Unrolling the Codex agent loop" that pries open this black box, and it's genuinely worth understanding — whether you use Codex or want to build something like it yourself.

So let's start with the heart of it: the agent loop. Every AI agent, underneath all the polish, is running a loop. It receives your input. It assembles a prompt — and that prompt isn't just your words, it bundles in the system instructions, the definitions of every tool the agent can use, and the conversation history so far. Then it sends all of that to the model for inference. The model returns something, and that something is one of two things. Either it's a tool call — meaning the model wants to run a shell command or edit a file — or it's a final reply. If it's a tool call, the agent executes the tool locally, takes the result, appends it back onto the prompt, and loops right back to ask the model again. If it's a final reply, the loop stops and waits for your next message.

Now here's one of Bolin's sharpest points, and it reframes how you should think about these agents. The agent's real output is not the message it writes back to you. For a coding agent, the valuable output is the code — the files it created or changed on your machine. That chatty assistant message at the end? That's just a signal. It's the agent saying "okay, this round is done." The work lives on disk, not in the text.

And that's why a single conversation turn — one sentence from you — can hide dozens of these inference-and-tool cycles. You see the final result. The loop saw everything in between.

Let's go a layer deeper into that prompt, because Codex structures it in three parts. First, the instructions — these come from your config file or a model-specific built-in config, and they're the static, system-level directives that set the ground rules. Second, the tools — and this is a rich set: Codex's own built-in tools like shell execution and file operations, plus the Responses API's native tools, plus anything you've mounted yourself through MCP. Each tool is described as a JSON schema, which is how the model knows when a tool exists and how to call it. And third, the input — the actual conversation history across all the roles: system, developer, user, assistant, along with the current sandbox permissions and project context.

Here's a detail that matters more than you'd guess. When the API server receives this payload, it reorders everything into a final structure: system message first, then tool definitions, then instructions, then the input messages. That ordering isn't cosmetic — it directly affects how well the prompt cache works, and we'll come back to why.

Now, how does the model actually turn that prompt into output? Think of inference as translation. Your text prompt becomes input tokens. The model samples, producing output tokens one at a time. Those tokens get turned back into text. And because they're generated one at a time, they can be streamed straight back to you — that's the typewriter effect you see in basically every LLM app. But there's a hard constraint hiding here: the context window. Every model has a fixed ceiling on the total number of tokens — input plus output combined. And that ceiling shapes everything about how you design an agent loop.

Which brings us to the problem that defines long conversations. As you keep talking, the tokens pile up, and you start creeping toward that context window cap. Codex handles this with something called auto compaction. The trigger is a setting — once your token usage crosses a defined threshold, compaction fires automatically. It calls a dedicated compaction endpoint, which returns summary items that can directly replace the bloated history. And the clever part: the result is stored as an encrypted compaction item. The model's understanding — its sense of what's been going on — is preserved, but the original raw messages stop eating your context.

This design has a nice secondary benefit. It supports Zero Data Retention customers — companies that, for compliance reasons, don't want their conversations stored on the server. Because Codex makes its requests statelessly, the server never needs to hold onto your original conversation. The encrypted compaction item carries the model's reasoning forward on its own. Compliance and continuity, at the same time.

Now let's talk about cost, because this is where a quiet optimization saves enormous amounts of compute. Remember: every single inference has to send the entire prompt. So as the conversation grows, the cost of each step grows too — and naively, that's quadratic growth. Painful. The fix is the prompt cache. If the server can recognize and reuse the parts of the prompt it already computed last time — a cache hit — you collapse that quadratic growth down to linear.

So how does Codex earn those cache hits? Two main moves. First, put the static stuff at the front. Instructions, tool definitions, sandbox settings — the things that don't change — go at the very start of the prompt, so they can be cached once and reused. This is exactly why that reordering we mentioned earlier matters. Second, and this is the subtle one: MCP tool ordering. If an MCP server lists its tools in a nondeterministic order — different every time — then the prompt looks different every time, and every single request becomes a cache miss. So Codex enforces a consistent ordering of tools. Easy to overlook, huge in impact. The payoff, in Bolin's words: even though the request payload grows quadratically, model sampling can still stay in linear time.

Step back and you can see the whole design is a set of balanced tradeoffs. Autonomy versus safety — the IDE integration lets you dial anywhere from cautious question-and-answer mode all the way to fully autonomous execution. Stateless versus continuity — stateless requests keep you compliant, while encrypted compaction keeps the reasoning thread alive. And context depth versus cost — auto compaction makes long conversations possible at all, and the prompt cache makes them affordable.

So let me leave you with the three things worth holding onto. One: an agent's real output isn't the message it sends you — for a coding agent, it's the code on your disk, and that assistant message is just a "done" signal. Two: the context window is the master constraint, and auto compaction is how you survive it — summarize and encrypt the history so the model keeps its understanding without drowning in tokens. And three: the prompt cache is what makes any of this economically sane, and something as humble as keeping your tool ordering consistent is the difference between linear and quadratic cost.

And maybe the deepest takeaway is the one Bolin points at directly: if you want to build your own coding agent, the lesson isn't how Codex is implemented — it's why these decisions were made. Context management isn't a nice-to-have feature you bolt on later. It's the precondition for letting an agent run for more than a few minutes at a time.