Table of Contents

“If AI can improve itself, won’t it just keep getting better until it’s uncontrollable?” This question has circulated in AI circles for decades, from early AIXI theory to recent Constitutional AI debates. The framing is misleading, though, because some form of AI self-improvement is already running in production right now. The more productive question is: how far is current AI self-improvement from a recursive runaway loop, where are the real bottlenecks, and how should engineers think about it?

TL;DR

AI self-improvement spans a wide capability spectrum — from “use AI outputs to train the next model version” to “AI autonomously rewrites its own training infrastructure and deploys a stronger successor.” The former is well-established (Constitutional AI, RLHF with AI feedback, automated evaluation); the latter remains strictly limited by evaluator reliability and unsolved alignment problems. Understanding this spectrum is more useful than debating whether AI will “cross the Rubicon.”

What Is It

Recursive Self-Improvement (RSI) at its core means: a system modifies itself such that the modified version is better at some objective, and the process can repeat. But this definition covers technically very different things:

Level 1: AI-assisted training data generation The most mature and widely deployed form. Anthropic’s Constitutional AI has Claude score and revise its own outputs against a set of principles, then uses high-scoring outputs as preference data for reinforcement learning. OpenAI’s RLHF pipeline includes similar AI feedback stages. This runs at production scale — but “self-improvement” is limited: the AI improves the next version, not itself in real time.

Level 2: AI-driven architecture and hyperparameter search Neural Architecture Search (NAS) and AutoML let AI automatically find better model configurations. Effective, but the search space is still defined by human engineers. AI finds the optimum within human-defined bounds.

Level 3: AI autonomously writing and running code to improve itself The fastest-moving frontier. Systems like Devin, SWE-agent, and OpenAI o3 demonstrate AI autonomously fixing bugs, writing tests, and optimizing algorithms. Currently these systems improve tools and code, not core model weights.

Level 4: Full recursive self-improvement loop AI modifies its own training pipeline and architecture, trains a stronger successor, which repeats the process. Theoretically the most powerful form; currently the most constrained.

Why It Matters

RSI matters because it is directly tied to the shape of the AI capability curve. Current AI progress depends primarily on external inputs: more compute (scaling laws), more high-quality data, and human researcher architecture innovations. If AI can reliably substitute for any of these, the pace of improvement could accelerate significantly. Three specific leverage points:

  1. Evaluation automation: If AI can reliably judge “did this change make the model better?”, human engineers become less critical in the training loop.
  2. Code automation: If AI can autonomously write and validate training code, ML research iteration speed could increase substantially.
  3. Knowledge distillation: Strong models generating training data for weaker models, propagating capabilities downward at lower cost.

All three are happening today, but reliability and autonomy remain well short of a fully closed loop.

How It Works

A typical semi-automated AI-assisted training improvement loop:

graph LR
    A[Current Model v1] --> B[Generate Candidate Outputs]
    B --> C[AI Evaluator Scoring]
    C --> D[Human Review Sampling]
    D --> E[Preference Dataset]
    E --> F[RL Fine-tuning]
    F --> G[New Model v2]
    G -->|Capability evaluation| A

The “human review sampling” step in the middle cannot currently be removed, for two fundamental reasons:

The evaluator bottleneck: Asking AI to evaluate its own outputs is equivalent to asking whether AI can reliably recognize outputs better than itself. Within well-understood capability domains this works (e.g., Constitutional AI principle adherence scoring). Near the capability frontier, AI evaluator reliability degrades quickly. This is why Scalable Oversight is a central problem in AI safety research.

Reward hacking: Any gap in an evaluation function will be found and exploited by optimization. The model appears to improve by the metric while violating the underlying intent. RL history is full of documented cases. Closing this gap requires either perfect evaluation functions (unsolved) or sufficiently robust human oversight.

Alternatives Compared

MechanismHuman InvolvementSpeedReliabilityStatus
Human RLHFHighSlowHighProduction standard
Constitutional AI / AI feedbackMediumMediumMedium-highProduction
NAS / AutoMLLowFast (bounded)High (in-scope)Widely used
AI-assisted code writingMediumFastMediumRapidly advancing
Full RSI loopVery lowTheoretically explosiveUnknownResearch stage

The “Rubicon” metaphor implies an irreversible threshold. In the RSI context, this is typically defined as: an AI system that can reliably generate successors stronger than itself, without requiring external human intervention. Current technology falls short of this threshold on several dimensions: evaluator reliability degrades at capability boundaries, complete training pipelines still require substantial human engineering maintenance, and alignment remains unsolved — a more capable AI is not automatically a more aligned one.

Conclusion

AI recursive self-improvement isn’t science fiction, nor is it an imminent threat. It’s a spectrum, with large differences in technical maturity across levels. Engineers may already be working with some form of AI self-improvement (Constitutional AI-trained models, NAS-optimized architectures) without framing it that way.

The metrics worth tracking are Scalable Oversight research progress and the degree of autonomy in AI-assisted ML research tooling. The intersection of those two vectors gives a far more accurate picture of where the technical frontier actually is than any Rubicon metaphor.

References

Ask this article

Answers come from this article only. Click any prompt below or open the chat at the bottom right.

🇺🇸 English

Here's a question that's been floating around AI circles for literally decades: if an AI can improve itself, won't it just keep getting better and better until, one day, it's completely out of our control? You hear it framed as this looming Rubicon we're about to cross.

But here's the thing — that framing is kind of misleading. Because some form of AI self-improvement? It's not coming someday. It's already running, in production, right now. So the more useful question isn't "will AI cross the line." It's: how far is what we have today from an actual runaway loop, where are the real bottlenecks, and how should an engineer actually think about this?

Let me start with the punchline. AI self-improvement isn't one thing — it's a whole spectrum. On one end, you've got "use the AI's outputs to help train the next version of the model." That's mature, that's everywhere. On the far end, you've got "the AI rewrites its own training infrastructure and deploys a stronger successor, all on its own." That end is still heavily locked down — mostly by two problems: can you trust the AI to evaluate itself, and the whole unsolved mess of alignment. Understanding that spectrum is way more valuable than arguing about a magic threshold.

So let's walk up the ladder, because the levels are genuinely different beasts.

Level one is AI-assisted training data generation. This is the mature, boring-in-a-good-way version. Take Anthropic's Constitutional AI: they have Claude score and revise its own outputs against a set of written principles, and then the high-scoring outputs become preference data for reinforcement learning. OpenAI's RLHF pipeline does something similar with AI feedback. This is at massive scale today. But notice the key limit — the AI is improving the *next* version. Not itself, in real time.

Level two is AI-driven architecture search. Things like Neural Architecture Search and AutoML, where the AI hunts for better model configurations. It works. But the search space is still drawn up by human engineers. The AI finds the best answer inside a box we built for it.

Level three — this is the fast-moving frontier — is AI autonomously writing and running code to improve itself. Think Devin, SWE-agent, OpenAI's o3. These systems fix bugs, write tests, optimize algorithms on their own. But crucially, they're improving *tools and code*. Not the core model weights.

And level four is the full loop. The AI modifies its own training pipeline, trains a stronger successor, and that successor repeats the whole process. Most powerful in theory. Most constrained in practice. Basically research-stage.

Now, why does any of this actually matter? Because it's tied directly to the shape of the AI capability curve. Right now, progress mostly comes from outside inputs — more compute, more good data, and smart humans inventing new architectures. If AI can reliably stand in for any of those, the pace could really accelerate. There are three specific leverage points here.

First, evaluation. If AI can reliably judge whether a change made the model better, human engineers get squeezed out of that part of the loop. Second, code. If AI can write and validate training code on its own, research iteration speeds way up. And third, distillation — strong models generating training data for weaker ones, pushing capability downward cheaply. And here's the honest part: all three are happening today. They're just not reliable enough or autonomous enough to close the loop.

So let me describe how the real loop actually works today, because there's one step in the middle that everything hinges on. You start with your current model. It generates a bunch of candidate outputs. An AI evaluator scores them. Then — and this is the load-bearing step — humans sample and review a slice of those scores. That feeds into a preference dataset, you do reinforcement fine-tuning, and out comes a new model. Then you evaluate its capabilities and go around again.

That human review step? Right now, you cannot delete it. And there are two deep reasons why.

The first is the evaluator bottleneck. Asking an AI to evaluate its own outputs is basically asking whether it can reliably recognize output that's *better than itself*. Inside domains it understands well — like scoring how well it followed a set of principles — that's fine. But push toward the frontier of its capability, and the reliability of that self-evaluation falls apart fast. This is exactly why "scalable oversight" is such a central problem in AI safety.

The second reason is reward hacking. And this one's almost a law of nature. Any gap, any loophole in your evaluation function will get found and exploited by the optimization process. The model's score goes up beautifully — while it quietly violates the actual intent behind the metric. The history of reinforcement learning is absolutely littered with these cases. And to close that gap you'd need either a perfect evaluation function, which nobody has, or robust enough human oversight to catch it.

So if you line up the mechanisms by how much human involvement they need — human RLHF is slow, high-reliability, and it's the production standard. Constitutional AI sits in the middle on both speed and human involvement, and it's in production. NAS and AutoML need very little human input and are fast and reliable, but only inside their defined scope. AI-assisted code writing is fast and advancing quickly, but only medium reliability. And the full recursive loop? Very little human involvement, theoretically explosive speed, and reliability that is frankly just... unknown. Research stage.

That's what the Rubicon metaphor is really pointing at — an AI that can *reliably* produce successors stronger than itself with no human in the loop. And today's tech falls short on several fronts at once. Evaluator reliability crumbles right at the capability boundary where you'd need it most. Full training pipelines still need serious human engineering to keep running. And alignment is unsolved — a more capable AI is not automatically a more aligned one. That last point is easy to forget and really important.

So let me leave you with the three things that actually matter here.

One: recursive self-improvement isn't science fiction, and it isn't an imminent threat. It's a spectrum, and the maturity gap between the low levels and the high levels is enormous. You might already be using AI self-improvement — a Constitutional-AI-trained model, a NAS-optimized architecture — without ever calling it that.

Two: the whole thing is gated by two hard problems — trustworthy evaluation and reward hacking. Until those are solved, the human stays in the loop. Not by choice, by necessity.

And three: if you actually want to track where the frontier is, stop watching for the Rubicon moment. Watch two real vectors instead — progress in scalable oversight research, and how autonomous AI-assisted ML tooling is getting. Where those two lines cross tells you far more than any dramatic metaphor ever will.

🇹🇼 中文

一九六五年，統計學家 John Good 提出過一個想法，他叫它「人類最後的發明」。意思是：假如有一天我們造出一個 AI，它厲害到能再造出比自己更強的 AI，那接下來的事人類就不用管了，AI 自己會接手，技術從此爆炸性成長。這台「能造出更強 AI 的 AI」，就是人類需要親手做的最後一件事。

這個老題目最近又被翻出來，是因為 Anthropic 一位共同創辦人寫了一篇文章。他說自己看了一堆資料後，很不情願地承認：他認為到二〇二八年底，有六成的機率，AI 的研發不再需要人類——AI 會自己開發出更厲害的 AI。如果真的發生，我們就「跨越了盧比孔河」。

盧比孔河是義大利北部一條小河。古羅馬規定在外的將領不能帶兵渡過它，渡過就等於掀起內戰。凱撒就是帶兵渡了河，走上一條回不了頭的路。所以「跨越盧比孔河」的意思是：有些事你一旦做了，就收不回來。放到 AI 上，就是問：現在的 AI 是不是已經強到能造出更強的 AI，一旦跨過去，它就不再需要我們了。

不過今天我不談科幻推測，我想回到工程現場問一個更實際的問題：今天 AI 自主成長的能力，到底能做到什麼地步？

先講清楚一件事——AI 的「自我成長」其實沒有明確定義。你能找到滿坑滿谷的論文，宣稱自己提出了讓 AI 自我改進的新方法，ICLR 二〇二六甚至有個專門的 workshop 在談這個。但你仔細看會發現，所謂「AI 自我成長」，本質上是「人類逐漸放手」的過程。很多研究過程裡還是有人介入，只是介入得比以前少一點，就宣稱自我成長了。所以接下來講的每一種做法，其實都還需要人類——真正的差別在於：人類能少到什麼地步。

我們回到機器學習的本質。機器學習可以拆成三步：第一，我要找什麼樣的函式；第二，我有哪些候選函式；第三，從候選裡挑最好的。第三步基本上是自動的，就是梯度下降。有意思的是前兩步都有一個「我」，過去這個「我」指的是人類。自我成長的討論，其實就是在問：這個「我」，有多少成分可以換成 AI 自己？

我們聚焦第一步。我們要找一個函式，輸入 x、輸出 y、帶一組參數。然後定一個 loss，目標是找到讓 loss 越小越好的參數。在一般的監督式學習裡，這個 loss 怎麼算？你先準備一堆輸入，丟進函式得到一堆輸出，每個輸出根據好壞算一個小 loss，加起來就是要最小化的大 L。而每個小 loss 怎麼來？你得先有一個標準答案，也就是 ground truth，拿模型的輸出跟標準答案比距離：差越大 loss 越大，越接近 loss 越小。

問題就在這裡：這些標準答案是人標出來的。監督式學習需要大量人工。所以第一個能讓 AI 接手的地方就是——能不能不要人來標答案？

第一種放手，是讓 AI 自己產生標準答案。一個直覺的想法是拿一個更強的 AI 產生答案，去教一個較弱的 AI，這就是大家熟悉的知識蒸餾。但這不是重點，因為它引入了一個「比目標模型更強的老師」。既然你已經有更強的 AI 了，這就不算「AI 造出比自己更強的 AI」。

真正該問的是：這個由 AI 產生、不一定正確的假答案，有沒有可能是模型用自己產生的？拿同一個模型的輸出來教它自己，怎麼可能變更好？

關鍵在自我修正。模型有很多自我修正的手段：透過生成時內部表示的變化，偵測自己可能答錯；或加一句「你再好好想想」；或用大量 token 做長時間推理——這些都可能把錯的答案改對。但要注意，單純的自我修正並不會強化模型本身，因為修正時參數沒變，你再問同一題，它還是會先答錯、再修正一遍。突破點在下一步：把「修正後的答案」當成標準答案，回頭去微調模型參數。這樣新模型看到同樣的輸入，就更傾向直接輸出正確答案，能力真的被留下來了。Anthropic 早期的 Constitutional AI 就是這個思路。

第二種放手，跟強化學習有關。有人會說，很多場景根本不需要標準答案，例如強化學習。強化學習裡沒有 ground truth，只有一個 reward function 幫輸出打分。為了跟監督式學習放進同一個框架，我這邊把 reward 當成「越小越好的 loss」來看，兩者其實一法通萬法通。

所以強化學習也常被算作一種自我成長。但人類的介入並沒有消失，只是躲進了 reward function 的設計裡：得由人把各種情況都考慮過，設計出一個 reward function，才能引導 AI 學習。

強化學習有個痛點：真實世界的 reward 常常很稀疏。以訓練機器手臂開門為例，只有真的把門打開才有分，碰到門板都是零，機器人幾乎學不起來。常用的解法叫 reward shaping：保留「開門才算成功」的真實 reward，但額外加一些代理獎勵來引導，例如靠近門板給一點分、碰到門板給更多分。

而 reward shaping 這件事，可以交給 AI 來做。人類只定真正的 loss——唯有開門才有分——讓另一個 AI 去設計代理獎勵，幫目標模型學得更好。它怎麼知道哪種代理獎勵好用？靠一個回饋迴圈：一個負責寫代理獎勵的 AI，產生第 k 版獎勵，拿去訓練目標模型；訓練完，用「真正的 reward」評測目標模型的行為；再把評測結果當回饋傳回去。如果這版在真正 reward 上表現不好，寫獎勵的 AI 就知道該改；表現好，就把好特性保留進下一版。

這類研究不少，早期有二〇二三年的論文，後來有 Revolve、還有近幾個月的 RF Agent。負責寫代理獎勵的通常是語言模型，被訓練的目標則不一定，常常就是一支機械手臂。有一篇二六年的論文讓機械手臂做傳接球，原本的 reward 只有一行，語言模型寫出的代理獎勵卻涵蓋很多面向——球離手很近也給分、手臂擺成某種姿勢也給分——藉此更有效率地引導學習。

插一句題外話：你身上的多巴胺，其實就是 reward shaping。對基因來說，真正的 reward 只有「成功傳宗接代」，這訊號太稀疏了——原始人得先打獵、取得食物、活下來，才有繁衍機會。於是大腦演化出多巴胺驅動的獎勵系統，讓你每達成一個小目標就開心一下。更精確地說，多巴胺給的是「追逐目標時的慾望」，而不是「達成後的滿足」——這也是為什麼有些東西你渴望時覺得會很快樂，真的得到卻沒那麼快樂。當然，不是基因「為了讓人活下去」才設計這套系統，而是沒演化出這套系統的生物，活不下去。

第三種放手，是當 loss 根本寫不出來的時候。reward shaping 至少還假設人類能把「真正的 loss」寫成函式。可是很多事根本寫不出規則。下圍棋還好，贏加一、輸減一；但你叫語言模型寫一篇文章，這篇該得幾分？你很難寫出這個評分函式。

解法是把另一個 AI 當成 reward function：輸入 x 和某個 y，讓它輸出這個 y 有多好。可是 AI 又怎麼知道好壞？這裡通常還是需要人類——你或許寫不出評分公式，但你看到一篇文章，多半能判斷它好不好、給個分數。於是人類提供分數，訓練一個模型讓它的評分盡量貼近人類，再用這個模型去訓練目標模型。這就是 RLHF，那個負責打分的模型就是 Reward Model。如果連打分都改用一個很強的語言模型來做，就成了 RLAIF。

但別忘了本文的規則：過程中不該引入一個比目標模型更強的 AI。於是真正的問題變成——如果打分的 AI 就是目標模型自己，它能不能用自己產生的 loss 把自己變強？

這還真不是不可能。讓 AI 自己產生 loss 有幾種做法。第一種是直接口頭評分：給它輸入和輸出，直接問「這個打幾分」，它通常會吐個分數，正不正確看它自己的能力。第二種是看 token 機率：問它「這個答案對嗎」，看下一個 token 的機率分布，把「對」的機率乘上負一，就能當 loss。第三種是多數決：對同一個輸入 sample 很多次，取多數決當成假答案，再拿它跟待評答案的距離當 loss。第四種是看信心，也就是 entropy：模型對輸出越有信心、分布越集中，答案越可能對、loss 越低；亂度越大代表越沒信心、loss 越高。

用 entropy 當學習訊號其實很早就被發現有用，而且跨模態。二〇二〇年在影像上就有一個叫 TENT 的方法，指出模型輸出的 entropy 越大、錯誤率越高，兩者高度相關，可以反過來拿它微調模型。本課實驗室的林冠廷同學在二〇二二年提出的 SUTA，也用類似思路，在不需要額外標註的情況下強化語音辨識。反倒是 NLP 引入得比較晚——直到去年才有一篇標題很直白的論文，叫《Entropy Minimization 在 LLM 推理中不合理地有效》，發現光是把最小化 entropy 當學習目標，竟然真的有用。

那到底能走多遠？最近有一篇論文做了相當完整的實驗，標題是《無監督 RLVR 能把 LLM 訓練 scale 到多遠》，這裡的無監督指的是 reward 由 LLM 自己定義、不需要人類。有兩個結果值得記住。第一，在訓練初期，AI 自己定的 reward 跟人類定的 reward 表現差不多，在三個資料集上，兩條線前期幾乎並駕齊驅。但第二，長期來看人類的 reward 還是勝出——正確的 reward 能較長時間把模型引導到更好的結果；而 AI 不斷拿自己定的 loss 訓練自己，最終有可能把自己越訓練越壞。

所以把整條光譜攤開來看，「AI 自我成長」不是一個開關，而是一條人類逐步放手的路。從監督式學習裡人類直接標答案；到自我修正加微調，模型修正自己的輸出，但仍需人設定修正機制；到強化學習裡人定真 reward、AI 定代理獎勵，人躲進 reward function 裡；到 RLHF，人提供偏好分數、模型學會打分；最後到 AI 完全用自己的信心當訊號，人類幾乎退場——但代價是它可能把自己訓壞。每往下一格，人類的介入就少一點，可是到目前為止，那隻手始終沒有完全離開。

最後幫大家收攏三個重點。第一，「AI 能不能創造更強的 AI」不是一個 yes or no 的問題，而是一條人類逐步放手的光譜，真正的技術主軸是：訓練所依賴的 loss 和 reward，能被 AI 接手到什麼程度。第二，越往光譜末端，人類介入越少，但目前每一種做法背後其實都還藏著人——不管是修正機制、reward function，還是偏好分數。第三，就算讓 AI 用自己的訊號自我訓練，短期看起來能追上人類，長期卻可能把自己帶偏。所以真正值得追蹤的，不是哪一天忽然跨過盧比孔河，而是這隻手還能退到多後面——這，才是技術邊界真正所在的位置。

Tags

#self-improving-ai#recursive-self-improvement#reinforcement-learning#alignment#scalable-oversight

Related Articles