Table of Contents
Sequence models face a fundamental tension on long-context tasks: Transformers are expressive but scale quadratically with sequence length; State Space Models (SSMs like Mamba) are efficient but store all context in a fixed-size hidden state that limits memory capacity. Google DeepMind’s paper “Titans: Learning to Memorize at Test Time” attacks this tension head-on, proposing a neural memory module that keeps learning — via gradient descent — while running inference.
TL;DR
Titans introduces a Neural Long-Term Memory (LTM) module whose parameters are updated at test time using a surprise-driven gradient signal. A forgetting mechanism prevents memory overflow. When integrated into a Transformer backbone (three variants: MAC, MAG, MAL), Titans outperforms both Transformers and Mamba on long-context benchmarks while maintaining near-linear complexity.
Design Philosophy: Memory Shouldn’t Be Frozen After Training
Traditional deep learning encodes all “knowledge” into fixed weights during training — inference is stateless. That works fine for short contexts, but breaks down when tasks require recalling information from tens of thousands of tokens ago (long documents, multi-turn dialogues, cross-chapter reasoning).
Titans’ core bet is: memory should update continuously at inference time, just as humans integrate new information into working memory while reading, rather than relying solely on pre-trained knowledge.
Inspired by Hopfield Networks and Modern Hopfield Networks, Titans models long-term memory as a small MLP whose parameters are the memory medium. Writing to memory means updating those parameters via gradient descent; reading means running a forward pass.
Core Concepts
Neural Long-Term Memory Module
The memory module $M$ is a small MLP with parameters $\theta$. For each token $x_t$ in the input sequence:
Writing (memory update): Compute the prediction error for $x_t$ and update $\theta$ via gradient descent: $$\theta_t = \theta_{t-1} - \eta \cdot \nabla_\theta \mathcal{L}(M_{\theta_{t-1}}(k_t), v_t)$$ where $k_t, v_t$ are key and value projections of $x_t$.
Reading (memory retrieval): Run a forward pass with query $q_t$: $$\hat{v}t = M{\theta_t}(q_t)$$
Surprise: Deciding What to Remember
Not every token deserves equal memorization. Titans uses surprise — the gradient norm of the prediction error — as the write strength signal. The more unexpected a token is to the current memory state, the larger the update:
$$s_t = |\nabla_\theta \mathcal{L}|$$
This focuses memorization on novel, rare, or anomalous information and ignores predictable or repetitive content — an intuitive fit with how human memory prioritizes surprising events.
Forgetting Mechanism
Unbounded accumulation causes interference. Titans adds exponential decay at each step:
$$\theta_t = (1 - \alpha) \cdot \theta_{t-1} - \eta \cdot \nabla_\theta \mathcal{L}$$
The forgetting rate $\alpha$ lets the model prioritize recent information while gradually releasing stale memories — preventing any single past token from permanently corrupting the memory state.
Momentum
Analogous to SGD with momentum, Titans adds a momentum term to smooth memory updates and prevent large oscillations from individual outlier tokens.
Three Integration Architectures
The paper proposes three ways to integrate the LTM module with a Transformer backbone:
graph TD
A[Input Sequence] --> B[Short-Term Memory\nSliding Window Attention]
A --> C[Long-Term Memory\nNeural LTM]
A --> D[Persistent Memory\nLearnable Params]
B --> E{Integration Mode}
C --> E
D --> E
E -->|MAC| F[Memory as Context]
E -->|MAG| G[Memory as Gate]
E -->|MAL| H[Memory as Layer]| Architecture | Integration | Characteristic |
|---|---|---|
| MAC (Memory as Context) | LTM outputs concatenated with input tokens before attention | Most intuitive; memory appears as extra tokens |
| MAG (Memory as Gate) | LTM output gates the attention output | More flexible; memory controls how much attention output flows through |
| MAL (Memory as Layer) | LTM interleaved as independent layers with attention | Modular; easiest to scale and swap |
MAG performs best across most benchmarks, while MAC shows more stable performance on tasks requiring precise memory localization.
Comparison with Alternatives
| Approach | Context Length | Memory Capacity | Updates at Inference | Complexity |
|---|---|---|---|---|
| Transformer | Limited (quadratic) | Unlimited (window-bound) | No | $O(n^2)$ |
| Mamba (SSM) | Theoretically unlimited | Fixed hidden state | No | $O(n)$ |
| RAG | Extended via retrieval | External database | No | $O(n)$ + retrieval |
| Titans (MAC/MAG/MAL) | Theoretically unlimited | Dynamically updated MLP | Yes | $O(n)$ |
The core differentiator is learning during inference — none of the alternatives offer this.
When to Use (and When Not To)
Good fit:
- Long document understanding (books, legal filings, technical specs)
- Long-horizon chat models that must recall early conversation turns
- Cross-chapter QA and multi-hop reasoning
- Any task where “what was said 50,000 tokens ago” matters
Poor fit:
- Short-context tasks where the memory overhead outweighs the benefit
- Edge inference requiring minimal latency (backprop per token adds cost)
- Deployment environments that prohibit parameter updates at runtime (certain compliance requirements)
Key Experimental Results
- SCROLLS / LongBench: Titans-MAG exceeds GPT-4 Turbo (128k context) on multiple subtasks
- Needle-in-a-Haystack: Titans locates a specific fact in 100k+ token documents at significantly higher success rates than Mamba
- Associative Recall: Near-perfect accuracy; SSMs degrade sharply as sequence length grows
Observations and Tradeoffs
Titans is an elegant idea — it turns “learning at inference time” from a research curiosity into an architectural primitive. But a few practical concerns are worth watching:
- Inference cost: Each token requires one backward pass through the LTM module. In production, the latency impact depends on LTM size and hardware, but it’s non-trivial.
- LTM sizing: The MLP’s width and depth cap memory capacity. Different task types need different sizes — another hyperparameter to tune.
- Forgetting rate sensitivity: $\alpha$ is task-sensitive and there’s no adaptive schedule yet. Getting it wrong causes either catastrophic forgetting or memory stagnation.
- Training stability: The interaction between the backbone and the updating LTM makes training curves noisier than pure Transformers.
Overall, Titans points to a compelling direction: models that retain learning capacity at inference time. This aligns with the broader Test-Time Compute trend (e.g., o1-style reasoning chains), but Titans targets memory rather than reasoning depth. The two ideas are likely complementary — a model that both reasons longer and remembers more is the natural next step.
References
Answers come from this article only. Click any prompt below or open the chat at the bottom right.
🇺🇸 English
Here's the tension at the heart of every long-context AI model. Transformers are brilliant — expressive, powerful — but their cost explodes as your input gets longer. Double the sequence, and you roughly quadruple the work. On the other side, you've got State Space Models like Mamba, which are wonderfully efficient and scale almost linearly. But they pay for that efficiency by cramming everything they've ever seen into a fixed-size memory slot. Eventually, that slot just runs out of room.
Google DeepMind's paper, "Titans: Learning to Memorize at Test Time," walks straight into that tension with a genuinely bold idea. What if a model could keep learning — actually updating itself — while it's running inference? Not during training. Right there, in the moment, as it reads.
Let me give you the one-sentence version first. Titans introduces what they call a Neural Long-Term Memory module. Its internal parameters get rewritten on the fly, at test time, driven by a signal they call "surprise." There's a forgetting mechanism so the memory doesn't overflow. Bolt this onto a Transformer, and it beats both plain Transformers and Mamba on long-context tasks — while staying near-linear in cost. That's the whole promise.
So let's talk about the philosophy, because this is where it gets interesting. In traditional deep learning, all the knowledge gets baked into frozen weights during training. Once you deploy the model, inference is stateless — it doesn't learn anything new from what you show it. That's totally fine for short tasks. But it falls apart when the model needs to recall something from tens of thousands of tokens ago. Think long legal documents, hour-long conversations, reasoning that spans multiple chapters of a book.
Titans makes a very human bet. Memory shouldn't be frozen after training — it should keep updating as you go. When you read a novel, you don't lean only on what you knew before you opened the book. You build up working memory, chapter by chapter. Titans does the same. And drawing inspiration from Hopfield Networks, it models this long-term memory as a small neural network — a little MLP — where the parameters themselves *are* the memory. Writing to memory means nudging those parameters. Reading from memory means running data through them. Storage and computation, fused into one.
Now, how does the memory actually work? Two operations. Writing and reading.
When a new token comes in, the module makes a prediction about it, measures how wrong it was, and then adjusts its own parameters a tiny bit to reduce that error — using gradient descent, the same mechanism that trains neural networks in the first place. Except here it's happening live, during inference. That's the write. Reading is simpler: you hand the memory a query and run a forward pass to pull out what it remembers.
But here's the clever part — the concept of surprise. Not every token deserves the same attention. Some are boring and predictable; some are genuinely new. Titans uses the *size* of that prediction error — how badly the memory was caught off guard — as the strength of the update. The more surprising a token, the harder it gets written in. Predictable, repetitive stuff barely moves the needle. And honestly, that mirrors human memory beautifully. You remember the shocking moment, the plot twist, the thing you didn't see coming. The mundane fades.
Which brings us to forgetting. If you just kept accumulating everything forever, old memories would start interfering with new ones — the whole thing would turn to mush. So Titans adds a gentle exponential decay at every step. There's a forgetting rate that slowly releases stale information, keeping recent stuff prioritized. It means no single token from way back in the past can permanently poison the memory. And there's one more touch borrowed from optimization: momentum, which smooths out the updates so a single weird outlier token doesn't yank the memory around violently.
Okay — so you've got this memory module. How do you actually plug it into a Transformer? The paper offers three flavors, and they all draw on three types of memory working together: a short-term memory from a sliding attention window, the long-term neural memory we just described, and a set of persistent learnable parameters. The difference is *how* the long-term memory's output gets combined with everything else.
The first is MAC — Memory as Context. Here, the memory's output gets stitched onto the input as extra tokens before attention runs. It's the most intuitive: memory literally shows up as additional context to attend to. The second is MAG — Memory as Gate. Instead of adding tokens, the memory acts like a valve, controlling how much of the attention output is allowed through. More flexible. The third is MAL — Memory as Layer. Here the memory becomes its own independent layer, interleaved with the attention layers. It's the most modular — easy to stack, easy to swap.
And the results have personality of their own. MAG tends to win on most benchmarks overall. But MAC is steadier when the task demands precise memory localization — pinpointing exactly where a fact lives.
Let me put Titans next to the alternatives so the leap is clear. A Transformer has effectively unlimited memory within its window, but that window is bounded and the cost is quadratic. Mamba is linear and theoretically unlimited in length, but its memory is a fixed-size box — no updates at inference. RAG extends things by fetching from an external database, but again, no learning during inference; it's just retrieval. Titans is linear in cost, theoretically unlimited in length, and — here's the one thing none of the others do — it actually updates its memory *during* inference. That's the differentiator. Learning while it runs.
When should you reach for this? It shines on long documents — books, legal filings, dense technical specs. It's great for long chat models that need to remember what you said at the very start of a marathon conversation. Cross-chapter question answering, multi-hop reasoning, anything where "what was said fifty thousand tokens ago" genuinely matters. Where it's a poor fit: short-context tasks, where the memory machinery is just overhead. Latency-sensitive edge devices, because doing a backward pass on every single token isn't free. And any environment that flat-out forbids updating parameters at runtime — certain compliance situations.
The experimental numbers back the story up. On long-context benchmarks, Titans in its MAG configuration beats GPT-4 Turbo with its 128k context on several subtasks. On the classic needle-in-a-haystack test — hiding one fact inside a hundred-thousand-token document and asking the model to find it — Titans locates it far more reliably than Mamba. And on associative recall, it's nearly perfect, while State Space Models fall apart as sequences get longer.
Now, I want to be honest about the tradeoffs, because this isn't magic. Every token costs an extra backward pass through the memory module, so latency in production is a real concern — it depends on how big your memory network is and what hardware you're on, but it's not negligible. The size of that memory MLP caps how much it can hold, and different tasks want different sizes — yet another knob to tune. The forgetting rate is finicky and task-sensitive, with no adaptive schedule yet; set it wrong and you either forget everything or freeze up and remember nothing new. And training is noisier, because the backbone and the constantly-updating memory interact in messy ways.
But step back, and the direction is genuinely exciting. Titans takes "learning at inference time" and turns it from a curiosity into an architectural building block. It rhymes with the broader test-time compute trend — the same instinct behind o1-style reasoning chains — except those models spend extra compute thinking *harder*, while Titans spends it remembering *more*. And those two ideas don't compete. They compose.
So let me leave you with three things to hold onto. First, the core move: memory as living parameters that rewrite themselves during inference, not frozen weights — driven by surprise, tempered by forgetting. Second, it breaks the long-context ceiling while staying near-linear, and it does it by learning on the fly, which is exactly what nothing else on the field does. And third, this is only half of a bigger picture. A model that reasons longer *and* remembers more — that's the natural next step. Titans just showed us what the "remembers more" half can look like.
🇹🇼 中文
這篇整理，來自對 Google Research 一篇論文的影片分析，論文叫做《Titans：Learning to Memorize at Test Time》。這篇被當成他們在 NeurIPS 發表的一部分推出，還配了專門的部落格文章、社群討論也蠻熱鬧的。有趣的是，做分析的人一開頭就很誠實地說：他自己也是被行銷吸引才點進來看的。而他看完之後的結論是——這是一篇好論文，但大概「一半是真的很酷的新東西，另一半是把腳踩在行銷油門上」。
我們就順著這個基調來看：它想解決什麼、核心想法是什麼、還有哪些其實根本不是新東西。
先講它想解決的問題，也就是 context window 的天花板。
現在的模型大致可以分兩類。一類是天生的序列模型，像是 RNN、LSTM；但這一類，基本上已經被 attention-based 的模型，也就是 transformer，給超越了。
transformer 的麻煩在哪？它只能注意到目前 context window 裡面的東西。對某些任務來說，這個視窗需要非常非常大——分析者舉的例子是影片理解，或者那種現實世界裡會同時發生一大堆事情、你得把全部納入考量才能決定下一步的超長任務。
問題其實很單純：你手上有一段很長很長的資料，可是模型的容量，只夠看其中一段，也就是 context window 那麼長的一段。這個視窗你要擺哪都可以，前面、中間、後面都行，它就像一個框，可以在整段長資料上滑動、對準任何一塊——但重點是，你沒辦法一次把所有東西都塞進同一個 context window，硬塞就會把模型撐爆。
那 Titans 的核心想法，就是「測試時記憶」。
它提出的架構，重點是讓一個模型——比如語言模型——在測試時，也就是推論的當下，學會記憶，藉此走出目前 context window 的範圍。
直覺上是這樣運作的：把一段非常長的文本切成好幾個部分，讓模型一段一段地跑過去；在跑的過程中，用一塊「記憶」去記住、去連結前面段落學到的東西，再把它帶到下一段。這樣一來，就算後面的段落已經看不到最前面的原始內容了，模型還是能靠記憶把它接起來——這就繞過了 transformer 那種「只能看到目前視窗」的限制。
這也正是分析者覺得很酷的地方：把「跨越很長的文本、記住早先的資訊」這件事，直接變成架構本身的能力。
但接下來，他也毫不客氣地點出：論文裡很多被叫做「memory」的東西，其實早就存在了。他的批評分兩種——有時候是把舊東西重新包裝，講得像是新發明；有時候是給既有的機制取一個新名字，讓它看起來像新東西。
他舉的歷史脈絡，是「跨段記憶」這條老路。早在 BERT 之後那一波，就有很多人在嘗試把模型推向很長的 context，其中一些變體，會明確用到我們今天會稱為 memory 的做法。
它大概是這樣運作的。第一步，先處理一個段落，在段落結尾產出一個「產物」——通常就是最後一個 token 的某種運算結果，或者說 hidden state。為什麼挑最後一個 token？因為序列裡的最後一個 token，天生就會注意到整段內容，所以它某種程度上，已經把整段的資訊整合起來了。第二步，把這個 hidden state 傳給下一個 context window。下一段在生成 token 的時候，雖然沒辦法直接回頭去看上一段的原始內容，但它可以注意到這個被傳過來的產物——而這個產物，理論上就是上一整段內容的一個壓縮版本。第三步，因為每一段都會從再前面一段收到這樣一個壓縮產物，資訊就會一路往後帶。
用一句話總結這種老做法的性格：在段落跟段落之間，它像 RNN——靠一個往後傳的狀態把各段串起來；但在單一個 context window 之內，它又像 transformer。分析者說這類想法其實不少，名字他自己也記不太清楚，可能叫 Transformer-XL 之類的，他也不太確定確切名稱。
除了這條跨段記憶的路線，論文裡還討論了另一條脈絡，叫 linear transformers——最基本的想法，就是把原本 softmax 那一套 attention 換掉。不過這部分更細的內容，不在這次整理的素材範圍裡，就先略過。
那來收個尾。
Titans 真正吸引人的地方，是把「在測試時保有記憶、跨越很長文本」這件事，直接做進架構裡，讓模型能處理遠遠超過單一 context window 的內容——這個方向本身，非常有價值。
但如果照分析者的判斷，看這篇論文的時候，值得保持一點清醒：它一半是新意，一半是行銷。那些被冠上「memory」之名的機制，有不少可以一路追溯到 BERT 之後那一波處理長 context 的跨段狀態傳遞——概念不見得是全新的，只是被重新命名、重新包裝了一次。
最後幫你把三個核心要點收攏一下。第一，它要解決的是 transformer 的 context window 天花板，資料太長、一次塞不進同一個視窗。第二，Titans 的核心，是讓模型在推論當下學會記憶，把長文本切段、靠記憶跨段接力。第三，別被「memory」這個字眼迷惑——很多做法其實是 BERT 之後跨段狀態傳遞的舊招，換了個新名字。真正的新意跟行銷包裝，記得分開看。
Tags
Related Articles
Harness Engineering: The Model Isn't Dumb, It Just Lacks Human Guidance
When an AI Agent performs poorly, it's not necessarily because the model is dumb. Starting from a small experiment where a Gemma 4 2B fixes a bug, this piece explains what a Harness is, how Harness Engineering differs from Prompt / Context Engineering, and how effective natural-language rules like agents.md really are.
J-lens: Anthropic's New Interpretability Tool for Reading Claude's Inner Thoughts via a 'Global Workspace'
Anthropic proposes J-lens, an interpretability tool that captures the 'verbalizable' representations inside a Transformer, and uses it to show that Claude contains a privileged subspace analogous to the neuroscientific 'global workspace' — a small set of vectors that broadcast, drive reasoning, respond to external steering, and even leak signals during deception and evaluation awareness.
TiDAR: Think in Diffusion, Talk in Autoregression (Paper Analysis)
TiDAR runs a diffusion model to draft tokens in parallel (Think), then lets an autoregressive decoder finalize output (Talk) — all in a single forward pass. Result: 5.91x faster than AR at comparable quality.