AI Ignites the Memory Chip Supercycle: Why HBM Demand Is Different This Time

Since 2023, something strange has been happening in the semiconductor world. The market has essentially split in two. On one side, consumer electronics chips — the stuff in phones and laptops — demand has been soft. On the other side, data center memory has been so supply-constrained that companies literally can't get enough of it. And the reason for that split comes down to one thing: AI. More specifically, a technical property of how large language models actually run that makes them hungry for memory in a way traditional computing never was.

Let me give you the punchline up front. When a large language model generates text, it has to keep two things loaded in ultra-fast memory at all times: the model's weights, and something called the KV cache. That requirement makes a specific kind of memory — HBM, High Bandwidth Memory — the single most important bottleneck in AI accelerators. The memory market grew seventy-eight percent in 2024. HBM supply is already spoken for through the end of 2026. And this upcycle is expected to run at least through 2028.

So first, what is HBM? Picture a stack of DRAM memory chips piled vertically, one on top of another, connected by tiny vertical wires that punch straight through the silicon — those are called through-silicon vias. That whole stack sits right next to the GPU on a shared silicon base. The payoff is dramatically higher bandwidth and capacity than conventional graphics memory. The cost is that it's much harder to manufacture and much more expensive.

Just to make the gap concrete — the current generation, HBM3e, moves more than 1.2 terabytes per second per GPU and gives you anywhere from 80 to nearly 200 gigabytes of capacity. Compare that to GDDR6, the memory you'd find in a gaming card: it tops out around 768 gigabytes per second and maybe 24 to 48 gigabytes of capacity. HBM also runs cooler because the wires are so short, and yes, it costs a lot more. This is why NVIDIA's H100 ships with 80 gigabytes of HBM3, and the H200 pushes that up to 141 gigabytes with HBM3e. Those specs aren't arbitrary — the AI workloads demand them.

Now here's the heart of it: why does AI create such unusual memory requirements? Because LLM inference just doesn't behave like gaming or scientific simulation or database queries.

Reason one: the model weights have to physically fit in memory. A 70-billion-parameter model at half precision needs roughly 140 gigabytes. And those weights have to be reachable by the GPU's compute cores on every single forward pass. You can't stash them in the CPU's slower memory and swap them in — the delay would kill any hope of real-time responses.

Reason two, and this is the big one: inference is memory-bandwidth-bound, not compute-bound. When the model generates text, it produces tokens one at a time, and for each token the attention mechanism keeps reading the KV cache and the weight matrices over and over. So the GPU's compute cores spend most of their time just... waiting. Waiting for data to arrive from memory. Which means throwing more compute cores at the problem doesn't help. Adding more memory bandwidth does. That's a genuinely counterintuitive flip from how we normally think about "faster chips."

Reason three: the KV cache grows with context length. As the model generates, it caches the key-value states for every token in the conversation. A short 4,000-token context might need a few gigabytes. A 128,000-token context? Orders of magnitude bigger. And here's the trap — users keep wanting longer and longer context windows, and every bit of that translates directly into more pressure on HBM.

So if you picture the whole thing: model weights and the KV cache both compete for two resources — capacity, so everything fits, and bandwidth, so the cores don't stall. HBM is what provides both.

Let's talk about the market, because the structure here is fascinating. On the supply side, making HBM requires this complex wafer-bonding and through-silicon-via process, and only three companies on Earth can do it commercially. SK Hynix dominates with about 62 percent of the market — and critically, they supply roughly 90 percent of NVIDIA's HBM. Micron sits around 21 percent. Samsung is at about 17 percent, held back by yield problems they're still working through. And across all three, capacity is basically committed through the end of 2026.

On the demand side, look at how data centers are eating up memory. Five years ago, data centers were about 32 percent of global DRAM demand. In 2025 that's roughly half. By 2030 it's projected to top 60 percent. Prices reflect it — overall DRAM prices rose 30 to 60 percent across 2024 and 2025, and NAND flash roughly doubled. As for how long this lasts, Micron said back in December 2025 that tight conditions would stretch beyond 2026, and analysts see the HBM market growing from about 35 billion dollars in 2025 to around 100 billion by 2028. That's a 40 percent compound annual growth rate.

But here's the question worth pausing on: why is this cycle different from past semiconductor cycles? Remember the 2021 COVID chip shortage? That was a demand shock plus supply-chain chaos — a temporary spike. This one has a fundamentally different shape.

The demand is structural. Hyperscalers — Google, Microsoft, Meta, Amazon — are making multi-year capital investments in AI infrastructure. They're signing multi-year procurement contracts for both GPUs and HBM. That's not consumer demand that snaps back after a shock; it's committed spending on a long horizon.

The supply is genuinely constrained. Building new HBM fab capacity takes two to three years. And Samsung's yield troubles mean real supply is growing slower than anyone planned.

And demand is sticky. Here's the subtle part — even if AI investment cools off tomorrow, all the models already deployed still need HBM to keep running inference. The installed base of AI accelerators creates ongoing demand that doesn't depend on anyone buying new hardware.

So what does this mean if you're an engineer building on this stuff? A few practical things. Cloud GPU costs are going to stay high — HBM is a big chunk of what an H100 or A100 actually costs, and as long as it's constrained, rental prices stay elevated. That also means model compression stops being an academic exercise and becomes economically rational: quantization down to INT8 or INT4, weight pruning, knowledge distillation — these directly shrink your HBM footprint, which directly cuts your infrastructure bill. And memory-efficient attention becomes a real engineering discipline. Techniques like FlashAttention and PagedAttention — the one behind vLLM — are all about squeezing the KV cache's memory footprint so the same hardware can serve more users or handle longer contexts.

So let me leave you with the core takeaways. First: the AI memory supercycle isn't hype — it falls directly out of a technical fact about how LLMs work. The decoder is bandwidth-bound, the weights are huge, and the KV cache grows with context. Second: HBM isn't just "faster RAM." It's the thing that makes large-scale LLM inference possible at all — pull it out and the whole architecture stalls. And third: because the demand is structural and the supply takes years to expand, this cycle behaves nothing like the temporary shortages we've seen before. How long it lasts really comes down to two racing forces — how fast the industry keeps pouring money into AI infrastructure, and how fast those three manufacturers can actually bring new HBM capacity online.