// AI Deep Dive
Memory Decides What AI You Can Run
The chip gets the headline. The memory decides which models fit and how fast they answer, it just got six times more expensive, and it will shape every hardware purchase you make for the next two years.
// Executive Summary
Most people shopping for an AI-capable machine read the spec sheet the way we were taught to: find the processor, compare the compute number. For running AI models, that's the wrong column. Once a model is loaded, the thing that decides how fast it answers is how quickly the machine can move the model's weights out of memory, and the thing that decides whether it runs at all is whether those weights fit.
Answer speed tracks memory bandwidth, not raw compute. Apple's own machine learning researchers found that generating tokens "is bounded by memory bandwidth, rather than by compute ability." Moving from 120 GB/s to 153 GB/s of bandwidth, a 28% increase, made text generation 19% to 27% faster.
Memory got about six times more expensive in five quarters. TrendForce reports conventional DRAM contract prices rose 45% to 50% in the last quarter of 2025, then 93% to 98% in the first quarter of 2026, and it still expects another 10% to 15% this quarter.
The data center is taking the wafers. TrendForce estimates high-bandwidth memory for AI accelerators will consume about 30% of the top three suppliers' DRAM wafer input in 2027, up from 18% at the end of 2025.
Rank AI hardware by memory capacity first, bandwidth second and compute third, buy it from more than one vendor, and plan to run it longer than you used to.
// The Hook
A Big Engine on a Narrow Pipe
Picture a huge engine fed through a garden hose. That's a fair sketch of most AI hardware from the inside. The processor can do a staggering amount of math. What it can't do is get the data it needs fast enough, because every word a model writes has to squeeze through the pipe between memory and the chip. Double the size of the engine and the hose stays the same width. The answers don't come out any faster.
I think about that every time I see an "AI PC" sticker. The marketing leads with the engine. Trillions of operations per second. Neural engines. GPU core counts. Microsoft even defines its Copilot+ PC category by it: an NPU that can do more than 40 trillion operations per second. Those numbers are real. They're just not the ones that decide whether a 70-billion-parameter model runs on your desk, or how long you wait for it to answer.
When people ask me what to buy to run AI locally, they almost always ask about the chip. Almost nobody asks how wide the pipe is, or how much the reservoir behind it holds. Those two numbers, memory bandwidth and memory capacity, end up governing the whole experience.
The same physics decides which companies can run capable models on their own hardware, what the next laptop refresh costs, and why the biggest chipmakers are fighting over memory instead of transistors.
// The Deep Dive
Memory Decides What AI You Can Run
Compute is abundant. Memory capacity and memory bandwidth are the scarce ingredients, and they set the rules for who runs what.
The conventional wisdom is simple: AI is a compute problem. Buy the most powerful processor you can afford, preferably with a big number next to "TOPS," and the AI will follow. It's the same instinct that sold megahertz in the '90s and core counts in the 2010s.
For training giant models, compute really is king. For using them, which is what almost every business does, the story is different. When a model writes an answer, it produces one token at a time, and for each token the machine has to read essentially every active weight in the model out of memory. The math is light. The data movement is heavy. NVIDIA's own engineering guidance says it plainly: model execution is "frequently memory-bandwidth bound," particularly in the weights. For AI inference, the memory system is the product and the processor is along for the ride.
What Memory-Bound AI Is
There's a simple rule of thumb that explains most of what you'll experience running a model: tokens per second is roughly memory bandwidth divided by the size of the model in memory. A model that takes up 40 GB, on a machine that can move 273 GB per second, has a ceiling of about seven tokens per second, no matter how many trillions of operations the chip can do. Real software lands below that ceiling, but never above it.
That rule separates three different numbers that spec sheets blur together:
Spec | What it decides | Where you feel it |
|---|---|---|
Compute (FLOPS / TOPS) | How fast the machine reads your prompt | Wait before the first word, long documents |
Memory capacity (GB) | Which models fit at all | Whether a model loads or refuses |
Memory bandwidth (GB/s) | How fast words come out | Reading speed of every answer |
The gap shows up at three levels.
Individual: Apple's researchers measured it directly. The jump from M4 to M5 raised bandwidth from 120 GB/s to 153 GB/s, and generation speed rose almost in lockstep, 19% to 27%.
Team: AMD's top Strix Halo chip, the Ryzen AI Max+ 395, supports up to 128 GB of memory, and AMD says Windows can now assign up to 96 GB of that to AI workloads, enough for models up to 128 billion parameters in a thin laptop.
Organizational: At data center scale, the imbalance is structural. A Berkeley-led analysis found peak server compute grew 3.0x every two years while DRAM bandwidth grew only 1.6x. Every generation, chips get relatively hungrier for data they can't get fast enough.
How It Works in Business Contexts
The last three years of AI hardware make more sense once you see them as a series of workarounds for the memory problem.
Stage one: the discrete GPU and its VRAM ceiling. A graphics card has very fast memory but not much of it. NVIDIA's GeForce RTX 5090 moves 1,792 GB/s, but it carries 32 GB. Anything bigger has to be squeezed, split across cards or spilled into slower system memory. The workstation RTX PRO 6000 has the same 1,792 GB/s with 96 GB, at workstation prices.
Stage two: unified memory changes who can play. Apple silicon, AMD's Strix Halo and NVIDIA's DGX Spark all put one large pool of memory next to the processor and let the GPU use most of it. The Mac Studio with M5 Ultra offers up to 512 GB at 1.2 TB/s. NVIDIA's DGX Spark pairs 128 GB with 273 GB/s and rates itself for models up to 200 billion parameters. Capacity that used to require a server now fits on a desk, and the AI labs have noticed. The Information reported that OpenAI has bought tens of thousands of Mac minis and Mac Studios to train computer-use agents. Apple is now reportedly weighing an AI server built around its M8 chips for customers who want to run models on their own hardware, possibly linked with NVIDIA's NVLink Fusion, with a potential release in 2029 and nothing final, while its in-house server chip, which analyst Ming-Chi Kuo expects in volume production late this year, stays inside Apple's own data centers.
Stage three: quantization as the great equalizer. Storing weights in fewer bits shrinks both the capacity you need and the bytes you move per token. NVIDIA says its 4-bit format cuts model memory by about 3.5x versus 16-bit with 1% or less accuracy loss on the benchmarks it tested.
Stage four: mixture-of-experts models bend the rule. Newer models only activate part of their weights per token. AMD's example is a 109-billion-parameter model with 17 billion active: you still need memory to hold all of it, but you only read the active slice, and AMD reports up to 15 tokens per second on the Ryzen AI Max+ 395.
Bandwidth sets the speed limit, if the model fits at all. Theoretical ceiling = memory bandwidth ÷ 40 GB model. Source: Apple, NVIDIA and AMD spec pages, verified Oct. 15, 2026.
Your AI Sherpa,
Mark R. Hinkle
Founding Publisher, The AIE Network
Follow me on LinkedIn
If you want to get in contact or give me feedback, reply to this email. I read every single one of them.
