More than once in recent years, a change in one layer of the AI infrastructure stack has pushed the layers around it to rearrange. Training did it to data center cooling. An air-cooled rack tops out around 30 to 40 kilowatts. A rack of NVIDIA GB200 chips, built to train frontier models, draws about 120. Fans cannot move that much heat, so the training buildout pulled direct liquid cooling from exotic to standard in about two product generations.
Model architecture did it to serving software. A transformer holds its entire conversation in memory while it generates, and the first systems to serve it wasted 60 to 80 percent of that memory to fragmentation. PagedAttention, now standard in the vLLM serving engine, applied the operating system's paging trick to the problem, cut the waste below 4 percent, and got 2 to 4 times the throughput out of the same hardware. It exists because the model handed the serving layer a memory-management problem.
Model structure did it to the network between chips. Mixture-of-experts models route each token to a few sub-networks rather than the whole model, which turns generation into constant all-to-all traffic between chips. Keep that traffic on the fast links inside one rack and it holds. Let it cross onto standard Ethernet and per-chip bandwidth falls more than tenfold, dragging throughput and cost with it. The interconnect inside a single rack went from a detail to something model teams plan around.
The commonality: a shift high in the stack steers the direction of layers down.
Compute gravity shifting towards inference
For most of the deep learning era, training was the main effort and focus point. As models transition into constant use, hundreds of millions of users are using billions of tokens a day. There is a second meaningful lever driving most of the compute efforts towards inference.
The older lever, scaling up pretraining, is losing force. Ilya Sutskever, who helped drive the scaling era, now says pretraining runs on data and that data is finite. The gains are increasingly coming from the other lever, spending more compute at inference. Noam Brown, who did much of that early work at OpenAI, makes the strongest version of the case. Pretraining gives a model fast, instinctive answers. Letting it spend compute at inference is what lets it slow down, weigh several options, test them, and correct itself, and that is where the harder problems start to fall. He treats the size of that inference budget as a real measure of intelligence, and argues that a single benchmark score has stopped meaning much, because a model's true ceiling sits behind how much inference you are willing to pay for.
Either way, common prediction, including Jensen's, is that inference will meaningfully outweigh pretraining in the not-so-far future. The spend is following, and here the popular claim runs ahead of the evidence. No major cloud discloses how its compute divides between training and inference. Deloitte forecasts inference at two-thirds of all compute in 2026. Dario Amodei puts the current split at roughly half and half, and NVIDIA's own last disclosed inference share, around 40 percent, is two and a half years old. Training compute is still growing fast in absolute terms, so a rising inference share does not mean training is shrinking; both are climbing at once, yet inference share may very well rise faster.
That centrality raises the question of what other layers of the AI infrastructure stack will be affected.
Where inference and training differ
Training runs whole sequences at once, in big batches, doing a forward pass, a backward pass, and a weight update, and it holds temporary state, the gradients and optimizer values that exist only while the model is learning. For a 405-billion-parameter model that state alone runs to about 6,500 gigabytes, more than 80 top-end chips just to hold it, before the model itself. Training is limited mostly by raw arithmetic. Inference does none of the backward work and runs forward only, but it splits into two phases that behave nothing alike. Reading the prompt, the prefill phase, is arithmetic-heavy, like training. Generating the answer one token at a time, the decode phase, is the hard part.
To produce a single token, the machine reads the entire model out of memory, uses each weight roughly twice, and throws it away, then does it again for the next token. On a 70-billion-parameter model that is 70 gigabytes hauled across the memory bus per token. The arithmetic units finish their share almost instantly and then sit idle waiting for the next load. Generation is memory-bound: the answer arrives when the data arrives, and a faster calculator would not help.
Batching is the usual fix, and it half-works. The weights are shared, so if you serve a hundred users at once you read the model once and use it a hundred times, and the arithmetic units finally get busy. You need roughly 280 users batched together before they are. But each user also has a private memory of their own conversation, the KV cache, the model's stored internal state for every token said so far, re-read on every new token so it is not recomputed each step. That memory is not shared. Batch a hundred users and you read a hundred separate caches. The trick that rescues the weights does nothing here, and on a 70-billion-parameter model a single user at a 128,000-token context is already carrying 43 gigabytes of it.
Long context makes that private memory large. Multi-turn makes it long-lived. One claim to correct, because it keeps getting repeated: that long context is starved by memory capacity. The models that matter fixed it themselves. Newer attention designs, like the multi-head latent attention in DeepSeek's models, cut the memory stored per token severalfold, and that relief came from architecture, not from the chips. The sharper problem is idle time. Watch a real agent session and most of the clock is a person reading and thinking, not a model generating. In one study of coding agents, more than nine-tenths of each session's wall-clock time was the human, and most of every new turn was context the model had already seen. I treat that as directional, since it is one kind of agent and a small group of users, but the direction is the point. In a multi-turn agent the expensive resource is not fresh computation. It is keeping a large private memory alive through long stretches of idleness, cheaply enough to resume without recomputing it.
So the asks are four. Get the weights to the processor faster, or stop moving them. Hold a large and growing per-user memory. Keep that memory alive across idle gaps at a bearable cost. And live with one bottleneck, the private memory during generation, that batching cannot touch.
Who is building for inference needs
Get the weights off the memory bus. The direct move is to stop re-reading the model, by keeping it in on-chip SRAM, the fast memory built onto the die itself. Cerebras does this at the limit, printing a processor the size of a dinner plate with 44 gigabytes of SRAM on the die, so the weights never leave. Groq spreads a model across hundreds of chips, each with a little on-chip SRAM, so together they hold the model without touching external memory. d-Matrix puts the arithmetic inside the memory. Etched and Taalas go all the way and burn the model, in Taalas's case one specific model, into the silicon. Each buys speed with flexibility, and none of them can also train. NVIDIA and AMD take the other road: they do not remove the re-reading, they amortize it with large fast memory and heavy batching, which wins on throughput per dollar, not the single user waiting for a token.
Hold the memory, cheaply. The clean move is to trade bandwidth for capacity, using cheaper memory like LPDDR instead of the high-bandwidth stacks (HBM) on the top chips. Qualcomm's inference card carries 768 gigabytes of LPDDR. Intel's part does the same, and Tenstorrent uses GDDR, the memory from graphics cards. The trade is always the same, more room at lower bandwidth. Worth noting against the grain: the HBM each chip carries per unit of compute has been falling, close to fourfold from the B200 to its successor, which undercuts reading the biggest chips as a capacity answer.
Keep it alive through the idle time. This ask is solved almost entirely above the chip. The answer everywhere is tiering: hold a session's memory in fast memory while it is active, push it out to cheaper storage while the human thinks, pull it back when they return. NVIDIA does it in its serving software. A row of memory and interconnect companies, Astera Labs, Enfabrica, and others, sell the hardware to pool and move that memory across machines. Astera is the one clearly real business in the group, at 308 million dollars of revenue in a single quarter, up 93 percent, and it is a connectivity company that bought its way into KV cache, not a startup built for it. What does not exist is a chip with a block for this. No processor on the market has hardware made to retain, evict, and re-hit conversation memory. The whole answer sits above the silicon.
The ask nobody ships against. The private-memory limit during generation, the one batching cannot fix, is the cleanest case of a real bottleneck with no product. Google's next inference chip narrows it, and NVIDIA's latest part claims to roughly double attention-layer performance, but neither removes the underlying limit. Etched is the only design that hardwires exactly this operation, and it has not shipped in volume or been measured by anyone independent. The hardest ask has no answer on the market.
Split the two phases. Because reading the prompt and generating the answer stress the machine so differently, the natural idea is to run them on different chips. This is where the most is happening, and the least is settled. NVIDIA announced a prefill-focused chip, Rubin CPX, in September 2025, then quietly pulled it about six months later and filled the slot with a licensed version of Groq's decode design. It has not said why. A reversal that fast reads more like a board or memory decision than a change to the processor's core. The strongest sign of real phase specialization is Google's next generation, which splits into two physically distinct chips, one weighted for training and one for inference, the inference part carrying more memory and a dedicated block for the fast, frequent chip-to-chip exchanges that generation depends on. It is the best evidence that phase specialization is coming. It also arrives in late 2027, not today, and both halves use the same memory generation, which is hard to square with a pure bandwidth story.
Rewire the network between chips. Training and inference stress the interconnect in opposite ways, and here the response is real and already shipping. Training syncs gradients in bulk: one all-reduce on a 70-billion-parameter model moves about 140 gigabytes and can tolerate delay. Decode's equivalent is around 128 kilobytes, a million times smaller, but it fires hundreds of times per token on the user's critical path, where it eats close to a quarter of generation time. Latency per message, not bandwidth, becomes the cost, so the answer is to do the arithmetic inside the switch and spare the data a second trip across the fabric. Broadcom's Tomahawk Ultra does exactly that, at 250 nanoseconds and full rate on the smallest packets, and it runs at half the raw bandwidth of Broadcom's throughput switch, a latency-first part rather than a volume-first one. Two other patterns are unique to inference and already have hardware and software against them: the KV cache handed from the prefill machine to the decode machine, hundreds of megabytes per request with no analogue in training, now carried by NVIDIA's NIXL and by Moonshot's Mooncake at more than 100 billion tokens a day in production; and the bandwidth cliff when mixture-of-experts traffic leaves the fast rack-level links for ordinary Ethernet, which is why the same model can run as much as four times cheaper per token inside a 72-chip NVLink rack, and why NVIDIA's pitch for opening that fabric to other vendors names agentic inference alongside training.
Repackage around memory. Packaging moves more slowly, and the honest read is that its inference story is mostly on paper. The cleanest inference move is not to pack more onto the package but to route memory off it: LPDDR modules like SK hynix's SOCAMM2, marketed explicitly to offload KV cache and cut time-to-first-token, add capacity without touching the interposer at all. The richest inference-named work, stacking DRAM directly under the compute die, from d-Matrix and a wave of research chips aimed at long-context decode, names the workload openly, but almost none of it ships in a flagship part, and 3D stacking runs hotter than a flat layout, which is a real limit. The one packaging-adjacent response actually in motion is bandwidth, not capacity: NVIDIA pushed its memory suppliers past the standard for the next HBM generation and lost at least a quarter of schedule for it. That is the generic axis the industry has always pushed, and no one has tied it to multi-turn.
The challenge of new entrants
A new accelerator is worthless without an ecosystem, the software stack that CUDA has owned for more than a decade and the rack it has to plug into, so a startup has two roads and both run through the incumbents. One is to design into NVIDIA's world, licensing its NVLink fabric to sit inside its racks and building on top of CUDA. The other is to go standalone and bet the market adopts a new platform, a bet the specialist cohort is mostly losing: most of them have no disclosed customer at all. And the designs that do work, NVIDIA tends to absorb: the roughly 20 billion dollar Groq arrangement, the 900-million-plus for Enfabrica, structured as licenses and team hires. Partner, get absorbed, or go unadopted.
The second gate is manufacturing. A competitive part needs leading-node wafers, advanced packaging, and high-bandwidth memory, and all three are prepaid, volume-committed, and sold out. HBM comes from just three suppliers, booked through 2026, and even Samsung needed around eighteen months to qualify a single stack against one accelerator. Leading-node and packaging capacity at TSMC is handed out the same way, by prepayment and multi-year commitment to the biggest customers. A startup does not walk up and buy this. It gets in through a giant partner or a sovereign backer willing to reserve the capacity, which is dependence by another name. The ones that dodge the memory gate by skipping HBM pay for it on the other side, in adoption, because a part that sits outside the standard racks is a part the market can't adopt.
A better design does not get you past either gate. To be believed, it has to run on the same scarce silicon and inside the same NVIDIA racks everyone benchmarks on, so it depends on the very capacity and ecosystem it was meant to avoid.
Put the two gates together and the base rate follows. Of the couple dozen merchant AI-silicon startups founded since 2014, about one reached independent scale, and the rest were absorbed, stalled, or shut down. The value in this shift is real. It just accrues to the companies that already own the ecosystem and the supply, and it is hardest to capture as the outsider trying to break in.
The payoff
Every solution to inference bottlenecks drives efficiency gain, a better throughput number or a lower latency, the kind of improvement that shows up as margin. When inference is a minority of the compute bill, shaving a third off the cost of a token is a line item. When inference is most of the bill, the same third is most of the savings, and the numbers in play are not small: a model runs as much as four times cheaper per token inside the right rack, PagedAttention pulled two to four times the throughput out of hardware that did not change, and the network work that eats close to a quarter of generation time is exactly what in-switch collectives are built to remove. These attack different terms, the weights, the private memory, the traffic between chips, so they compound rather than overlap. As inference share climbs, the fraction of all AI spend that these gains apply to climbs with it.
The deeper reason to want it is that cost and capability are the same axis here. A model's new ceiling is how much inference you are willing to pay for, that letting it slow down to weigh options and correct itself is where the harder problems fall. The cost of a token is not only an operating number, it is the price of intelligence. Cut it in half and the same budget buys twice the deliberation on the same model, twice the search before an answer is returned. The inference-specific stack, the chips that stop re-reading the weights, the memory tiers that keep a session alive cheaply, the switches that do the arithmetic in the fabric, is not just making today's workload cheaper. It is lowering the price of every additional unit of thinking, which sets how much intelligence the market can afford to run. Whoever intends to lead will not pass on an advantage this large.