The GPU Scarcity Paradox
TL;DR
The GPU shortage isn't what you think. The industry doesn't have a supply problem. It has a utilization problem masquerading as one. We are extracting barely one-third of the power from the hardware we've already deployed.
The industry keeps buying GPUs because its dashboards make wasted capacity look like unavailable capacity. As measurement and operations catch up, operators will discover they have more compute than they realized, and the advantage will shift from who owns the most GPUs to who operates them best.
The Infrastructure Arms Race
Microsoft, Google, Meta, and Amazon are on track to spend nearly $700 billion on AI infrastructure in 2026.1 Amazon alone committed $131 billion in CapEx in 2025, with expectations for $200 billion in 2026.2 The hypothesis driving all this investment has remained the same since late 2023: GPU supply is structurally constrained, and if you don't secure capacity now, you will fall behind.
For a while, that thesis was hard to argue with. In 2023 and 2024, wait times for H100 GPUs stretched eight to twelve months on AWS and Azure.3 H100 cards traded on secondary markets for north of six figures, a 300% premium over NVIDIA's list price.4 Real constraints existed, and in some cases, capacity remains genuinely scarce.
But utilization data tells a different story. Anyscale reports that production AI workloads achieve well below 50% sustained GPU utilization, even under load.5 Fujitsu found that over 75% of organizations report GPU utilization below 70% at peak, dropping to 30% off-peak.6
GPU scarcity is real, but the market overstates it because it conflates "allocated" with "actually working." Once you measure what GPUs are really doing, a lot of the "shortage" turns out to be waste. The waste persists despite demand because nobody measures it, and because buying more GPUs is easier than fully using the ones you have.
The Measurement Problem
Walk into any data center and ask about GPU utilization: they'll likely show you a dashboard reading 95%.
Most teams think their GPUs are fully utilized because they're looking at the wrong metric. Their dashboards show allocation, meaning how many GPUs are claimed, not how much actual compute is happening per GPU. That gap is where the waste hides.
Anyscale's 2026 report confirms the pattern: most teams track GPU utilization through nvidia-smi or orchestrator-level dashboards rather than hardware profiling counters.5 The default monitoring path reports metrics that overstate utilization by 50–70 percentage points.5
This isn't a minor discrepancy. Every downstream decision, from capacity planning to procurement to scaling, operates on data that conflates "allocated" with "working." Teams buy more GPUs because dashboards say they need more, when the real problem may be extracting value from what they have.
Fixing the measurement is the prerequisite to fixing everything else, because you can't optimize what you can't see.
Once teams correctly measure waste, it becomes visible: idle time between bursty training steps, overprovisioned "warm pools" kept alive to prevent inference cold-starts, and compute silicon starved of data by slow storage pipelines.
Measurement also has a harsher requirement that most monitoring stacks fail: the numbers have to be good enough to bill against. A utilization figure nobody would put on an invoice is a figure nobody fully trusts, and untrusted numbers don't change procurement decisions.
From Waste to Idle
Here's the twist: when you fix utilization, you don't eliminate waste. You move it.
Better utilization packs work onto fewer GPUs, which means the rest sit completely idle. Poor utilization and idle capacity have a direct, inverse relationship: fixing one creates the other.
Even before you factor in optimization efforts, idle capacity is a persistent reality in almost every workload: between training runs, when inference traffic drops, and across seasonal demand troughs. GPU clusters are sized for peak demand, and peak demand is intermittent by nature.6
This idle capacity isn't just a technical inefficiency. It's a financial time bomb. Modern clouds are taking on debt to buy GPUs, but silicon is a rapidly depreciating asset. With new architectures arriving every 18 months, the revenue-per-GPU is in a race to the bottom.
If an operator cannot extract value from a chip during its short window of relevance, the debt will outlive the hardware.
In this market, "buying more" isn't a strategy. It's a default in slow motion.
The survivors will be the operators that extract value from every node they have already deployed.
One could argue this is simply how infrastructure works. Power plants, airlines, and telecom networks all carry reserve capacity. Idle hardware is expected and priced into the business model.
But GPU infrastructure differs in a critical way: the workloads are software-defined. An idle airplane can't fly a different route for 45 minutes. An idle GPU can take on different work the moment software decides it should. The binding constraint isn't physics. It's whether anyone can prove, for a given hour, that a given node is healthy, available, and accounted for.
For operators who have already invested in the hardware, idle GPUs are pure cost: power, cooling, and depreciation with zero return. Yet as peak demand grows, leadership is left with the same decision: buy more.
The Orchestration Gap
The fiber optic bubble of 1996–2001 offers a useful parallel to GPU clusters.
Telecom companies invested over $500 billion laying 80 million miles of fiber across North America. By 2005, 85% of it was dark.7 The bust destroyed $2 trillion in market value. Yet the overbuild created the foundation for the modern internet.
The lesson wasn't that the investment was wasted. It was that value came not from laying the most fiber, but from software that could light it, route through it, and monetize dormant capacity.
AI infrastructure is at a comparable stage. According to McKinsey, building the forecasted data center capacity for 2030 will require trillions in CapEx.8 Whether that capital flows into raw silicon, advanced cooling, or real estate, the return on that massive physical investment is entirely dictated by the software and operations layer above it.
Right now, hundreds of billions flow to hardware, and a fraction funds the operations layer that determines whether that hardware produces value.9
The true winners of the fiber crash weren't the companies that dug the most trenches.
While giants holding the debt for millions of miles of "dark fiber" went bankrupt, companies like Equinix survived and went on to become the world's largest internet exchanges.10 Their secret? They realized raw physical cable was a low-margin race to the bottom.
Instead of competing on sheer volume, they built the interconnection layer: the routing and orchestration hubs that allowed fragmented, overbuilt networks to actually talk to each other. They turned static glass into a fluid, monetizable market.
Today, the broader industry is repeating the telecom mistake, treating "dark silicon" as the only competitive advantage. But the value is moving up the stack.
The survivors won't be the operators with the largest hardware fleets. They will be the ones who adopt the Equinix playbook: stop competing on how much silicon you hold, and start competing on how well it runs.
The Fix Is Operational
The fix isn't another procurement cycle. It's an operations and software layer that makes deployed hardware dependable and measurable. Dependable means a node proves it works before it earns, and is watched every hour after. Measurable means every hour of its life is recorded well enough to settle an invoice, an SLA question, or an audit. Three things follow.
Utilization is invisible without billing-grade telemetry. A dashboard that conflates "allocated" with "working" can't price anything, sell anything, or fix anything. Per-node state history kept to the standard of an invoice is what turns waste from an impression into a number someone owns.
Wasted capacity looks unavailable because nobody can prove otherwise. A healthy node with no evidence of its health can't be scheduled with confidence, sold to a customer, or underwritten by a lender. It reads as shortage. Proof is what puts it back on the market.
The advantage shifts to whoever operates hardware best. Fleets are converging on the same silicon from the same vendors. What will differ is the operating record: how fast failures are caught and diagnosed, how quickly a node returns to service, and whether everyone who pays against the meter trusts it.
More GPUs are needed. But the industry's most expensive problem is not the GPUs it cannot buy. It is the ones it already owns and cannot use.
Underutilized, unmeasured hardware is the problem Lilac is built to fix. We build the software layer GPU fleet owners use to run their hardware as a dependable, measurable cloud: node lifecycle from delivery through burn-in and acceptance, 24x7 monitoring, and metering strong enough to settle an invoice. Read how we think about billing-grade metering, or see the consoles providers and customers use.
Sources
- CNBC. "Google, Microsoft, Meta, and Amazon ramp AI cash." (Feb 6, 2026).
- TechCrunch. "Amazon and Google are winning the AI CapEx race." (Feb 5, 2026).
- LLM Utils. "NVIDIA H100 GPUs: Supply and Demand."
- Jarvislabs. "H100 GPU Price Analysis."
- Anyscale. "GPU In-Efficiency in AI Workloads." (Jan 21, 2026).
- AI Infrastructure Alliance / ClearML. "The State of AI Infrastructure at Scale 2024."
- Internet History. "Boom, Bubble, Bust: The Fiber Optic Mania."
- McKinsey. "The cost of compute: A $7 trillion race to scale data centers."
- Mordor Intelligence. "AI Infrastructure Market Analysis."
- DrPeering. "How Equinix Beat MAE-East: IX Playbook Tactics."
← All news