Every on-prem AI program starts with a procurement spreadsheet and disappoints by month six. The GPUs arrive, the racks fill, the ribbon gets cut, and then utilization settles somewhere in the forties while three teams argue about who is hoarding nodes. The uncomfortable finding is that almost none of that gap is a hardware problem. You bought compute; what you now operate is a thermal envelope, a firmware matrix, a scheduler with political constraints, and a failure surface your existing observability stack cannot see. The silicon was the easy part.
Utilization is worth more than the discount you negotiated
Run the arithmetic on a mid-size cluster: 64 GPUs at roughly $30,000 each is about $1.9 million of compute before fabric, racks, or power. Spread over a three-year life, each single percentage point of utilization is worth roughly $19,000 a year in effective capacity. A cluster sitting at 45% instead of 65% is stranding about $380,000 of annual capacity — and no negotiation you will ever run recovers that. A hard-won 5% purchase discount is a one-time $96,000. Operational discipline beats commercial discipline inside year one, and keeps paying long after the discount is forgotten.
Which is why "we need more GPUs" is usually the wrong first sentence in a capacity review.
Most capacity shortages are scheduling artifacts
The complaint is always supply. The data usually says queue policy. Whole-node reservations for a single interactive notebook, a training job pinned to eight GPUs it uses at 30%, and no mechanism to reclaim either — that is not a shortage, it is a leak. Partitioning helps: one 8-GPU node carved into seven MIG slices turns a waiting list of 12 researchers into 56 concurrent slots. Not every workload tolerates a 1g slice, but notebook sessions, small fine-tunes, and inference canaries almost always do.
Two numbers belong on the wall, not one. Utilization alone is vanity: a cluster at 95% utilization with six-hour queue times is worse for the organization than one at 80% with ten-minute queues, because the second one lets people ship. Track queue time by team and priority class, and the shortage turns out to be concentrated in one or two scheduling decisions.
The failures are physical, boring, and silent
Nobody pages you when a rack's inlet temperature creeps above the envelope. A hall provisioned for 6 kW racks that receives 10 kW GPU nodes does not fail — it throttles. Clock speeds drop, throughput falls 15–25%, and every dashboard stays green because the pods are Running and the health checks pass. The same is true of memory: correctable ECC errors accumulate for days before an uncorrectable one takes the node down, and a reboot clears the symptom while leaving the bad HBM in service.
A cluster rarely fails loudly. It degrades at 0.9x speed for eleven weeks, until somebody finally looks at goodput instead of uptime.
Firmware is the third silent one. GPU, BMC, NIC, and switch firmware drifting across a fleet turns a routine driver upgrade into a multi-day compatibility investigation, and it turns "identical" nodes into snowflakes no runbook covers.
Your monitoring sees the pod, not the die
node_exporter tells you the host is up. It has nothing to say about GPU 3 quietly dying. The mechanism that fills the gap is DCGM scraped by dcgm-exporter into Prometheus, which gives you per-device thermal, clock, ECC, NVLink, and XID fields. The point is not to collect them; it is to name the handful that drive action.
| Signal | How it shows up before it hurts | Policy |
|---|---|---|
| Uncorrectable ECC | any non-zero delta over a window | cordon at next checkpoint, RMA |
| XID 79 | GPU falls off the bus, repeatedly | drain the node, do not just reboot |
| Thermal throttle | SM clock under rated for minutes | shift queue, audit inlet temp |
| NVLink CRC errors | rising hourly error rate | block new jobs on the node |
A small set of XID codes — 48, 63, 74, 79, 94, 95 — covers most of what actually takes a GPU node out of service. You do not need a thousand metrics. You need these and a policy attached to each.
Remediation without blast radius is just faster damage
Automation that cordons and drains on every ECC blip destroys more work than it saves. A drain that throws away 72 hours of training progress is worse than four hours of degraded throughput on a node you already know is marginal. The policy has to evaluate the workload before it touches the node:
- How old is the newest checkpoint, and does that fit the job's restart budget?
- Is this job gang-scheduled, and are its peers healthy enough to survive a restart?
- Is the node hosting a singleton workload with no reschedule path?
- Does the degradation actually cost more than the restart would?
Answer those in a policy engine, not in a runbook that depends on who is on call. The value of auto-remediation is that the same decision gets made identically at 3 p.m. and 3 a.m., with the reasoning written down.
Where this is heading: goodput as the unit of account
The next version of this discipline stops reporting utilization and starts reporting goodput — useful work completed per GPU-hour, after throttling, after restarts, after queue time. Utilization measures whether the light is on; goodput measures whether the cluster is doing anything. Once you can compute it, the operational questions change shape: a node with rising ECC deltas and a thermal trend becomes a scheduled cordon three days out instead of an outage at midnight.
That is the real end state of on-prem AI infrastructure — not a bigger cluster, but a fleet with the same boring, instrumented, policy-driven reliability you already expect from compute you have run for a decade. The teams that get there will not be the ones with the fastest interconnect. They will be the ones who treated the GPU as an operational asset instead of a trophy.