Our on-prem GPU plan assumes today's serving efficiency, and that assumption expires

Imagine signing a five-year fuel contract for a fleet, then watching the engines get more efficient every quarter. That is the shape of on-premises GPU planning today.

The assumption hiding inside every on-prem GPU plan

Every sovereign GPU capacity plan contains a silent assumption: that the cost of serving a request tomorrow looks roughly like the cost of serving it today. Nobody writes that sentence into the business case. It sits underneath the sizing model, in the tokens-per-request figure, the concurrency estimate and the peak-load multiplier.

That assumption has an expiry date, and the plan rarely says when.

Why sovereign delivery makes this harder

In national-scale government delivery, sovereign and on-premises AI is a real constraint, not a preference. Data residency, network isolation and security alignment decide where the workload runs before anyone discusses model choice. That means capital hardware, a procurement cycle measured in years, and a commitment signed long before the workload is fully understood.

Public cloud buyers can be wrong quietly and resize next quarter. On-premises buyers are wrong loudly, in an asset register, for the life of the depreciation schedule.

And they can be wrong in both directions. Undersize, and services queue behind a GPU pool. Oversize, and an entity has paid for capacity that efficiency improvements later made unnecessary. Both outcomes come from the same mistake: sizing once against a peak-load estimate and treating that number as fixed.

Three things to separate before anyone signs

1. Model-level efficiency

There is real, published progress here. One recent method, self-supervised confidence fine-tuning, reduces the number of tokens a reasoning model generates by up to 25 percent at matched accuracy, and it does so without early-stopping tricks or length penalties during training. Fewer tokens for the same answer quality is a genuine reduction in work.

Then read the small print. The same source is explicit that this addresses computational efficiency through a novel approach to training rather than inference optimization. It makes no claim about GPU throughput, serving stacks, batching behaviour or inference hardware. Those are different questions, measured on different instruments.

2. Serving-stack and hardware efficiency

This is the layer most executives actually mean when they say serving efficiency: continuous batching, quantisation formats, speculative decoding, newer accelerator generations. It is also the layer where I have no evidence to offer in this pass. No MLPerf Inference data, no vLLM or SGLang benchmark set, no generation-over-generation hardware comparison.

So it gets no number in the plan. Not zero, not a placeholder percentage. It gets a named uncertainty and a date to measure it.

3. The quality of the evidence itself

This is the part worth sitting with. Of the three sources returned when looking for serving-efficiency evidence, two were not about AI infrastructure at all. One addresses pure mathematics, specifically the Bernoulli free boundary problem. The other focuses on condensed matter physics and magnetic materials. Both are legitimate work. Neither tells you anything about tokens per second per dollar.

A capacity plan assembled from that kind of pile ages exactly as badly as an article that forces a conclusion its sources cannot carry. Thin evidence, stated confidently, is the actual risk in sovereign GPU procurement. The hardware is the easy part.

The counterarguments worth taking seriously

Token savings do reach the serving bill

A well-informed reader will say that a 25 percent token reduction eventually shows up as a serving-side saving once the fine-tuned model is in production, so training-time and inference-time efficiency are not disconnected. That is correct, and I would not argue otherwise.

My objection is narrower. The mechanism is different, the evidence base is different, and the realisation is conditional on you actually fine-tuning and deploying that model on your own workload. Treating a training-time result as a throughput forecast is how borrowed numbers enter a five-year budget and quietly become commitments.

The bigger risk is hardware generations, not token counts

Someone closer to procurement would go further: the dominant risk is generation-over-generation improvement in accelerators and serving frameworks, not model-level token efficiency. I agree, and that is precisely why the distinction needs to be explicit. Two different clocks are running. One is the model clock, the other is the silicon and serving-framework clock. A plan that merges them into a single efficiency assumption cannot tell you which one broke when the numbers miss.

The worked example from the field

Inside a UAE federal government entity, we run an on-premises multi-agent AI platform for public services across web, portal and mobile, aligned to national security standards, alongside an Arabic-first internal AI-agents platform and AI coding copilots for more than 30 engineers. The pattern that surprises people is this: agentic workloads do not consume tokens the way chat workloads do.

A single citizen request can fan out into retrieval, several tool calls, a policy check and a verification step. One user action, many model calls. In that architecture, demand growth from adding agent steps can move faster than any efficiency gain from the model or the hardware. The efficiency curve and the demand curve are both moving, and neither one is in the original sizing spreadsheet.

This is the same lesson as automating 130-plus services in an enterprise setting. The technology rarely breaks the plan. The plan breaks because it was written once and never given an owner or a review date.

What I would write into procurement

Booked headroom, priced up front

Agree expansion capacity and its commercial terms at signature, not at the point of crisis. Headroom you have already negotiated is an architectural option. Headroom you negotiate under pressure is a penalty.

A re-benchmarking date on your own workload

Fix a calendar date to re-measure tokens per request, concurrency and utilisation on your real traffic, with your serving stack, not on a vendor slide. Published efficiency results are a reason to re-measure, never a substitute for measuring.

A named owner for the assumption

Most organisations do not have a data problem or an AI problem; they have an ownership problem. Capacity assumptions are no different. TOGAF already provides the home for this in architecture change management, Phase H. Most programmes treat that phase as paperwork. For GPU capacity through 2026, it is the control that carries the most money.

The takeaway

Do not size a sovereign GPU estate against a peak-load estimate. Size it against a review cycle, with headroom booked and a re-benchmarking date in the contract. Write down what you do not know, including the numbers you could not find, and give that list an owner.

What event would trigger a re-size in your current plan, and who is authorised to call it?