Physical infrastructure · AI compute

Will Inference Change the Data Centre Architecture?

Training and inference are increasingly discussed as if they require different data centres. Sometimes they do; often they don't. A GB300 rack running reasoning inference is architecturally close to one training the underlying model. A metro colo serving efficient 7B models under a tight latency SLA is not. The rest of this piece is about when the two diverge and when they stay one.

Reader takeaways
The thesis
Inference workloads vary across model scale, latency SLA, utilisation profile, geographic constraint and compute coupling. Infrastructure varies with them. GB200/GB300 run training and inference on the same rack-scale liquid-cooled platform. A metro colo runs efficient 7B models on 20 kW air-cooled racks. Both are inference. The infrastructure around them differs materially depending on where the workload lands on those five variables.
Workload mechanics
Training is forward + backward pass on large batches and is throughput-bound. In this analysis, training is modelled at 90%+ utilisation. Inference is forward-pass only, latency-bound, with more variable utilisation tied to user demand.
Rack-level TDP
GB200/GB300 NVL72 racks are in the ~120-140 kW class (NVIDIA specifies GB300 NVL72 up to 142 kW) and require DTC. DTC is a density mandate; the workload label is orthogonal. Inference racks span roughly 20-100 kW depending on silicon choice and can often stay air-cooled at the low end.
Where they diverge
Cooling architecture, network fabric (east-west vs north-south), storage stack (dataset vs KV cache), silicon mix (dense GPU vs heterogeneous), grid connection (cheap power vs metro proximity), and power chain (sidecar-first vs traditional PDU).
Four regimes
Q1 consolidated (plateau + relaxed SLA). Current default AI-infrastructure architecture, GB200/GB300 scope. Q2 metro edge (plateau + strict SLA). Small models on air-cooled colo racks, CDN-like. Q3 purpose-built inference (grow + relaxed SLA). Batch/agent workloads on custom silicon. The awkward case: if latency thresholds relax faster than model sizes shrink, Q3 collapses back into Q1. Q4 metro giant (grow + strict SLA). Technically feasible; the regime least likely to scale on economics alone. Which regime lands where depends on the five threshold variables above.
Value migration
The current AI capex cycle is overwhelmingly organised around training-scale accelerator infrastructure. As inference regimes diversify, value migrates away from a single rack-scale GPU architecture toward a broader serving stack: accelerator silicon, memory bandwidth, KV-cache networking, orchestration software, metro capacity, peering, and power efficiency.

01The workload of inference, and the infrastructure around it

The default today is consolidation. Hyperscaler AI clusters run mixed workloads on the same racks that trained the models. NVIDIA GB200 NVL72 and GB300 NVL72 are both rack-scale, liquid-cooled platforms explicitly positioned to span training and reasoning inference. That is architectural evidence that inference workloads can run on training-DC infrastructure.

But the workload of inference varies enormously across model size, latency requirement, utilisation profile, geographic constraint and compute coupling. The physical infrastructure around it varies with those variables. GB300 NVL72 is a 100+ kW-class rack-scale liquid-cooled platform explicitly designed for AI reasoning. A metro colo runs efficient 7B-70B models on air-cooled 20 kW racks with an Ethernet ToR switch. Both are inference workloads. Four regimes span the range in between.

Figure 1
One workload regime, multiple physical architectures. Which thresholds push inference into its own infrastructure?
One workload regime. Multiple physical architectures possible. WHICH THRESHOLDS PUSH INFERENCE INTO ITS OWN INFRASTRUCTURE? TODAY · CONSOLIDATED (default 2022-2026) GB200 NVL72 rack serves BOTH workloads TRAINING Forward + backward throughput-bound · 24/7 flat INFERENCE Forward only latency-bound · spiky 40-60% SHARED TODAY ✓ Same rack GB200 NVL72 ✓ Same cooling DTC + row CDU ✓ Same power Sidecar / 800V DC ✓ Same fabric WHY IT WORKS Training capex justifies the build Inference pays the run 95%+ effective utilisation Location: training WHERE INFERENCE DIVERGES · FIVE VISIBLE DELTAS IN THE PHYSICAL STACK BOTTLENECK TRAINING Throughput gradient sync INFERENCE Latency 100ms-2s SLA FUNDAMENTAL DELTA Common silicon; optimisation target varies UTILISATION TRAINING Flat 90%+ 24/7 base load INFERENCE 40-60% peaks 3-5x troughs CAPEX ROI DELTA Inference capex earns fewer productive hours SILICON MIX TRAINING 4 vendors NVIDIA/AMD/TPU/Trn INFERENCE 10+ vendors GPU/LPU/TPU/SoC VENDOR CAPTURE DELTA Heterogeneous silicon = different DC integration GEOGRAPHY TRAINING Cheap power TX / VA / Iceland INFERENCE Metro proximity NY / London / Tokyo REAL-ESTATE DELTA $/MW and land cost differ materially RACK DENSITY TRAINING ~120-140 kW DTC mandated INFERENCE 20-100 kW air / hybrid works COOLING/POWER DELTA Lower density unlocks unlocks new form factors TOMORROW · ONE ARCHITECTURE OR MULTIPLE REGIMES? SCENARIO Q1 · CONSOLIDATED Training + inference stay together Hyperscaler status quo Sidecar / 800V DC / DTC continues → ? ← what this essay asks SEE §13 SCENARIOS Q2/Q3/Q4 · DIVERGED Inference-specific DCs emerge New vendor field, new geography Purpose-built / metro edge / metro giant DRIVER VARIABLES · WHAT CAUSES THE FIVE DELTAS ABOVE (SEE §02) Model scale Latency SLA Utilisation profile Geographic constraint Compute coupling Drivers determine which architecture. Outputs are what changes when the drivers cross a threshold.
Today, GB200/GB300 rack-scale platforms span training and reasoning inference on the same physical infrastructure. The visible deltas in the physical stack when inference stops looking like training: bottleneck (throughput vs latency), utilisation (sustained 90%+ vs variable 40-60%), silicon mix (concentrated on training accelerators vs broader inference design space), geography (cheap-power hyperscale vs regional / metro / edge), rack density (~120-140 kW rack-scale for GB200/GB300 NVL72 vs 20-100 kW server-scale). These are outputs. The threshold variables that drive them (model scale, latency SLA, utilisation profile, geographic constraint, compute coupling) are introduced in §02.

02Workload mechanics at the chip

Training and inference stress the same silicon in different ways. The chip-level difference is what propagates outward through the stack.

Figure 2
Chip-level workload mechanics. Training vs inference on the same silicon
TRAINING Forward + backward pass. Throughput-bound. DATAFLOW 1. Forward pass: input batch → activations → loss 2. Backward pass: loss gradient propagated through weights 3. Optimizer step: weight update (SGD / Adam / etc.) All three every iteration. Iteration = milliseconds to seconds. MEMORY FOOTPRINT • Model weights (resident in HBM) • Activation checkpoints across layers (recomputed or stored) • Optimizer state (~2-4x weight size for Adam) • Gradients (weight-size) METRICS THAT MATTER Batch size 1024-8192 Target utilisation 90%+ Memory BW demand Very high East-west traffic Very high (grad sync) Bottleneck ordering Memory bandwidth binds first; then interconnect; compute rarely. INFERENCE Forward pass only. Latency-bound. DATAFLOW 1. Prefill: input tokens → KV cache populated 2. Decode: one token at a time, KV cache grows 3. Return: full response streamed to user No backprop, no gradient sync. Two phases: prefill (parallel) + decode (sequential). MEMORY FOOTPRINT • Model weights (resident in HBM, read-only) • KV cache per active session (grows with context length) • Activation buffers (single token wide) • No optimizer state, no gradients METRICS THAT MATTER Batch size (typical) 1-256 Latency SLA 100ms - 2s Memory BW demand Very high (decode) East-west traffic Low (or none) Bottleneck ordering Memory bandwidth first, then latency (network + serial decode), then compute. Common silicon; bottleneck ordering diverges by workload. That is why the DC design diverges.
Training and inference stress silicon differently. Training is throughput-bound with high east-west traffic for gradient sync. Inference decomposes further into two sub-phases with distinct bottlenecks: prefill (compute-bound, parallelisable, batchable) and decode (typically memory-bandwidth-bound and autoregressive: each sequence generates one token at a time, although many sequences can be decoded concurrently). Three bottleneck modes on the same underlying hardware. Everything downstream (rack integration, network fabric, facility design) responds to which mode dominates.
PREFILL / DECODE DISAGGREGATION

Two sub-phases with different resource needs. Prefill is compute-bound and scales with batch size; decode is typically memory-bandwidth-bound and autoregressive: each sequence generates one token at a time, with many sequences decoded concurrently. Recent serving architectures (NVIDIA Dynamo distributed inference, DistServe, Splitwise) run the two phases on separate worker pools so each phase uses hardware matched to its bottleneck.

KV cache becomes network traffic. When prefill and decode run on separate nodes, the KV cache (populated during prefill) must transfer to the decode node before token generation starts. For large-model prefill/decode disaggregation, KV-cache transfer can push interconnect requirements into the hundreds of GB/s, which pulls RDMA-class interconnect (InfiniBand, RoCE) into inference clusters that would otherwise use pure Ethernet.

Large-model inference does not eliminate east-west traffic. It changes what the east-west traffic is for.

Where single-node inference becomes multi-node. A model fits in a single HBM domain when the working set (weights + KV cache at target context length) stays within the aggregate HBM available on one server or one NVLink domain. Public reference points: H100 SXM node = 8 × 80 GB = 640 GB HBM3; B200 SXM node = 8 × 192 GB = 1.5 TB HBM3e; GB200 / GB300 NVL72 = 72 GPUs × ~192 GB ≈ 13.8 TB HBM3e in one NVLink domain. A 671B-parameter FP8 model with a 1M-token context can require hundreds of GB of KV cache per active sequence, which pushes past any single node into rack-scale coupling. Where the boundary falls determines whether the compute plane fits inside NVLink / NVSwitch or requires RDMA-class scale-out.

KV CACHE MATH · THE INTERCONNECT PRESSURE

Byte footprint per token in the KV cache is a function of architecture:

Bytes_per_token = 2 × n_layers × n_kv_heads × d_head × b_precision

(the factor 2 counts Keys and Values.)

Architecture matters. A dense Multi-Head Attention (MHA) 671B model would explode the KV cache. DeepSeek V3 / R1 uses Multi-Head Latent Attention (MLA), compressing state into a latent dimension d_c = 512 plus a decoupled RoPE key dimension d_R = 64. Per token per layer: (512 + 64) × 1 byte (FP8) = 576 bytes. Across DeepSeek-V3's 61 layers: 61 × 576 ≈ 34 KB per token at FP8 (~69 KB at FP16). Llama-3.1-405B with Grouped-Query Attention (126 layers, 8 KV heads, d_head = 128, FP16) lands closer to 500 KB per token, more than an order of magnitude larger than the MLA case. Model-architecture choice is therefore a first-order network requirement rather than a downstream detail.

Transfer time on prefill-to-decode handover:

t_transfer = Prompt_length (L) × Bytes_per_token / Fabric_bandwidth (B_link)

Worked example (AUTHOR SCENARIO). A 128k-token prompt on Llama-3.1-405B (FP8 approximation ~33 GB of KV state) transferred over a 400 Gbps interface (~40 GB/s effective after protocol overhead) takes ~830 ms. That blows a 100 ms TTFT budget on the disaggregated handover alone, unless the transfer is pipelined layer-by-layer, served over multi-rail RDMA, or kept inside rack-scale NVLink.

MLA-style compressed cache stays inside a rack under most working conditions. MHA and GQA at frontier scale force multi-rail RDMA or rack-scale NVLink into the compute plane.

Figure 2b
KV cache footprint per token. MLA (DeepSeek V3 / R1) vs GQA (Llama-3.1-405B) at frontier scale.
KV cache footprint per token · two attention architectures AUTHOR SCENARIO · FP8 REFERENCE MULTI-HEAD LATENT ATTENTION (MLA) DeepSeek V3 / R1 61 layers · latent d_c = 512 · RoPE d_R = 64 PER-TOKEN FOOTPRINT DERIVATION (d_c + d_R) × b_precision × n_layers = (512 + 64) × 1 B × 61 = 35,136 B ~34 KB / token at FP8 (~69 KB at FP16) Relative footprint (scaled to GQA reference) Cache stays inside a rack under most working conditions. GROUPED-QUERY ATTENTION (GQA) Llama-3.1-405B 126 layers · 8 KV heads · d_head = 128 PER-TOKEN FOOTPRINT DERIVATION 2 × n_layers × n_kv_heads × d_head × b_precision = 2 × 126 × 8 × 128 × 2 B (FP16) = ~504 KB ~500 KB / token at FP16 (~250 KB at FP8) ~15× more per token than MLA at the same scale Frontier scale pushes past a single node. Model-architecture choice propagates directly into fabric requirements.
Byte-per-token footprint differs by more than an order of magnitude between the two dominant attention architectures at frontier scale. MLA compresses to ~34 KB per token at FP8 (DeepSeek V3 / R1, 61 layers, d_c=512, d_R=64). GQA at 405B parameters lands closer to ~500 KB per token at FP16 (Llama-3.1-405B, 126 layers, 8 KV heads, d_head=128). At high context length, the ratio determines whether the KV cache stays inside a single NVLink domain or forces multi-rail RDMA into the compute plane.
Figure 2c
KV-cache transfer time by fabric bandwidth. 128k prompt at frontier scale. TTFT budget as constraint.
KV-cache transfer time for a 128k prompt at frontier scale AUTHOR SCENARIO · 100 ms TTFT BUDGET FABRIC BANDWIDTH effective throughput MLA · DeepSeek V3 128k prompt ≈ 4.4 GB payload (FP8) GQA · Llama-3.1-405B 128k prompt ≈ 33 GB payload (FP8) 100 Gbps ~10 GB/s effective Legacy RoCE / top-of-rack ~440 ms BLOWS 100 MS BUDGET 4.4 GB / 10 GB/s ~3.3 s FAR OUTSIDE BUDGET 33 GB / 10 GB/s · ~33× over budget 400 Gbps ~40 GB/s effective Current-gen InfiniBand NDR ~110 ms MARGINAL · JUST ABOVE BUDGET 4.4 GB / 40 GB/s ~830 ms BLOWS BUDGET BY ~8× 33 GB / 40 GB/s 800 Gbps ~80 GB/s effective Next-gen IB XDR / ConnectX-8 ~55 ms FITS 100 MS BUDGET 4.4 GB / 80 GB/s ~413 ms BLOWS BUDGET BY ~4× 33 GB / 80 GB/s NVLink 5 intra-rack Rack-scale, aggregate ~7 TB/s GB200 / GB300 NVL72 domain <1 ms TRIVIAL payload negligible vs bandwidth ~5 ms FITS EASILY only intra-rack NVLink absorbs the flow Only NVLink-domain transfers keep GQA within TTFT at frontier scale. MLA fits at 800 Gbps or better inter-node.
Transfer time is payload divided by effective fabric bandwidth. For a 128k prompt, GQA-style caches (~33 GB at FP8) require an intra-rack NVLink domain to meet a 100 ms TTFT budget on the prefill-to-decode handover; MLA-style caches (~4.4 GB) fit on 800 Gbps inter-node RDMA, and marginal at 400 Gbps. This is why attention-architecture choice propagates directly into fabric requirements: an MLA model can be served on a purpose-built inference cluster with 400 Gbps oversubscribed RoCEv2, while a GQA-405B model at frontier scale needs rack-scale NVLink (or aggressive layer-by-layer pipelining) to hit the same SLA.
THE INFERENCE DIVERGENCE THRESHOLD · FIVE VARIABLES

Inference does not become a separate data-centre architecture because it is called inference. It diverges when the combination of five variables makes the shared AI-factory architecture economically or physically suboptimal.

1. Model scale: how much compute and memory must be colocated. Small models fit on one server; frontier reasoning models need rack-scale coupling.

2. Latency SLA: how close must compute be to the user. Batch and agent workflows tolerate minutes; interactive chat needs sub-second TTFT.

3. Utilisation profile: how predictable and continuously loadable demand is. Training is fully schedulable; user-facing inference is not.

4. Geographic constraint: whether the workload can move to cheap-power geography, or user proximity dominates.

5. Compute coupling: whether inference runs independently on a single node, or requires tightly coupled distributed compute (large model + MoE routing + prefill/decode disaggregation).

Rack density, silicon mix, cooling architecture and network topology are outputs of these five variables. Physical architecture only changes when enough of the five cross their threshold together.

These five variables do most of the work. They don't move together, and they aren't independent. That is why two inference sites can look nothing alike even when they serve the same model.

03Rack-level TDP and form factor

Rack density diverges materially with silicon choice; the ~120-140 kW rack-scale vs 20-100 kW server-scale comparison below carries the point.

Figure 3
Rack-level TDP and form factor. ~120-140 kW GB200/GB300 NVL72 rack vs 20-100 kW server-scale rack
TRAINING RACK GB200 NVL72 class · ~120-140 kW · DTC required 42-48U RACK NVLink switch tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray GPU compute tray DTC manifold DTC manifold DC busbar drop from sidecar 72 GPUs across 18 trays (NVL72 nomenclature) SPECIFICATIONS Rack power 120-200 kW Per-GPU TDP 1000-2000 W Cooling DTC required Air cannot handle > 50 kW/rack Interconnect NVLink + IB IB fabric NDR / XDR Power delivery Sidecar ±400V DC Rack DC/DC 800V → 50V Reference platforms NVIDIA GB200 NVL72 Vera Rubin NVL144 · AMD MI300X racks Silicon depth 72 GPUs per rack (NVL72). ~$3-5M silicon per rack. HBM3e/HBM4 memory (192-288 GB/GPU). NVLink switch fabric for non-blocking east-west. Rack-scale integration is the training platform. INFERENCE RACK Heterogeneous silicon · 20-100 kW · air or hybrid cooling 42U RACK · 1U/2U SERVERS 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server 1U inference server ToR switch (Ethernet) Rack PDU 208V or 415V AC from row PDU Traditional PDU path SPECIFICATIONS Rack power (range) 20-100 kW Per-chip TDP 200-1000 W Cooling Air / hybrid / DTC Air limit ~40 kW/rack Interconnect Ethernet ToR Fabric 100/400 GbE Power delivery Rack PDU + PSU Voltage 208 or 415 VAC Reference platforms Groq LPU node · Cerebras CS-3 NVIDIA L40S / H100 PCIe · AWS Inf2 · TPU v5e Silicon depth 8-32 chips per rack depending on platform. Silicon $ per rack ranges 5-10x below training racks. Higher server density (1-2U form factor). Server-scale, not rack-scale. Cheaper per unit compute.
GB200/GB300 NVL72 are rack-scale integrated platforms in the ~120-140 kW class (NVIDIA specifies GB300 NVL72 up to 142 kW/rack). They mandate direct-to-chip liquid cooling; rack-scale power delivery such as the sidecar/800V architecture becomes relevant at this density. Server-scale racks span 20-100 kW, run 1-2U server form factor with mixed silicon, and can often stay air-cooled at the low end. The rack-scale versus server-scale distinction is one of the major physical differences. It drives downstream choices in cooling architecture, fabric selection and power delivery.

04Grid-to-chip for the training data centre

The training DC is what most 800V DC / sidecar / OCP Diablo material describes.

Figure 4
Training DC grid-to-chip. Full stack from utility MV to GPU cold plate
TRAINING DATA CENTRE · GRID-TO-CHIP 100-500 MW site. Cheap-power geography. Sidecar-first power chain. DTC cooling. High east-west network. POWER PATH FACILITY MV Utility feed 115-345 kV HV step down to 34.5 kV or 13.8 kV MV feed MV/LV STEP-DOWN Transformer MV to 480 VAC liquid-cooled unit 2-5 MW per pod FACILITY BESS + UPS Ride-through 5-15 min BESS grid smoothing role 40 MW+ per site SIDECAR RACK 660 kW class AC-DC + BBU + ±400V/800V bus Delta / LITEON / Chicony ~800V DC RACK-INTEGRATED PLATFORM GB200 NVL72 Rack DC/DC (Vicor/MPS/ Navitas/EPC) → 50V rail 120-200 kW/rack CHIP GPU cold plate to die 50V → on-die POL → 0.7-1.1V @ die Ferric / MPS / Vicor on-die HBM3e/HBM4 attached COOLING PATH · DTC MANDATED AT TRAINING DENSITY Facility water plant Cooling towers or dry cool Chilled water 20-45°C 50-500 MW rejection scale Row CDU (L2L) ~2.4 MW L2L Delta EVA / CoolIT-Ecolab Motivair / Boyd Row manifold Blind-mate QDs Leak-detection cable Sensata / TE / nVent Cold plate on GPU 2500-3500 W/socket CoolIT / Boyd / Motivair Fluid: PG25 or PG40 Residual air (CRAH) Handles PSU + PDU losses Sidecar itself air-cooled Existing hall infra NETWORK FABRIC · HIGH EAST-WEST BANDWIDTH INTRA-RACK NVLink 5 switch tray 1.8 TB/s per GPU Non-blocking 72-way INTER-RACK InfiniBand NDR / XDR 400-800 Gb/s per link Fat-tree or dragonfly topology MODEL PARALLELISM Tensor + pipeline + data parallel Spans thousands of GPUs Gradient AllReduce dominates traffic NORTH-SOUTH (minor) Egress low Dataset ingest to filesystem Checkpoint replication STORAGE STACK · PARALLEL FILESYSTEM DATASET STORAGE Parallel filesystem (Lustre, GPFS, WEKA, VAST) Petabyte-scale training data All-NVMe tier for hot access HDD tier for cold archival 50-500 GB/s aggregate read bandwidth Vendors: DDN, Weka, VAST, Pure Storage, IBM Storage Scale CHECKPOINT STORAGE Object storage (S3 or S3-compatible) Model state at intervals (min-hr) Versioned; recovery from failure 100s of TB per checkpoint at 405B scale 10s of MB/s per GPU checkpoint write Vendors: AWS S3, Azure Blob, GCS, MinIO, Wasabi ARCHIVAL Cold storage for released models Historical model versions Compliance-driven retention Access frequency: rare Tape or Glacier-tier cold object Vendors: AWS Glacier, Azure Archive, tape libraries
Training DC grid-to-chip stack. Utility MV feed → MV/LV step-down → facility BESS + UPS → sidecar rack (~800V DC output) → rack-integrated GB200 NVL72 platform → chip cold plate. Cooling: facility water plant → row CDU (L2L) → row manifold → cold plate; residual air handles PSU + PDU losses. Network: NVLink intra-rack + InfiniBand inter-rack, high east-west bandwidth for gradient AllReduce (and all-to-all expert routing for MoE, which can occur in both training and inference). Storage: parallel filesystem for training data + object storage for checkpoints + archival tier.

05Grid-to-chip for the inference data centre

The inference DC can diverge at every layer. How much it diverges depends on the regime.

Figure 5
Inference DC grid-to-chip. Diverges at every layer from the training stack
INFERENCE DATA CENTRE · GRID-TO-CHIP 10-100 MW site. Metro geography. Traditional PDU or lighter sidecar. Air or hybrid cooling. High north-south network. POWER PATH FACILITY MV Utility feed 13.8-34.5 kV MV metro utility grade smaller service size MV/LV STEP-DOWN Transformer MV to 480 VAC smaller unit (0.5-2 MW) colo-style deployment UPS + BATTERY Traditional UPS Longer ride-through (5-15 min for SLA) Tier III/IV redundancy ROW PDU Rack PDU + PSU Traditional 415V AC to rack PDU Sidecar optional 415V AC SERVER-SCALE PLATFORM 1U/2U inference servers In-server PSU (12V) → on-board DC/DC → chip 20-100 kW/rack CHIP (heterogeneous) GPU / ASIC / LPU / TPU / SoC L40S / Inf2 / TPU v5e / Groq LPU / Cerebras 200-1000 W per chip DIVERGENCE FROM TRAINING · Smaller MV service · simpler UPS (Tier III/IV focus) · sidecar optional · 1U/2U server form factor · heterogeneous silicon · wider range of accelerator cost/performance points COOLING PATH · AIR-COOLING VIABLE AT LOW DENSITY Facility chiller / dry cool Standard colo plant Chilled water 6-12°C 10-100 MW rejection scale CRAH / fanwall Room-level air cooling Vertiv / Stulz / Schneider Standard hall infra Hot-aisle containment Delivered chilled air to rack front Hot exhaust captured at back Works to ~40 kW/rack Server fans / heat sinks Traditional forced-air heat sink 300-600 W per chip air-cooled DTC only at high-end (H100/B200 racks) Cooling flexibility Air at 20-40 kW/rack Rear-door hybrid at 40-80 kW DTC only 80+ kW/rack DIVERGENCE FROM TRAINING · Standard colo cooling plant works · DTC only at high density · existing brownfield thermal envelope usable · lower peak thermal per rack NETWORK FABRIC · NORTH-SOUTH DOMINANT INTRA-RACK Ethernet ToR switch 100/400 GbE per server Model fits within server INTER-RACK Leaf-spine Ethernet fabric Standard datacentre topology Load balancer / API gateway USER PEERING Multi-ISP peering Anycast BGP DDoS mitigation layer EAST-WEST (minor) Minimal cross-rack Model replication only No gradient sync STORAGE STACK · RESIDENT MODEL + KV CACHE MODEL WEIGHTS Resident in GPU HBM Loaded once at server boot Model registry: S3-style KV CACHE HBM + system memory tier Grows with active sessions Paged attention / prefix cache EMBEDDINGS Vector DB (SSD tier) Pinecone / Milvus / Weaviate / Qdrant RAG lookups per query MODEL REGISTRY Versioned model artifacts Distributed cache close to serving Small footprint per node
Inference DC grid-to-chip stack. Smaller MV service; simpler UPS (Tier III/IV redundancy focus); sidecar optional (traditional PDU + rack PSU path viable); heterogeneous silicon (GPU / ASIC / LPU / TPU / SoC); wider range of accelerator cost/performance points. Cooling: illustrative zones only (air-dominant below ~40 kW/rack; rear-door/hybrid ~40-80 kW; DTC above), boundaries depend on server config, inlet temp, allowable component temps and facility design. Network: request-plane traffic is north-south dominant for independent-server inference; distributed inference (large models + MoE + PD disaggregation) adds a high-bandwidth east-west compute plane for GPU-to-GPU and KV cache movement. Storage: model weights resident + KV cache + vector DB + model registry.

06Utilisation patterns

For this scenario, training runs flat at 90%+. Inference is spiky, tied to user demand cycles. The 90%+ training figure and the 40-60% average inference figure used through this section are scenario anchors for modelling. Treat them as illustrative.

Figure 6
Utilisation patterns (AUTHOR SCENARIO anchors). Training modelled flat 24/7. Inference is spiky.
AUTHOR SCENARIO · ILLUSTRATIVE UTILISATION PATTERNS 100% 80% 60% 40% 20% 0% Chip utilisation (%) Mon Tue Wed Thu Fri Sat Sun Mon Training (flat 90%+ target) Inference (spiky by hour + day) Chip utilisation vs time-of-week TRAINING PATTERN Constant 90%+ target. Downtime is a productivity loss (dollars per GPU-hour). Grid draw is flat. Cooling load is flat. Facility optimised for base load. INFERENCE PATTERN Peak-to-trough 3-5x typical. Peak evening (western daylight hours), trough overnight. Weekend variation. Average utilisation ~40-60% unless spillover to batch workloads.
For this scenario, training is modelled at 90%+ utilisation around the clock. Inference varies 3-5x peak-to-trough tied to user demand cycles, with weekend and diurnal variation. The utilisation delta is what drives asymmetric capex ROI: training capex earns fully utilised hours; inference capex runs at ~40-60% average utilisation unless spillover batch workloads fill the trough. Facility power and cooling design differs because of this. All percentages here are AUTHOR SCENARIO anchors for modelling.
RAMP-RATE PHYSICS · THE GRID INTERFACE CONSEQUENCE

The inference load profile is not merely non-flat; it is bursty at the second-to-minute scale. Model reload, batch triggers and coordinated request bursts produce di/dt events large enough to matter at the utility interface. Peak-to-trough ratios of 3:1 to 4:1 at facility scale are broadly consistent with published hyperscaler observations. Instantaneous ramps within a single interconnect billing cycle can be larger.

Facility-side mitigations (AUTHOR SCENARIO, not universal):

• Facility BESS or kinetic flywheels at the substation for peak shaving and ramp-rate limiting; sizing at ~10-50% of facility peak MW depending on utility contract terms.

• Rack-level BBU engineered for transient smoothing (sub-second capacitor + short-duration battery) rather than three-minute ride-through.

• Software-defined dynamic power capping: over-provision compute density, under-provision utility MW, and clip the top of the ramp in scheduler firmware.

• Utility contract structure: some interconnects prohibit certain ramp rates or charge demand penalties for peak spikes. Contract terms materially affect facility topology choice.

Step-load physics runs in different time domains. Silicon-level dynamic voltage and frequency scaling (DVFS) drops load from 100% to ~20% in microseconds, producing high-frequency ringing on the rack busbar. Token batch transitions run in milliseconds to seconds, producing low-frequency surges through the facility UPS and MV transformer. The mitigation stack has to match the time domain.

BBU chemistry matters. Standard lithium-ion (LFP / NMC) rack BBUs have moderate internal resistance and limited C-rates. They handle 2-3 minutes of ride-through, and are the wrong device class for high-cycle di/dt smoothing at millisecond timescales; sustained micro-cycling degrades them thermally. Millisecond-scale di/dt on the 800V DC bus wants a different device class: ultracapacitors or high-C-rate lithium titanate (LTO) cells integrated at the intermediate bus, sized for absorb-and-return rather than long-duration discharge. Facility-side, flywheels or BESS handle minute-scale ramp-rate compliance (MW / min under interconnect terms). Rack BBU plus intermediate LTO / ultracap plus facility BESS is a stack matched to distinct time domains, each device class doing what it is built for.

Partial PUE consequence at low load. Legacy chiller plants and cooling towers fall off their optimal efficiency curve when facility load drops to ~40% overnight. Facility partial PUE degrades unless variable-frequency drives (VFDs) and staged CDUs are specifically tuned for low-load turndown, or spillover batch workloads fill the trough. The 3-5x peak-to-trough utilisation profile therefore carries a facility-efficiency cost on top of the capex ROI drag on the compute itself.

Thermal loop inertia has a different time constant. When an inference cluster spikes from 20% to 100% load during a batch trigger or prime-time surge, GPU die temperatures move within milliseconds. The facility water loop responds in tens of seconds to minutes: fluid transit through hundreds of metres of secondary piping plus the heat capacity of the water mass together set the mechanical response time.

Two mechanical stress paths. Variable-speed pumps on the secondary CDU-to-rack loop actuate valves rapidly under dynamic load, producing pressure transients and water hammer in blind-mate quick disconnects. In low-density air / hybrid facilities (Q2 / Q3), severe diurnal cycling drives thermal expansion and contraction on cold-plate solder joints, dry-break couplings, and CDU heat exchangers. Failure rates accelerate compared to flat 24 / 7 training loads. Facility-level dynamic flow modulation, variable-ΔT loop control and CDU MTBF at inference-scale cycling are line items in a Q3 or Q4 diligence that don't appear in a Q1 training-DC diligence.

07Cooling architecture

GB200/GB300-class rack density (~120-140 kW) mandates DTC. Inference rack density often does not.

Figure 7
Cooling architecture by rack density. Air, hybrid, DTC zones and where training vs inference land
DTC + Immersion DTC required Hybrid (rear-door) Air-cooled viable AIR-COOLED VIABLE CRAH / fanwall HYBRID Rear-door heat exchanger DTC REQUIRED Cold plate on chip + row CDU 0 20 40 80 120 160 200+ Rack power density (kW/rack) INFERENCE typical range (20-100 kW/rack) TRAINING typical range (120-200 kW/rack) Efficient silicon (Groq, Cerebras, TPU v5e, SoC) stays here H100/L40S clusters can move here as density rises Training on GB200/Vera Rubin NVL72 AIR ZONE (0-40 kW/rack) CRAH / fanwall handles heat. No liquid infrastructure. Cheapest capex, most brownfield-friendly. HYBRID (40-80 kW/rack) Rear-door heat exchanger. Water in the rack door. Transitional path. USystems / Motivair / Boyd. DTC (80-200+ kW/rack) Cold plate on chip + row CDU + facility water plant. Full liquid stack. Training-DC standard.
Cooling follows rack density, not the word "inference". Typical zones: air-cooled works to roughly 40 kW/rack (depends on server config, inlet temp, rear-door design); hybrid rear-door in the roughly 40-80 kW band; DTC above. Actual thresholds vary by platform, workload duty cycle, allowable inlet temperature and facility conditions. Inference on efficient silicon (Groq LPU, Cerebras, TPU v5e, Qualcomm Cloud AI) can remain in the air-cooled zone at appropriate server/rack configurations; H100/L40S clusters can move into hybrid cooling as rack density rises; GB200/GB300 NVL72 mandates DTC regardless of whether the workload is training or inference. Density is the trigger. The workload label is the wrong variable.

08Networking + storage stack

Training needs high east-west bandwidth and parallel filesystem. Inference needs north-south egress and a resident model + KV cache stack.

Figure 8
Where inference stops looking like training: the network and storage stack
TRAINING · NETWORK + STORAGE East-west dominates. Parallel filesystem. NETWORK FABRIC Rack 1 NVL72 72 GPU NVLink Rack 2 NVL72 Rack 3 Rack 4 Rack 5 Rack 6 Rack 7 InfiniBand Spine (NDR/XDR fat-tree, 400-800 Gb/s) Model parallelism: gradient AllReduce → TB/s aggregate east-west STORAGE STACK Parallel filesystem Lustre / GPFS / WEKA / VAST 50-500 GB/s aggregate Petabyte-scale training data Checkpoint storage S3-compatible object 100s of TB per checkpoint Versioned; recovery Archival Cold tier / Glacier Model registry Historical versions TRAFFIC BREAKDOWN East-west (gradient AllReduce, activation exchange) N-S DESIGN IMPLICATIONS • Non-blocking fat-tree fabric. IB dominates. • Storage is a big-BW parallel filesystem, tightly coupled to compute. • Egress bandwidth to internet is minor. No user-facing peering pressure. • Vendors: NVIDIA Quantum IB / Cornelis, DDN, WEKA, VAST. INFERENCE · NETWORK + STORAGE North-south dominates. Resident model + KV cache. NETWORK FABRIC User requests (multi-ISP peering, anycast BGP) Load balancer + API gateway (L4/L7) Ethernet leaf-spine (100/400 GbE) Rack 1 1U servers Rack 2 Rack 3 Rack 4 Rack 5 Rack 6 n Independent servers. Minimal east-west. Model fits within server. STORAGE STACK Model weights HBM resident Loaded at boot KV cache HBM + sysmem Grows / session Vector DB Pinecone/Milvus SSD tier Model registry S3 versioned Rollback path TRAFFIC BREAKDOWN E-W North-south (user requests, model updates) DESIGN IMPLICATIONS • Standard leaf-spine Ethernet + multi-ISP peering (Cloudflare, Fastly, Akamai style). • Storage is HBM-resident model + KV cache; small SSD footprint. Vector DB is optional (RAG).
Training network is a non-blocking east-west fat-tree (InfiniBand NDR/XDR) with parallel filesystem storage. Inference has two network planes: a request plane (user to load balancer to server) and a compute plane (GPU-to-GPU + KV-cache movement + all-to-all expert routing for MoE). Small independent-server inference uses only the request plane. Large distributed inference (GB200 NVL72 serving 1T+ MoE, or disaggregated prefill/decode) uses both planes; the compute-plane fabric is rack-scale NVLink/NVSwitch within the NVL72 domain and RDMA-class Ethernet or InfiniBand between nodes. Storage is HBM-resident model + KV cache per session + vector DB for RAG.
TWO NETWORK PLANES FOR INFERENCE

Request plane. User request enters via multi-ISP peering + anycast BGP → load balancer → API gateway → inference server. Ethernet-based, standard datacentre topology.

Compute plane. Inside the inference cluster: GPU-to-GPU tensor exchange (for model-parallel inference), KV-cache movement (for disaggregated prefill/decode), all-to-all expert routing (for MoE). Rack-scale NVLink/NVSwitch inside the GB200/GB300 NVL72 domain; RDMA-class Ethernet or InfiniBand between nodes. A north-south-only model of inference misses this compute plane.

Small independent-server inference uses only the request plane. Large distributed inference (GB200 NVL72 serving 1T+ MoE, or PD-disaggregated deployments) uses both. Inference network requirements depend on the workload regime.

FABRIC DIVERGENCE · THE BIGGEST CAPEX SWING

Power and cooling look similar across training and rack-scale inference. The topology decouples on the fabric.

Training clusters demand full-bisection east-west bandwidth to prevent GPU stalling during all-reduce and pipeline exchange. Design point: non-blocking fat-tree, InfiniBand NDR / XDR or Ultra Ethernet, deep spine tiers, high optics count.

Purpose-built inference clusters (Q3) commonly operate on oversubscribed spine layers or pure scale-up pods with RoCEv2 egress, even when running MoE with all-to-all inside the rack-scale NVLink domain.

Cluster BOM implication (AUTHOR SCENARIO at ~1 MW cluster class, not benchmark): networking capex lands at roughly 20-25% of total cluster BOM for a non-blocking training fabric, and roughly 8-12% for a purpose-built inference fabric with oversubscription. The difference is largely optics count and switch-tier depth, and it changes the per-MW capex of a facility optimised for inference-only tenancy.

Intra-pod versus disaggregated. The oversubscription argument holds when tensor parallelism and MoE all-to-all stay inside a single rack-scale NVLink domain. It stops holding the moment prefill and decode are physically separated across rows or rooms. The KV-cache handover becomes an elephant flow on the critical path for TTFT, and the math in §02 (~830 ms for a 128k prompt of Llama-3.1-405B state over a 400 Gbps interface) shows why.

Disaggregation does not eliminate fabric capex. It swaps symmetric all-reduce bisection bandwidth for high-throughput point-to-point burst bandwidth. Oversubscribing the spine without priority-flow control (PFC) or coordinated packet reservation lets elephant flows collide, causing hash collisions and tail-latency blowups on decode workers. A cluster running disaggregated prefill/decode across nodes needs low-oversubscription (or non-blocking) links between the two pools, even if the same cluster tolerates oversubscription for other traffic. Fabric capex percentage lands between the two anchor cases depending on the disaggregation footprint.

The ECMP trap. Standard leaf-spine oversubscription routes flows using Equal-Cost Multi-Path (ECMP) hashing. Under disaggregated serving (Dynamo, DistServe, Splitwise), KV-cache transfers are elephant flows: ECMP hashes them into the same physical spine links, driving buffer exhaustion, packet drops, and tail-latency blowups on the decode worker.

Architectural countermeasures. Packet-sprayed RoCEv2 (or Ultra Ethernet Consortium standards) with dynamic load balancing breaks elephant flows across all available paths at packet granularity, avoiding the ECMP hash-collision failure mode. Alternative: rail-optimised point-to-point KV-transfer crossbars separated physically from the general request plane. Either approach bifurcates the network into a cheap commodity front-end (request plane) and a bespoke burst-optimised backend fabric (compute plane) with dedicated traffic engineering. Disaggregation therefore does not simply relax network capex; it splits the network into two distinct fabrics with different unit economics.

09Silicon mix

Training is concentrated on a relatively narrow set of high-end accelerator ecosystems. Inference has a broader hardware design space.

Figure 9
Silicon mix by regime
TRAINING SILICON Concentrated on a small set of training-accelerator ecosystems. NVIDIA (dominant) H100 / H200 B100 / B200 / GB200 Vera Rubin (2026-2027) AMD (challenger) MI300X / MI325X MI350 / MI400 Growing hyperscaler share AWS custom Trainium2 Trainium3 (roadmap) Anthropic / internal workloads Google TPU TPU v5p TPU v6 (Ironwood) Trillium/successor stack CHARACTERISTICS Common: dense HBM (192-288 GB), high memory bandwidth (5-8 TB/s), NVLink or vendor-fabric, 1000-2000 W TDP per chip, rack-scale platform integration. PROCUREMENT DYNAMICS • NVIDIA captures >80% of merchant training silicon. • AMD MI300X/MI325X growing at hyperscaler-driven pace. • AWS Trainium / Google TPU: internal-only, not merchant sold. • Supply chain concentration: TSMC advanced node, SK Hynix / Micron / Samsung HBM. • Rack platform integration: NVIDIA reference designs + ODM (Wistron, Foxconn, Quanta, Supermicro). Silicon $ per rack: $3-5M for GB200 NVL72 class. VALUE CAPTURE Silicon vendor (NVIDIA / AMD): 40-50% of rack $ · HBM (SK Hynix / Micron): 15-20% Substrate + packaging (TSMC CoWoS): 5-10% · ODM (Wistron/Foxconn): 5-8% Power delivery + interconnect: 8-12% · Cooling: 5-8% · Enclosure + integration: 3-5% Author illustrative allocation for GB200 class. INFERENCE SILICON Heterogeneous. Many vendors, many workload fits. GPU (general) L40S / L4 H100 PCIe / H200 Custom ASIC (LPU) Groq LPU SambaNova SN40L Wafer-scale Cerebras WSE-3 CS-3 system Hyperscaler ASIC AWS Inferentia2 Google TPU v5e Edge / SoC Qualcomm Cloud AI 100 Apple / Meta / MSFT custom Emerging Tenstorrent FuriosaAI CHARACTERISTICS Diverse: memory topology varies, TDP 100-1000 W, PCIe or vendor fabric, workload-specific optimisation (throughput / latency / batch-size / model-size fit). PROCUREMENT DYNAMICS • No single-vendor dominance. Fit-to-workload matters more than raw performance. • NVIDIA GPU still large share (existing developer stack, CUDA lock-in). • Custom ASIC growth conditional on model architecture stability + volume threshold. • Hyperscaler internal ASIC: AWS Inferentia2, Google TPU v5e already substantial share of internal inference. • Edge / SoC: Apple Intelligence, Meta AR/VR, Qualcomm-in-device inference are separate markets. Silicon $ per rack: 5-10x below training racks. Supply chain: less TSMC-N3/N2 concentrated; wider fab node distribution. VALUE CAPTURE (illustrative) Silicon vendor: 25-40% of rack $ (depends on chip choice) Memory (HBM or DDR-based): 10-20% · System integrator (Dell/HPE/Supermicro): 15-25% Networking (ToR + peering): 8-12% · Power (PDU + PSU): 5-8% · Cooling (air-scale): 3-5% More even distribution than training. Multiple viable BOMs.
Training silicon is concentrated around a small set of high-end accelerator ecosystems (NVIDIA GB200/GB300, AMD MI300X/MI325X, AWS Trainium, Google TPU) with TSMC advanced node + HBM supply-chain concentration. Inference silicon has a broader design space because the optimisation target shifts from maximum training throughput toward cost/token, latency and memory bandwidth. NVIDIA and AMD span dual-use accelerators; Google TPU v5e and AWS Inferentia2 are examples of hyperscaler-specific inference silicon; specialist architectures (Groq LPU, Cerebras WSE, Meta MTIA and others) widen the design space further. The wider vendor and fab-node distribution reflects that inference workloads accept more architectural variance than training.
PHYSICAL ENVELOPE VARIES ACROSS SILICON FAMILIES

Different silicon families sit at different points in the power / cooling / interconnect envelope. A facility optimised for one footprint carries stranding risk when the tenant workload rotates through others.

• NVIDIA GB200 / GB300 NVL72: 72-GPU rack-scale liquid-cooled platform, ~120-140 kW / rack, 100% direct-to-chip liquid cooling, sidecar / 800V DC power delivery, InfiniBand or Ultra Ethernet scale-out.

• Google TPU v5e / v6e: pod-scale designs with different air / liquid hybrid arrangements and different bus architectures; rack densities and cooling loops are Google-internal.

• AWS Trainium2 / Inferentia2: hyperscaler-internal rack designs with their own bus and cooling architectures, optimised for AWS-specific workloads.

• Groq LPU and other SRAM-heavy inference engines: different thermal dissipation curves and step-power profiles; typically lower rack densities than rack-scale GPU platforms.

Underwriting risk. A colo tuned exclusively to a single vendor rack footprint (bespoke 120 kW liquid loop, bespoke busbar) carries architectural-bet risk over a 10-15 year lease as tenant workloads rotate through ASIC generations. The IC-memo question is whether the physical envelope can absorb rack-scale NVLink AND hybrid air/liquid TPU-class AND SRAM-driven inference silicon within its design life.

10Power chain implications

Training-scale rack density pulls sidecar / 800V DC architecture forward. Inference at moderate density stays on traditional PDU + rack PSU longer.

Figure 10
Power chain implications. Sidecar / 800V DC pull vs traditional PDU longevity.
POWER CHAIN LAYER TRAINING DC INFERENCE DC MV service size 100-500 MW site. Utility HV feed (115-345 kV). 10-100 MW site. Metro utility MV (13.8-34.5 kV). UPS / BESS Facility BESS 5-15 min. Grid smoothing role. 40 MW+ site. Traditional UPS + Li-ion BBU. Tier III/IV redundancy focus (SLA-driven). Rack-adjacent power Sidecar rack (660 kW class). ±400V DC / 800V DC busbar to compute. Traditional rack PDU 208-415 VAC. Sidecar optional (only above 60-80 kW/rack). In-rack conversion Rack DC/DC (1RU 90 kW class) OR on-tray DC/DC. 800V → 50V → chip. In-server PSU (12V or 48V bus). Traditional server PSU vendors. BBU / ride-through Shelf-integrated BBU (60-90 sec). Facility BESS handles longer duration. Facility UPS handles all ride-through. Rack has no dedicated BBU shelf. Redundancy tier Typically Tier II/III. Interruption tolerable (training restarts from checkpoint). Tier III/IV. User-facing SLA drives higher uptime (99.99%+). Power vendors Delta / LITEON / Chicony (sidecar) · Vicor / MPS / Navitas / EPC (rack DC-DC) Vertiv / Schneider / Eaton (UPS + PDU) · server ODMs handle in-server PSU 800V DC relevance Central. OCP Diablo 400 / Meta / Google / MSFT are pulling this in. Peripheral. Relevant only at high-density inference (H100/L40S dense racks). SUPPLIER IMPLICATIONS • Training DC pulls sidecar / 800V DC / OCP Diablo forward. Delta / LITEON / Chicony scope grows. • Inference DC keeps Vertiv / Schneider / Eaton scope healthy on traditional UPS + PDU. • Rack DC/DC (Vicor / MPS / Navitas / EPC) growth is training-DC-driven; inference growth is smaller. • Server PSU vendors (Chicony, Delta, LITE-ON, FSP) retain relevance in inference-only DCs.
Moderate-density inference can extend the commercial life of conventional PDU + rack-PSU architectures. High-density inference can pull the same 800V / rack-scale power architecture being developed for training. The same UPS, PDU, PSU, DC-DC and electrical vendors can serve both regimes; architecture and rack density determine the particular product mix. Traditional UPS/PDU vendors retain scope at moderate density; sidecar and 800V DC architectures become increasingly relevant as rack densities move into the ~100 kW-and-above range, whether training or inference.

11Grid connection and geography

Training generally has much greater geographic flexibility than strict-SLA inference and can trade location for cheap power and capacity.

Figure 11
Geography: cheap power vs user proximity
TRAINING GEOGRAPHY Chases cheap 24/7 power. Latency to user is not a driver. POWER-CHEAP HUBS (US) West Texas / Permian natural gas + wind + solar Meta, Crusoe, xAI Northern Virginia nuclear + PJM grid AWS, Microsoft, Meta Pacific Northwest hydro power MSFT Quincy, Google Dalles Iowa / Nebraska wind + coal POWER-CHEAP HUBS (INTERNATIONAL) Iceland / Norway geothermal, hydro Verne / Green Mountain Middle East (UAE, Saudi) sovereign compute G42, HUMAIN Malaysia / Singapore APAC hub, LNG grid AirTrunk, YTL, ByteDance Ireland / Nordics EU renewable + interconnect MSFT Ireland, DK / SE hubs SELECTION CRITERIA • Power availability (100-500 MW site scale, 24/7 base load) • Power price ($30-70/MWh typical target) • Water rights (for facility cooling plant) • Interconnect fiber (dark fiber path to hyperscaler backbone) • Land cost (100+ acre sites needed) • Permitting speed (varies by jurisdiction) • Tax incentives (state/local abatements) Latency to user population: not a primary driver. Anything within 200ms of major peering hubs works. INFERENCE GEOGRAPHY Chases metro proximity. Latency SLA is the primary driver. METRO EDGE HUBS (US) New York 32M metro 111 8th, 60 Hudson Los Angeles 18M metro One Wilshire, Equinix Chicago 10M metro 350 Cermak Dallas 7M metro Infomart Atlanta 6M metro METRO EDGE HUBS (INTERNATIONAL) London 14M metro Slough, Docklands Frankfurt DE-CIX hub 3M metro Tokyo 37M metro Otemachi Singapore SEA hub 6M metro Mumbai 21M metro SELECTION CRITERIA • Latency envelope to user population (~20-100ms round-trip) • Metro population size (weighted by AI query concentration) • Peering hub access (IX presence, dark fiber to backbone) • Existing colo footprint (10-100 MW sites, not greenfield 500 MW) • Regulatory jurisdiction (data residency, sovereign compute) • Existing hyperscaler edge zones (AWS Local Zones, Azure Edge) • Power price is a secondary consideration Latency to user population: primary driver. Colo operators (Equinix, Digital Realty, CoreSite) are natural landlords.
Training DCs can trade geography for cheap 24/7 power and capacity (representative regions: US Sun Belt, Nordics, Middle East). Strict-SLA inference has less freedom to make that trade, pushing some deployments toward major metros and regional hubs. The resulting site requirements can pull the two workloads toward different real-estate and supplier ecosystems.
LATENCY BUDGET DECOMPOSITION · WHY GEOGRAPHY BINDS SOME REGIMES

User-observable interactive-inference latency has three components: network transit (round trip between user and facility), model inference time (TTFT for the first token; ITL for each subsequent token), and application / serving stack overhead.

Formal decomposition as inequality constraints against SLA:

TTFT = t_propagation + t_queue + t_prefill + t_KV_transfer ≤ SLA_TTFT
ITL  = t_propagation + t_decode ≤ SLA_ITL

Fibre reality check. Silica glass has refractive index n ≈ 1.468, so signal propagation is c/n ≈ 204 km/ms: about 4.9 µs per km one-way, or ~10 µs per km round-trip. Fibre routing is rarely a straight line; a typical optical tortuosity factor of 1.3-1.4× means an 800 km metro radius carries roughly 10.5-11.5 ms of unpadded transport RTT before any router hop, transport-layer overhead or peering asymmetry. That is already above 20% of a 50 ms ITL budget before compute runs.

Interactive worked example (AUTHOR SCENARIO). Consider a 50 ms ITL target with roughly tens of tokens generated per second. If the network round-trip budget is capped at ~10-20 ms for consistent user experience, the fibre distance between user and facility falls to roughly 500-800 km at the speed-of-light limit alone. That range is where the "strict-SLA" side of the matrix lives. Metro and regional hub geography follows from the ITL budget, not from a workload preference.

Batch and asynchronous reasoning. Multi-second or minute-scale response budgets remove the geography lock entirely. A 5-minute deep-search agent can run on a remote 500 MW campus in Wyoming or Scandinavia at negligible latency cost. That is where the "relaxed-SLA" side of the matrix lives, and why the same underlying model can be deployed on two entirely different physical stacks depending on how the SLA is written.

12Cost per query decomposition

The cost stack shifts across model size and hardware choice.

Figure 12
Cost per query decomposition. What drives $/1M output tokens across model size.
AUTHOR SCENARIO · ILLUSTRATIVE COST DECOMPOSITION Cost per 1M output tokens ($) $100 $50 $20 $10 $5 $1 SMALL (7B) Llama 3.1 8B $4-6 MEDIUM (70B) Llama 3.1 70B, Claude Sonnet class $20-30 LARGE (405B) Llama 3.1 405B, GPT-4 class $60-80 1T+ MoE GPT-4 successor, Claude Opus class $70-100 EDGE SoC Apple Intelligence, on-device Marginal cost ~ $0 CUSTOM ASIC Groq LPU, TPU v5e for 70B $10-14 COST STACK Compute (GPU/ASIC) Memory + KV cache Networking + egress Power + cooling Facility opex READING THE COST STACK Compute typically dominates but not always. Memory + KV cache share grows with context length. Networking + egress can be material in metro DCs (peering costs). Actual mix depends on utilisation, batch size and quantisation. Custom ASIC (Groq, TPU v5e) collapses compute cost at fit workloads. Edge SoC (Apple, Meta) collapses marginal cost to near zero. Large MoE models get expensive fast. Author scenario. Actual $/1M tokens varies by hardware, batch size, quantisation, context length, and vendor pricing.
Cost per query decomposes across compute, memory, networking, power and facility opex. The bars above are AUTHOR SCENARIO ranges. Treat them as illustrative decomposition, not benchmark numbers. Actual $/1M tokens depends on: input/output token ratio, model, quantisation, batch size, utilisation, accelerator price + depreciation period, electricity, cooling, networking, SLA, software stack, and whether the source is provider-API pricing or infrastructure economics. What holds across the decomposition: compute typically dominates, memory + KV cache share grows with context length, and custom ASICs (Groq LPU, Cerebras, TPU v5e, Meta MTIA, AWS Inferentia2) can collapse compute cost dramatically for fit workloads.

13Four inference regimes

For the scenario matrix, five variables are reduced to two dominant axes: model regime × latency SLA. The other three (utilisation, geography, compute coupling) determine the physical intensity within each quadrant.

Figure 13
Four scenarios for the training/inference split. Model regime × latency SLA.
← RELAXED SLA (batch, agent, offline) STRICT SLA (<200ms user-facing) → ← MODELS PLATEAU (efficient, distilled) MODELS KEEP GROWING (1T+, MoE) → Q1 · TOP-LEFT · CURRENT DEFAULT Consolidated. Training + inference share DC. Models plateau (efficient 7B-70B). SLA is relaxed (batch, agent, offline). Training-DC infrastructure serves both. Delta / LITEON / Chicony scope grows; sidecar / 800V DC continues. Utilisation split: training 24/7, inference fills troughs. Today most inference runs on training-DC infrastructure. This is the world hyperscalers underwrote in 2022-2026. DEFAULT Q2 · TOP-RIGHT · METRO EDGE FLEET Metro edge inference (small models, tight SLA). Efficient 7B-70B models resident on metro colo footprints. Strict <200ms SLA (chat, voice, real-time apps). 10-50 metros. 1-20 MW each. Traditional PDU + air cooling. Colo landlords (Equinix, Digital Realty) capture value. Apple Intelligence, Cloudflare Workers AI, Meta AR/VR direction. CDN-like architecture for AI inference. GROWS Q3 · BOTTOM-LEFT · PURPOSE-BUILT INFERENCE Purpose-built inference DCs (batch/agent). Big models (1T+, MoE) need specialised inference accelerators. SLA relaxed (agent workflows, batch inference, offline). Standalone DCs near cheap power. Groq / Cerebras / TPU v5e at scale. Purpose-built cooling / power / network. Emerges if model complexity grows faster than efficient inference silicon. Anthropic / OpenAI agent workloads → this quadrant. EMERGES Q4 · BOTTOM-RIGHT · METRO GIANT DCs Metro giant inference DCs (big models, tight SLA). Big models require GB200-class silicon AND tight SLA. 200-500 MW metro DCs. Challenging real estate + power. Retrofit hyperscaler edge zones with dense GPU capacity. High cost per query but latency SLA justifies it. The regime carrying the highest combined infrastructure constraints. Metro power sourcing is the bottleneck. OpenAI / Anthropic / xAI metro compute footprints trend here. SELECTIVE
Four regimes on two axes: latency SLA × model regime. Q1 (plateau + relaxed) is the current default. Consolidated training + inference DCs. Q2 (plateau + strict) is metro edge fleet. CDN-like architecture for AI. Q3 (grow + relaxed) is purpose-built inference DCs for big-model batch / agent workloads. Q4 (grow + strict) is metro giant DCs.

14Detailed topology per scenario

The physical stack for each quadrant of Figure 13.

Figure 14a
Consolidated. Training + inference share the same DC. Current default. (Q1)
Q1 · MODELS PLATEAU + RELAXED SLA Consolidated training + inference in the same DC. Current default. Training capex justifies inference tenancy. AUTHOR SCENARIO not market benchmark PHYSICAL TOPOLOGY · SHARED GRID-TO-CHIP STACK FACILITY 100-500 MW site Cheap-power geo TX / VA / PNW Utility HV feed 115-345 kV Chosen for training economics SIDECAR + BESS + UPS Sidecar rack 660 kW Delta / LITEON / Chicony ±400V DC / 800V DC bus Facility BESS 5-15 min Grid smoothing role Full training-DC power chain applies RACK-INTEGRATED PLATFORM GB200 NVL72 racks (~120-140 kW each) 72 GPU per rack · NVLink + IB fabric DTC cooling · row CDU (2.4 MW L2L) DUAL-MODE USE: Training runs 24/7 flat Inference fills utilisation troughs via workload scheduling (Kubernetes / Slurm) STORAGE STACK Parallel filesystem Lustre / GPFS / WEKA / VAST Checkpoint S3 Model versioning + rollback Inference KV cache HBM + system memory tier Model registry (shared) Serves both modes NETWORK FABRIC InfiniBand NDR/XDR Non-blocking east-west NVLink 5 intra-rack 1.8 TB/s per GPU North-south egress Minor bandwidth Standard hyperscaler backbone COOLING + UTILISATION COOLING Full DTC + facility CDU 120-200 kW/rack training density mandates it. Row CDU + row manifold + cold plate. Delta EVA / CoolIT-Ecolab / Motivair / Boyd UTILISATION Training 24/7 base + inference spillover Inference queued during training-slow windows. Scheduler priority: training first, inference on capacity. Effective utilisation approaches 95%+. CAPEX ECONOMICS Training capex earns full utilisation Inference marginal cost is low (already-paid-for capex). The economics work when workloads sustain high utilisation. This is the current market equilibrium.
Q1 Consolidated. Training + inference share the same DC. Full training-DC stack applies: sidecar / 800V DC, DTC cooling, IB fabric, parallel filesystem, power-cheap geography. Inference workloads fill utilisation troughs via scheduler priority. The economics work when workloads sustain high utilisation. Current default for hyperscaler AI infrastructure 2022-2026.
Figure 14b
Metro edge fleet. Small models on colo racks under strict SLA. (Q2) · ILLUSTRATIVE SCENARIO
Q2 · MODELS PLATEAU + STRICT SLA Metro edge fleet. Efficient 7B-70B models on many small colo-scale DCs across major metros. CDN-like architecture. AUTHOR SCENARIO not market benchmark PHYSICAL TOPOLOGY · DISTRIBUTED METRO FLEET NEW YORK METRO 5-20 MW site Equinix / Digital Realty 111 8th, 60 Hudson LOS ANGELES METRO 5-20 MW site One Wilshire, Equinix CHICAGO METRO 5-15 MW site 350 Cermak LONDON METRO 5-15 MW site Slough / Docklands TOKYO / SGP / FRA 5-15 MW each Otemachi, DE-CIX 10-50 METROS Total footprint 100-500 MW aggregated Model updates central SINGLE METRO SITE STACK · TYPICAL 5-15 MW METRO POWER Utility MV 13.8 kV 10-25 MVA service Colo landlord provides Metro grid pricing $60-120/MWh typical POWER CHAIN (Tier III/IV) Traditional UPS + BBU Redundant N+1 or 2N 15-30 min ride-through Vertiv / Schneider / Eaton SLA-driven redundancy EFFICIENT INFERENCE PLATFORM Small models (7B-70B) resident on: NVIDIA L40S / L4 (mid-range GPU) Groq LPU (compact form factor) Qualcomm Cloud AI 100 Rack density 20-40 kW · air-cooled · 1U/2U servers USER PEERING Multi-ISP peering Cloudflare, Fastly, Akamai style Anycast BGP Nearest metro serves query IX presence critical DE-CIX, LINX, AMS-IX, NYIIX MODEL DISTRIBUTION Central training DC Model training happens here Model registry sync Weights propagate to all metros Rolling update pattern CDN-like model refresh COOLING + UTILISATION + METRO ECONOMICS COOLING (low-density air) CRAH / fanwall standard 20-40 kW/rack — well within air envelope. Existing colo hall infrastructure works. Brownfield colo footprint retrofits directly. UTILISATION (spiky, 40-60% avg) User-demand tied. Peak evening. Per-metro peak-to-trough 3-5x. Aggregated across time zones smooths some. 40-60% average utilisation. METRO ECONOMICS Higher power price. Colo lease cost. $/query higher than Q1 or Q2 but SLA justifies. Colo landlords capture material value. Equinix / Digital Realty / CoreSite / Iron Mountain. SUPPLIER IMPLICATIONS Q2 • Colo landlords (Equinix, Digital Realty, CoreSite, Iron Mountain) are the primary beneficiaries at metro-DC layer • Efficient inference silicon (Groq LPU, NVIDIA L40S, Qualcomm Cloud AI) captures rack-scale procurement volume • Networking + peering vendors (Cloudflare, Fastly, DE-CIX, LINX operators) grow with metro-edge inference expansion
Q2 Metro edge fleet. 10-50 small colo-scale DCs across major metros and regional hubs. 5-20 MW per site. Efficient 7B-70B models resident on mid-range GPU / Groq LPU / Qualcomm Cloud AI at 20-40 kW/rack. Air-cooled. Traditional Tier III/IV UPS + PDU. Multi-ISP peering with anycast BGP for lowest-latency serving. Model updates propagate from central training DC. Colo landlords are one obvious beneficiary of this regime, though the actual value distribution depends on which vendors capture the metro-fleet buildout.
Figure 14c
Purpose-built inference DCs. Big models on cheap-power geography with relaxed SLA. (Q3) · ILLUSTRATIVE SCENARIO
Q3 · MODELS GROW + RELAXED SLA Purpose-built inference DCs. Specialised silicon (LPU, wafer-scale, TPU v5e) at cheap-power sites for batch / agent workloads. AUTHOR SCENARIO not market benchmark PHYSICAL TOPOLOGY · INFERENCE-OPTIMISED STACK FACILITY 50-200 MW site Cheap-power geo (same as training) Utility MV feed Smaller footprint Standalone site not colocated w/ training POWER CHAIN (SIMPLIFIED) Traditional PDU or simpler sidecar 208-415 VAC + PSU UPS: Tier II/III No BESS grid-smoothing Vertiv / Schneider / Eaton scope. Simpler than training. INFERENCE PLATFORM Custom silicon at scale Groq LPU nodes SambaNova SN40L Cerebras CS-3 wafer Google TPU v5e at scale AWS Inferentia2 fleet Batch inference optimised Rack density 40-100 kW STORAGE STACK Model weights (HBM) Loaded at node boot KV cache tier HBM + NVMe overflow Batch job storage S3 for input / output artefacts Model registry Cross-node cache NETWORK FABRIC Ethernet leaf-spine Standard datacenter fabric 200-400 GbE Cisco / Arista / Nokia Batch queue ingest API gateway; relaxed latency No IB required (no east-west) COOLING + UTILISATION + BATCH ECONOMICS COOLING (mid-density) Air or hybrid (rear-door) 40-100 kW/rack typical for LPU / TPU / Inferentia platforms. CoolIT rear-door units, Vertiv rack-level, USystems ColdLogik. DTC not required unless H100-class inference density. UTILISATION Batch queue driven, 70-90% Agent workflows fill queue efficiently. Batch inference during off-peak hours. Higher utilisation than metro-edge inference DCs. CAPEX ECONOMICS Silicon $ ~5x lower than training rack Cost per query 2-3x below GB200 for fit workloads. Economic case strongest for stable, batchable, highly-utilised workloads. Anthropic / OpenAI agent workloads → this quadrant.
Q3 Purpose-built inference DCs. Cheap-power geography (same as training), but simpler power chain (traditional PDU + rack PSU), no InfiniBand (Ethernet leaf-spine), air / hybrid cooling, specialised inference silicon (Groq LPU / Cerebras / TPU v5e / Inferentia2). The economic case strengthens when workloads are stable, batchable and highly utilisable. Utilisation 70-90% via batch queue. This is the awkward case. If latency thresholds relax faster than model sizes shrink, Q3 collapses back into Q1 and there is no independent reason to build it as its own DC type.
Figure 14d
Metro giant. Big models under strict SLA. The regime least likely to scale on economics alone. (Q4) · ILLUSTRATIVE SCENARIO
Q4 · MODELS GROW + STRICT SLA Metro giant DCs. Big models require GB200-class silicon AND user-facing SLA drives metro proximity. Economically hardest. AUTHOR SCENARIO not market benchmark PHYSICAL TOPOLOGY · METRO-PROXIMATE HIGH-DENSITY DC METRO FACILITY 200-500 MW Metro-proximate Latency <100ms to users Utility HV feed 115-230 kV (metro) POWER SOURCING IS THE BOTTLENECK FULL SIDECAR + BESS Sidecar rack 660 kW class Delta / LITEON / Chicony ±400V DC / 800V DC bus Facility BESS 15+ min Grid participation likely Full training-DC power chain applies, in metro GB200 / RUBIN PLATFORM (in metro) Rack-integrated dense GPU GB200 NVL72 (~120-140 kW/rack) Vera Rubin NVL144 (250 kW/rack) 72-144 GPU per rack DTC cooling mandated Same rack platform as training, deployed in metro real estate STORAGE Model weights (HBM) 1T+ MoE fits across NVL72 KV cache tier Deep per-session cache growth Vector DB RAG lookups per query Model registry (metro-local) Update pushed from central NETWORK FABRIC Multi-ISP peering Anycast BGP + heavy peering Ethernet leaf-spine 400 GbE + 800 GbE spines InfiniBand (partial) If big-model shards span racks Hybrid fabric: N-S + partial E-W COOLING + UTILISATION + WHY THIS IS HARD COOLING (full DTC) DTC required at 100-250 kW/rack Same cooling stack as training DC. Facility water plant + row CDU + cold plate. Metro water rights become a constraint. UTILISATION (spiky, 60-75% avg) User-driven with agent overflow Big-model inference peaks at prime time. Overnight batch fills some trough. 60-75% average, better than pure Q3. WHY Q4 IS ECONOMICALLY HARDEST Metro real estate + 200-500 MW power Interconnect + zoning + water permits are metro-tight. Illustrative scenario: $/query ~3-5x above Q1 for big-model inference. Only user-facing SLA + high query value justifies.
Q4 Metro giant DCs. GB200-class silicon deployed in metro real estate for tight-SLA big-model inference. Same power chain (sidecar + 800V DC) and cooling stack (full DTC) as Q1 training DCs, but constrained by metro power sourcing, water rights, and land availability. Hybrid network (Ethernet peering + partial InfiniBand). Illustrative scenario: $/query ~3-5x above Q1. Economically defensible when user-facing SLA and query value justify the premium. Technically feasible; the regime least likely to scale on economics alone. Q4 is the one I would not underwrite from latency alone.
Q4 · UTILITY QUEUE vs MODEL HALF-LIFE

Q4's binding constraint is timing. Even when the physical case works, the utility-interconnect and permitting lead time is a first-order underwriting problem.

Interconnect lead time. A 200-500 MW facility in a Tier-1 metro (Northern Virginia, Frankfurt, Slough, Tokyo, Singapore) requires a 115-230 kV transmission interconnect. Current utility queues in these zones commonly run 5-8 years to substation energisation, before environmental permitting, water-rights acquisition and facility construction.

Model architecture half-life. Frontier model distillation and custom ASIC architectures have typically evolved on an 18-24 month cadence. Over a 5-8 year utility-interconnect window, that translates to two to three architecture generations of change. Compression, distillation, sparsity, MoE routing efficiency and edge silicon capability each reduce the memory and compute footprint per unit intelligence delivered.

The underwriting mismatch. A 300 MW metro giant sited in 2026 and energised in 2032 will serve a model architecture that likely does not exist yet, and may not need the physical envelope the site was designed around. Q4 therefore combines the highest capex, the longest lead time, and the shortest useful architectural life. The lead time of metro utility interconnection exceeds the architectural half-life of frontier model footprints. That is why the Q4 case cannot be built from latency alone.

15Stranded architecture. The infrastructure-risk view.

Infrastructure risk here is a match problem. Does today's physical architecture serve tomorrow's inference regime, or has it been built for the wrong workload. Build an ~150 kW-class 800V rack + liquid facility (as a scenario) for a workload that later turns out to be low-density latency-sensitive inference, and you have stranded capex. Build metro air-cooled colo for a workload that turns out to require frontier reasoning at 100+ kW/rack, and you have the reverse. The risk quadrant below charts model-efficiency progress against silicon-efficiency progress; each quadrant produces a different stranded-capex profile.

Figure 15
Risk quadrant. What could displace purpose-built inference DCs. Model efficiency × silicon efficiency.
← SLOW MODEL EFFICIENCY (models keep growing) FAST MODEL EFFICIENCY (distillation, sparsity, pruning) → ← SLOW SILICON EFFICIENCY FAST SILICON EFFICIENCY (custom ASIC, in-memory compute) → Q1 · TOP-LEFT · PURPOSE-BUILT INFERENCE DOMINATES Both improvement axes are slow. Big models keep growing. Silicon improves incrementally. Cost per query stays high. Batch inference DCs justify capex. Groq / Cerebras / TPU v5e scale up as dedicated fleet. Metro giant DCs (Q4) also grow to serve tight-SLA workloads. Current thesis path. Sidecar / 800V DC / rack-level infrastructure wins. CURRENT Q2 · TOP-RIGHT · MODEL EFFICIENCY WINS Efficient models on general-purpose silicon. Distillation, sparsity, MoE routing dramatically shrink model size. 70B-class models replace 405B-class for most workloads. Inference fits on general GPU / consumer hardware. Metro edge fleet (Q3 above) scales. Purpose-built LPU less needed. DeepSeek-style efficiency + Llama 4/5 open-model race → this quadrant. CONSUMER Q3 · BOTTOM-LEFT · CUSTOM ASIC SPECIALISATION Custom silicon wins for stable-model workloads. Models grow BUT custom ASIC delivers 5-10x cost/query improvement. Groq LPU / Cerebras / hyperscaler-internal chips dominate inference. NVIDIA/AMD share of inference silicon shrinks. Purpose-built inference DCs get built around custom silicon. In-memory compute (Groq TSP / Cerebras WSE) architecture wins. CUSTOM Q4 · BOTTOM-RIGHT · BOTH ACCELERATE Inference converges with training / edge. Both efficiency axes race ahead. Model size flattens; silicon 10x cheaper. Inference converges toward consolidated training-DC use (Q1 above). Or moves to edge (Apple / Meta SoC on-device). Purpose-built inference DC as a category loses structural rationale. The dark case for inference-DC infrastructure vendors. ELIMINATE
Risk quadrant for purpose-built inference DCs. Two axes: model efficiency (algorithmic progress) × silicon efficiency (chip progress). Q1 both slow. Purpose-built inference DCs dominate. Q2 model efficiency wins. Inference fits on general silicon and moves to consumer / metro edge. Q3 custom silicon wins. Inference DC design becomes ASIC-specialised. Q4 both accelerate. Inference converges toward consolidated training-DC use or edge deployment; purpose-built inference-DC category loses rationale.
STRANDED-CAPEX SCENARIOS

~150 kW-class 800V rack + liquid facility (scenario) → low-density latency-sensitive inference. Infrastructure optimised for the wrong workload. Rack-scale power chain and liquid loop stay stranded at low utilisation. Geographic mismatch to user proximity.

Metro air-cooled colo → frontier reasoning inference. Density mismatch: a 40 kW/rack air envelope cannot accommodate a full GB300 NVL72-class rack. Retrofit to DTC requires a facility cooling architecture capable of rejecting the added liquid heat load, which metro colo sites may not have.

Purpose-built inference DC on Groq LPU → model architecture shifts away from LPU-fit workloads. ASIC lock-in strands the silicon investment. Sidecar / cooling / facility remain reusable but the semiconductor content depreciates fast.

Cheap-power hyperscale inference DC → sovereign / metro compute mandates. Regulatory shift to data residency reduces the addressable workload. Cheap-power siting no longer optimal.

16What would falsify this framework

There are five ways I could be wrong.

FALSIFIERS

1. Inference remains overwhelmingly centralised. If frontier models continue growing and latency SLAs stay relaxed enough that no metro deployment is required, Consolidation stays the answer and the other regimes never scale. Test: track OpenAI / Anthropic / xAI inference footprint by geography over 12-24 months.

2. Model efficiency wins the race. If distillation, sparsity and MoE routing shrink frontier-model inference below what fits on a single node, purpose-built inference DC (Q3) loses its scale rationale. Test: track cost per query at a fixed quality / capability target across efficient (Llama, DeepSeek, Qwen) and frontier models.

3. Compute efficiency outruns demand. If per-token compute cost drops faster than query volume grows, inference infrastructure needs (in aggregate) plateau or shrink. The four regimes converge back toward Q1. Test: track $/1M tokens vs global inference query volume growth.

4. Serving software erases distributed-vs-centralised distinctions. If disaggregated serving (Dynamo, DistServe, Splitwise) makes heterogeneous distributed inference economically competitive with centralised systems, the geographic threshold weakens. The metro / regional / hyperscale continuum becomes a software problem, not an infrastructure one. Test: track inference deployments running on Dynamo-style disaggregated architectures across geographies.

5. Geography becomes economically irrelevant. If network latency, model compression, caching and serving optimisation reduce the economic value of physical proximity below the metro-DC premium, the metro/edge regimes lose their infrastructure rationale. Inference converges back to cheap-power geography. Test: track user-observable TTFT and inter-token latency for centralised vs metro-deployed serving of the same model at fixed cost/query.

17Value migration and supplier implications

If inference fragments geographically and architecturally, the value pool fragments with it. The AI infrastructure capex cycle is currently concentrated around training-scale accelerator infrastructure. As inference diversifies, the economic bottlenecks broaden outward to serving silicon, memory bandwidth, networking, orchestration, metro capacity, peering and power efficiency.

Figure 17
Where value migrates as inference matures
STAGE 1 · TRAINING-DOMINANT Single-template AI infrastructure Utility HV feed Training silicon (dense GPU) NVIDIA + hyperscaler chips Training-DC infrastructure Sidecar + DTC + IB + FS Training + inference (same DC) VENDOR CAPTURE Silicon: NVIDIA, AMD, hyperscalers Infrastructure: Delta, LITEON, Vertiv Real estate: hyperscaler + neocloud STAGE 2 · MIXED SERVING Consolidated (training + inference) Utility HV feed Training silicon (dual-mode) GB200 serves training AND inference Training-DC infrastructure Shared stack via scheduler Utilisation scheduler Training 24/7 + inference fills Serving users + training VENDOR CAPTURE Silicon: NVIDIA GB200 still dominant Infrastructure: sidecar / DTC / IB Real estate: cheap-power geo hubs STAGE 3 · REGIME-SPECIALISED SERVING Split infrastructure (per scenario) Training + inference facilities diverge Training DCs Cheap-power geo GB200 / Vera Rubin Sidecar + DTC Inference DCs (mix) Q2/Q3/Q4 mix Custom ASIC + metro Air/hybrid cooling Model production Training runs, checkpoints Serving users Batch, agent, metro edge NEW VENDOR CAPTURE Colo landlords (Equinix, Digital Realty) · Custom silicon (Groq, Cerebras, TPU v5e) Metro networking (Cloudflare, Fastly, DE-CIX) · Vector DB (Pinecone, Milvus) WHAT CONCENTRATES Silicon vendors (NVIDIA + AMD for training; heterogeneous for inference) Rack DC/DC (Vicor / MPS / Navitas / EPC) — survives all three stages Sidecar (Delta / LITEON / Chicony) — training-DC-anchored, still relevant WHAT EMERGES Inference-specific silicon (Groq, Cerebras, TPU v5e, Inferentia2) Metro DC operators (colo landlords + edge zone operators) Networking + peering + vector DB (Cloudflare, Pinecone, DE-CIX) Underwrite regime-matched architecture with room to cross regime boundaries.
Training-dominant: the AI infrastructure capex pool remains heavily concentrated in rack-scale accelerator infrastructure, advanced power and advanced cooling. Mixed serving: training silicon runs both workloads on shared racks; consolidation still wins in most cases. Regime-specialised: training concentrates in cheap-power geography; inference splits across metro edge (Q2), purpose-built (Q3) and metro giant (Q4). Rack DC/DC survives across the whole transition. Which vendor holds which layer, and when, is the underwriting question.

18Underwriting the inference DC

For infrastructure sponsors, colo landlords and IC memos, the framework above translates into specific underwriting profiles per regime. The matrix below summarises the target physical and economic metrics per regime; each column is an AUTHOR SCENARIO calibrated against the physical anchors elsewhere in this essay.

Figure 18
Underwriting sensitivity matrix. Target physical envelope, cooling stack, grid capacity factor and stranding risk per regime. AUTHOR SCENARIO for IC-memo diligence.
Underwriting sensitivity per regime AUTHOR SCENARIO · NOT MARKET BENCHMARK METRIC what the IC memo asks Q1 · TRAINING MEGA-CAMPUS Consolidated (training + inference) Q3 · PURPOSE-BUILT INFERENCE Cheap-power geography, batch / agent Q2 / Q4 · METRO EDGE / METRO GIANT Strict SLA, metro proximity TARGET POWER DENSITY Rack power envelope the facility floor, busbar and water plant must accommodate. ~120-140 kW / rack GB200 / GB300 NVL72 class Sidecar / 800V DC delivery ~40-80 kW / rack Hybrid DTC + air, mid-tier GPU / ASIC Traditional PDU still viable Q2: 20-50 kW / Q4: ~100 kW Brownfield colo envelope constrains Q2 Q4 pulls rack-scale into metro RE COOLING TOPOLOGY Whether the facility water plant can absorb the liquid heat load at density. 100% direct-to-chip liquid Facility water plant (FWS / W32) Row CDU + cold plate on ASIC + HBM Hybrid DTC + high-density CRAC Rear-door heat exchanger + row CDU Sized for specialised inference silicon Air retrofit / Hybrid CDU loops Q2: air with rear-door for hot racks Q4: DTC in metro RE (site-limited) GRID CAPACITY FACTOR Utility contract shape and ramp-rate tolerance the interconnect must accept. >90% steady baseload Flat 24/7 profile, low di/dt Interconnect sized for peak = average ~55-75% (diurnal swing) Batch queue fills troughs BESS + software power capping viable Variable / peak-following 3:1 to 4:1 peak-to-trough Demand-charge exposure real PUE SENSITIVITY How much opex is driven by facility overhead vs compute utilisation. Extreme Every 0.02 PUE = MM/yr at scale Free cooling + heat reuse mandatory Moderate Overhead matters, but utilisation dominates Cheap power greatly outweighs PUE gains Secondary Latency / availability outrank PUE Metro power price dominates opex PRIMARY STRANDING RISK What could leave the facility carrying infrastructure the workload no longer uses. Silicon obsolescence + delayed power availability Rack-scale bespoke tuning most exposed ASIC lock-in + overspecified networking or cooling Groq / Cerebras / TPU vendor churn risk Structural retrofit ceiling Floor loading, water plumbing, MV feed Brownfield inability to accept next density Match the physical envelope to the regime the tenant workload will occupy over the lease. All metrics AUTHOR SCENARIO for illustrative underwriting. Site-specific values require diligence at platform, contract and interconnect level.
Underwriting profile per regime. Q1 mega-campus targets rack-scale liquid infrastructure at cheap-power geography; the exposure is silicon obsolescence and power availability. Q3 purpose-built inference is hybrid-cooled with mid-tier density; the exposure is ASIC vendor churn and overspecified fabric. Q2 / Q4 metro is anchored at strict-SLA sites where the constraint is retrofit ceiling: floor loading, water plumbing and MV interconnect capacity that a brownfield colo may or may not have.
REGIME UNDERWRITING DECISION MATRIX · Q1 / Q2 / Q3 / Q4 DROP-IN

Per-regime dimensions for IC-memo integration. All values AUTHOR SCENARIO, calibrated against the physical anchors elsewhere in this essay.

DIMENSION Q1 · CONSOLIDATED Q2 · METRO EDGE Q3 · PURPOSE-BUILT Q4 · METRO GIANT
Typical accelerator GB200 / GB300 NVL72 L40S / H200 / custom ASIC B200 / MI350 / Groq LPU GB300 NVL72 / TPU v6e
Target rack density 120-140 kW / rack 15-35 kW / rack 40-80 kW / rack 100-140 kW / rack
Primary cooling topology 100% DTC liquid (W32 / FWS) Air (CRAC / CRAH) + rear-door Hybrid DTC / air 100% DTC liquid (chilled loop)
Network fabric architecture Non-blocking fat-tree (IB / UEC) Standard top-of-rack Ethernet Oversubscribed RoCEv2 (2:1 or 3:1) Low-oversubscription RoCEv2 / IB
Network capex (% cluster BOM) 20-25% <5% 8-12% 15-18%
Capacity factor / load profile 90%+ flat baseload 35-50% diurnal swing 50-70% (batch-padded) 40-60% diurnal peak
Power distribution 800V DC busbar + sidecar 415V AC / standard PDU 48V or 800V DC 800V DC busbar + sidecar
Primary stranding risk Stranded power if models shrink Density ceiling (cannot retrofit to DTC) ASIC / architecture obsolescence High $/MW build with weak metro peering

All values AUTHOR SCENARIO for illustrative underwriting. Site-specific values require diligence at platform, contract and interconnect level.

CAPACITY FACTOR · THE CAPEX AMORTISATION ARITHMETIC

The capacity-factor row of the matrix drives the deal-level economics of an inference DC. For an infrastructure sponsor underwriting an accelerator rack (roughly $3.5M for a GB200 NVL72, or roughly $500k for an inference ASIC rack) depreciated straight-line over a 3-4 year lifecycle, the capital cost per operational hour is fixed:

CapEx per hour = Total rack CapEx / (Years × 8760 × Capacity factor)

What the arithmetic does. At a 90% training capacity factor (~7,884 productive hours per year), amortisation runs efficiently. At a 45% inference capacity factor (~3,942 productive hours per year), the hourly capital charge literally doubles at the same rack cost. Values AUTHOR SCENARIO; the ratio is the point, and the absolute rack price is illustrative only.

Why Q3 operators chase spillover workload. The doubled hourly capital charge is what forces Q3 and Q4 operators to aggressively backfill diurnal troughs with un-checkpointed background workloads: synthetic data generation, model distillation runs, embedding pre-computation, feature-store rebuild. Without a synthetic baseload to fill the night-time trough, dedicated inference facilities carry a structurally depressed IRR relative to a training campus at the same rack cost.

Underwriting takeaway. The Q3 economic case stands on the operator's ability to source spillover workload. The physical envelope alone does not close the underwriting. Ask the sponsor: what fills the trough, and at what marginal revenue per productive GPU-hour?

IC-MEMO CHECKLIST BEFORE FUNDING AN INFERENCE DC

Regime targeting. Which regime does the facility target, and can it accommodate the adjacent regime without retrofit? Q1 to Q4 crossover (rack-scale into metro RE) and Q2 to Q3 crossover (edge fleet vs cheap-power hyperscale) are the two most common shifts.

Grid interconnect and ramp-rate agreement. Does the utility contract permit inference-scale di/dt without demand penalties or throttling? What is the ramp-rate limit written into the interconnect study? Contract terms materially affect facility topology choice.

Rack density envelope. Does the facility floor loading, water plumbing and power delivery envelope support ~120-140 kW rack-scale AND 40-80 kW hybrid AND 20-40 kW air-cooled tenants within its design life? Retrofit windows narrow fast once the shell is built.

Cooling architecture flexibility. Can the facility water plant absorb the added liquid heat load required to retrofit for direct-to-chip? Or is the plant sized only for air CRAC? Retrofit to DTC requires a facility cooling architecture capable of rejecting the liquid heat load, which metro colo sites may not have.

Silicon-agnostic vs bespoke. Is the physical envelope tuned to a single vendor footprint (NVIDIA rack-scale, TPU pod, Groq rack)? What is the stranding risk if tenant workload rotates through a different ASIC over a 10-15 year lease? Silicon-agnostic envelope is the default underwriting position; bespoke tuning is a concentrated architectural bet with a specific tenant thesis behind it.

Method + sources. Anchor references from OCP Open Rack v3 and Diablo 400 material for training rack specifications; NVIDIA GB200 NVL72 and Vera Rubin platform disclosures for chip-level TDP and interconnect; public disclosures on Groq LPU, Cerebras WSE, AWS Inferentia2, Google TPU v5e for inference-specific silicon; public case studies on hyperscaler AI infrastructure geography for grid-connection patterns.

Rack-density assumptions in Figure 3, utilisation anchors in Figure 6, and cost-per-query decomposition in Figure 12 are AUTHOR SCENARIO illustrative estimates. Supplier quotations and disclosed product BOMs are outside the source set. The four-regime matrix in Figure 13 is directional; the actual split between training and inference infrastructure depends on model efficiency progress, silicon cost trajectory, and hyperscaler procurement choices.

No advisory relationship with any named party. Nothing here is investment advice.