The hardware model is estimated, not benchmarked — treat throughput as ±40%. It is fitted to three anchor
points for one 8×B300 node (2,304 GB VRAM) running Kimi K3 at native MXFP4: ~45 tok/s for a lone session,
~22 tok/s at 64 generating, ~8 tok/s at 256, with node output capped at ~2,000 tok/s. Weights take ~1.4 TB,
leaving ~700 GB for KV cache.
- Two ceilings, whichever hits first. Memory caps how many sessions can hold context at once; speed
caps how many can generate at once without falling below your quality floor. Long context is a memory
problem; many short sessions is a speed problem.
- Duty cycle is the biggest lever nobody sets. An agent session spends most of its wall-clock waiting
on tools, not decoding. At 30% duty you fit three times the sessions of 100%.
- Utilisation is the second. You buy for peak and pay for all 730 hours. At 30% average-to-peak you
are paying for idle iron 70% of the time — which is exactly why the API wins for bursty workloads.
- Engineering cost is real and it is not one person forever. Budget roughly one ML-infra engineer per
100–200 GPUs once you're past a single node.
- Reject any PCIe quote. A 2.8T MoE needs NVLink plus non-blocking InfiniBand for expert all-to-all;
the usual 26% PCIe inference saving does not apply here.
- Reserved prices are rising, not falling — H100 1-year reserved went from ~$1.70 to ~$2.35/GPU-hr
between Oct 2025 and Mar 2026. Waiting for a better price has been the losing move.
Model and prices as of 18 Aug 2026. Built by Kai.