How much VRAM do you need to run an LLM? A sizing guide with the actual arithmetic

Weights: the first and largest number

Model weights occupy parameters × bytes per parameter. At 16-bit precision (fp16 or bf16) every parameter is two bytes, so a 7-billion-parameter model is about 14 GB, a 13B model 26 GB, a 70B model 140 GB. Quantisation shrinks that: int8 halves it, and 4-bit formats (GPTQ, AWQ, GGUF Q4) bring it to roughly 0.5–0.6 bytes per parameter including scale factors, so a 70B model fits in about 35–40 GB. Quality loss at int8 is negligible for most tasks; at int4 it is small but measurable on reasoning benchmarks, which is why many teams run int4 for chat and fp16 for evaluation.

KV cache: the number that grows with your traffic

During generation the model stores key and value tensors for every token in context so it does not recompute attention. Per token, that is 2 × layers × hidden_size × bytes (the 2 covers keys and values). For a Llama-3-style 8B model (32 layers, 4,096 hidden, fp16) it is about 0.5 MB per token; a 70B model (80 layers, 8,192 hidden) is roughly 2.5 MB per token before grouped-query attention reduces it, around 0.3 MB after. Multiply by the tokens in flight: context length × concurrent sequences. Serving 16 concurrent requests at 8,000 tokens each on an 8B model is 128,000 tokens × 0.5 MB ≈ 64 GB of cache, which is more than the weights.

Modern servers such as vLLM allocate cache in pages and reclaim it as sequences finish, so what matters is peak concurrent tokens, not the theoretical maximum of every request at full context.

The sizing table

"Comfortable serving budget" assumes a few thousand tokens of context for a handful of concurrent users with 15% headroom. Push context to 32K or 128K and the cache dominates; budget for it explicitly.

Headroom you cannot skip

• CUDA context and framework overhead: 1–2 GB per process.
• Activations: proportional to batch × sequence; small at inference, large during fine-tuning.
• Fragmentation: allocators waste a few percent; paged attention reduces but does not eliminate it.
• Fine-tuning: optimizer states and gradients multiply the weight footprint by 3–4× at full precision, which is why LoRA and QLoRA exist.

A reliable rule: if the weights plus your planned cache exceed 85% of the card, go up a tier or add tensor parallelism across two cards.

Matching the numbers to a GPU pod

FastInfra GPU Cloud lists 49 GPU types with per-minute billing, so sizing experiments cost minutes, not hours. The current rates are read from the live catalog:

A 24 GB card such as the RTX 4090 (24, from $0.96/hr) serves 7–8B models at fp16 or 13B at int4 for a small team. An 80 GB H100 (80, from $3.12/hr) serves a 70B model at int4 with room for cache, or a 30B model at fp16 with long context. For fp16 70B serving you need two 80 GB cards with tensor parallelism. Configure a pod on the GPU Cloud page, pull a vLLM image, and point the same OpenAI SDK you use for the hosted API at your pod.

When not to run it yourself

Renting a GPU makes sense when utilisation is high and steady, when you need a model that is not in the catalog, or when data must not leave your pod. For bursty or low traffic the hosted API is cheaper: you pay per token with no idle time, and the same model id is routed to the lowest-priced healthy upstream among 785 ids. Work the break-even with your measured tokens per day against the pod's hourly rate; the GPU cloud guides cluster walks through that comparison.

Measuring instead of estimating

The arithmetic above gets you to the right tier; the pod tells you the truth. After loading a model with vLLM or a similar server, watch nvidia-smi while you run a representative load test: peak memory used, GPU utilisation, and tokens per second at your target concurrency. Three outcomes are common. Memory headroom above 25% with utilisation under 40% means you can drop a tier or raise max-num-seqs to serve more users on the same card. Memory near the ceiling with out-of-memory errors under load means your KV budget was optimistic: lower max-model-len, reduce concurrency, or quantise. Healthy memory but low tokens per second means the bottleneck is compute or batching, not VRAM, and a card with more bandwidth will help more than one with more memory.

Record the numbers alongside the hourly rate from the GPU Cloud page and the per-token price of the same model on the pricing page; that pair is what the pod-versus-API decision rests on.