To run an open-weight language model locally, the device that does the math needs enough fast memory for the quantized weights, the KV cache for your context length and concurrency, and a runtime margin. Capacity decides which models fit, memory bandwidth decides how fast they generate text, and compute decides how fast they read long prompts. As a rule of thumb, a 16 GB GPU runs 8B-class models well, 24 to 32 GB covers 32B-class models at 4-bit, and 70B-class models need roughly 48 GB or more: a workstation card, two GPUs, or a unified-memory machine.
The memory formula
Every sizing decision starts from one sum:
memory ≈ weights + KV cache + runtime overhead
weights = parameters × bits per weight / 8
KV cache = 2 × layers × kv_heads × head_dim × bytes per element × tokens in flight
A 70.6-billion-parameter model needs about 141 GB for weights at 16 bits and about 42 GB at 4.8 bits per weight.
The KV cache stores attention keys and values for every token the model is tracking, so it grows with each prompt and output token. The factor of 2 covers keys and values, and the other terms come from the model's config.json: num_hidden_layers, num_key_value_heads and head_dim. Most current models use grouped-query attention, with far fewer KV heads than query heads.
Runtime overhead covers the CUDA, ROCm or Metal context and compute buffers. Budget 1 to 2 GiB per device for llama.cpp, whose automatic fitting keeps a 1024 MiB margin per device by default, and more for servers that preallocate memory for large batches. On unified memory, also leave room for the operating system.
KV cache grows with context and concurrency
The per-token cost follows directly from the config files:
Llama 3.1 8B: 2 × 32 layers × 8 kv_heads × 128 head_dim × 2 bytes = 128 KiB per token
Qwen3-32B: 2 × 64 layers × 8 kv_heads × 128 head_dim × 2 bytes = 256 KiB per token
Llama 3.3 70B: 2 × 80 layers × 8 kv_heads × 128 head_dim × 2 bytes = 320 KiB per token
At 32,768 tokens that is 4, 8 and 10 GiB with a 16-bit cache. At 131,072 tokens the 70B cache reaches 40 GiB, as much as its 4-bit weights.
Concurrency multiplies the cache: four users with 16K-token conversations need as much as one 64K conversation. vLLM builds a KV pool from the memory left after the weights, and llama-server shares one context budget (-c) across its slots (--parallel). The local LLM inference servers guide explains how each server schedules that pool.
Two levers shrink the cache. An 8-bit cache, -ctk q8_0 -ctv q8_0 in llama.cpp or --kv-cache-dtype fp8 in vLLM, roughly halves it, and capping the context at what the application uses costs nothing. Sliding-window, hybrid and multi-head latent attention store less per token, so check the KV buffer size that llama.cpp prints at load time.
Worked examples for 8B, 32B and 70B models
The llama.cpp quantize documentation lists bits per weight measured on Llama 3.1 8B: Q8_0 8.50, Q6_K 6.56, Q5_K_M 5.70, Q4_K_M 4.89, IQ4_XS 4.46 and Q3_K_M 4.00. Larger models land slightly lower; the published Q4_K_M files of Qwen3-32B and Llama 3.3 70B work out to about 4.8. This script applies the formula with a 32K context and 1.5 GiB of overhead:
GiB = 2**30
def weights_gib(params, bits_per_weight):
return params * bits_per_weight / 8 / GiB
def kv_gib(layers, kv_heads, head_dim, tokens, bytes_per_elem=2):
return 2 * layers * kv_heads * head_dim * bytes_per_elem * tokens / GiB
models = {
"Llama 3.1 8B": (8.03e9, 32),
"Qwen3-32B": (32.76e9, 64),
"Llama 3.3 70B": (70.55e9, 80),
}
for name, (params, layers) in models.items():
kv = kv_gib(layers, 8, 128, 32768)
for quant, bpw in (("BF16", 16.0), ("Q8_0", 8.5), ("Q4_K_M", 4.85)):
w = weights_gib(params, bpw)
print(f"{name:14} {quant:7} weights {w:6.1f} GiB KV@32K {kv:5.1f} GiB total {w + kv + 1.5:6.1f} GiB")
Llama 3.1 8B BF16 weights 15.0 GiB KV@32K 4.0 GiB total 20.5 GiB
Llama 3.1 8B Q8_0 weights 7.9 GiB KV@32K 4.0 GiB total 13.4 GiB
Llama 3.1 8B Q4_K_M weights 4.5 GiB KV@32K 4.0 GiB total 10.0 GiB
Qwen3-32B BF16 weights 61.0 GiB KV@32K 8.0 GiB total 70.5 GiB
Qwen3-32B Q8_0 weights 32.4 GiB KV@32K 8.0 GiB total 41.9 GiB
Qwen3-32B Q4_K_M weights 18.5 GiB KV@32K 8.0 GiB total 28.0 GiB
Llama 3.3 70B BF16 weights 131.4 GiB KV@32K 10.0 GiB total 142.9 GiB
Llama 3.3 70B Q8_0 weights 69.8 GiB KV@32K 10.0 GiB total 81.3 GiB
Llama 3.3 70B Q4_K_M weights 39.8 GiB KV@32K 10.0 GiB total 51.3 GiB
The 8B model fits a 12 GB card at Q4_K_M and a 16 GB card at Q8_0, so a 16 GB card has no capacity reason to go below Q6_K. The 32B model at Q4_K_M needs a 32 GB card for a full 32K cache; on 24 GB it fits with a 16K context and an 8-bit cache, at about 22 GiB. The 70B model at Q4_K_M needs about 51 GiB: two 32 GB GPUs, a 72 or 96 GB workstation card, or 64 GB or more of unified memory. A 48 GB card runs it with about 8K of context, and Q8_0 needs a 96 GB card or 128 GB of unified memory.
Quantization formats and what they cost
Quantization stores weights with fewer bits plus per-block scales. It saves memory and speeds up generation, and it adds error.
GGUF K-quants and I-quants
GGUF is the llama.cpp file format, also used by Ollama and LM Studio. K-quants such as Q4_K_M and Q6_K quantize blocks of weights and keep sensitive tensors at a higher type. I-quants such as IQ4_XS pack more aggressively. Both improve with an importance matrix (imatrix) computed on calibration text, and the llama.cpp documentation measures their loss with perplexity and KL divergence. GGUF runs on CUDA, ROCm, Vulkan, Metal and CPU, which makes it the most portable choice.
8-bit and FP8
Q8_0 is the 8-bit GGUF type. On GPU servers the common 8-bit format is FP8. According to the vLLM FP8 documentation, FP8 compute needs NVIDIA compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell), older GPUs fall back to weight-only W8A16 kernels, and FP8 halves model memory with up to 1.6x throughput and minimal accuracy impact.
AWQ, GPTQ and 4-bit floating point
AWQ and GPTQ are 4-bit weight-only methods for GPU servers such as vLLM and SGLang. They calibrate scales for groups of weights, keep activations in 16-bit, and many publishers ship them directly, for example Qwen/Qwen3-32B-AWQ. Blackwell GPUs add native FP4 math, and OpenAI's gpt-oss models store their expert weights in MXFP4.
How much quality you lose
8-bit formats are usually hard to tell apart from BF16. Q6_K and Q5_K_M add a small, measurable perplexity increase and are a safe default when memory allows. 4-bit is the usual compromise: the loss is measurable, and you are most likely to notice it where small errors matter, such as code, math and exact recall from long contexts. Below 4 bits the error grows quickly, and small models suffer more than large ones. A larger model at 4-bit often beats a smaller one at 8-bit in the same memory, but test that on your own tasks, as described in the AI agent evaluations guide.
Bandwidth sets generation speed, compute sets prompt speed
Generating one token with a dense model reads every weight once, plus the KV cache for the current context. At batch size 1, data movement dominates, so the ceiling is simple:
decode tokens/s ≤ memory bandwidth / bytes read per token
bytes read per token ≈ weight bytes (dense) or active-expert bytes (MoE) + KV cache bytes at current depth
The Llama 3.3 70B Q4_K_M file is 42.5 GB, so the ceiling is about 42 tokens per second on 1792 GB/s, 14 on 614 GB/s and 6 on 256 GB/s. The 8B Q4_K_M file (4.9 GB) gets about 91 on 448 GB/s. Real runtimes land below these ceilings and slow down as context grows: at 32K the 70B model's cache adds about 10.7 GB of reads per token.
The llama.cpp quantize table shows the pattern on one machine with Llama 3.1 8B: generation rises from about 29 tokens per second at F16 to 51 at Q8_0 and 72 at Q4_K_M, while prompt processing stays between about 700 and 920 tokens per second across all types.
Prompt processing handles hundreds of tokens per batch, so each weight read serves many tokens and matrix throughput becomes the limit. The work is about 2 × parameters × prompt tokens operations, roughly 5 × 10^14 for an 8,000-token prompt on a 32B model. That sets time to first token for retrieval-augmented generation, whose prompts carry retrieved passages as in the pgvector hybrid search design, and for coding agents that send large files. Check both specs: NVIDIA DGX Spark has a modest 273 GB/s decode ceiling, yet NVIDIA rates its GB10 chip at up to 1 PFLOP of sparse FP4 compute.
Batching serves many sequences per weight read, so server throughput keeps rising until compute or cache runs out. Agent loops care about single-request latency instead, because each step waits for the previous call, as the production AI agent architecture guide discusses.
Mixture-of-experts (MoE) models split the constraints: memory follows total parameters, decode speed follows active ones. gpt-oss-120b has 117 billion parameters with 5.1 billion active, and its model card says it fits a single 80 GB GPU; its MXFP4 GGUF file is 63.4 GB. gpt-oss-20b runs within 16 GB. That is why high-capacity machines with modest bandwidth stay useful.
Hardware options by capability
NVIDIA GPUs by VRAM class
NVIDIA GPUs have the widest runtime support, because vLLM, SGLang, TensorRT-LLM and llama.cpp all treat CUDA as the reference platform. Compare capacity first, then bandwidth within a class: the 16 GB GeForce cards run the same models, but the RTX 5080 has more than twice the bandwidth of the RTX 5060 Ti. The RTX PRO 6000 Blackwell holds a 70B model at 8-bit on one card. GeForce RTX 50 cards do not support NVLink, so multi-GPU traffic goes over PCIe.
Unified memory: Apple Silicon, Ryzen AI Max, DGX Spark
Unified-memory systems give the CPU and GPU one LPDDR pool: far more model memory than consumer GPUs, at lower bandwidth than a fast GDDR7 card, except for the top Apple chip. On the Mac Studio, Apple offers the M5 Ultra from 96 GB, says the 512 GB configuration arrives in late October, and supports clustering over Thunderbolt 5 with RDMA; runtimes include llama.cpp on Metal and MLX. AMD's Ryzen AI Max+ 395 uses a 256-bit LPDDR5x-8000 interface, which works out to 256 GB/s, and the Ryzen AI Max+ PRO 495 moves to LPDDR5x-8533, with OEM systems listed for the third quarter of 2026; llama.cpp runs on both through ROCm or Vulkan. NVIDIA DGX Spark runs the CUDA stack and has a 200 Gbps ConnectX-7 port for linking units.
By the ceiling formula, these machines run 70B dense models at single-digit to low double-digit token rates and MoE models much faster. Their GPUs have far smaller power budgets than a 575 W card, so measure time to first token with real prompts.
| Platform | Memory for the model | Bandwidth | Power | Dense fit at 32K context |
|---|---|---|---|---|
| GeForce RTX 5060 Ti 16 GB | 16 GB GDDR7 | 448 GB/s | 180 W | 8B at Q8_0 |
| GeForce RTX 5070 Ti / 5080 | 16 GB GDDR7 | 896 / 960 GB/s | 300 / 360 W | 8B at Q8_0 |
| GeForce RTX 5090 | 32 GB GDDR7 | 1792 GB/s | 575 W | 32B at Q4_K_M |
| RTX PRO 4000 / 4500 Blackwell | 24 / 32 GB GDDR7 ECC | 672 / 896 GB/s | 145 / 200 W | 32B at 4-bit |
| RTX PRO 5000 Blackwell | 48 or 72 GB GDDR7 ECC | 1344 GB/s | 300 W | 32B at Q8_0, 70B at Q4_K_M on 72 GB |
| RTX PRO 6000 Blackwell | 96 GB GDDR7 ECC | 1792 GB/s | 600 W or 300 W (Max-Q) | 70B at Q8_0 |
| DGX Spark | 128 GB LPDDR5x | 273 GB/s | 240 W supply | 70B at Q8_0 |
| Ryzen AI Max+ 395 | 128 GB, up to 96 GB for graphics | 256 GB/s | 45 to 120 W | 70B at Q6_K |
| Ryzen AI Max+ PRO 495 | 192 GB, up to 160 GB for graphics | about 273 GB/s | 45 to 120 W | 70B at Q8_0 |
| Mac Studio, M5 Max | up to 128 GB | up to 614 GB/s | 480 W system maximum | 70B at Q8_0 |
| Mac Studio, M5 Ultra | up to 512 GB | 1.2 TB/s | 480 W system maximum | 70B at BF16 |
The last column follows from the worked examples for a single user. The table describes capability, not value for money.
Multi-GPU and CPU offload
Two GPUs double capacity, but speed depends on the split. With a layer split, the llama.cpp default, each GPU holds a range of layers and their KV cache, and at batch size 1 the GPUs work in turn: more capacity, same decode speed. Only small activations cross PCIe per token, so x8 or x4 links are acceptable. With tensor parallelism (--tensor-parallel-size 2 in vLLM), both GPUs read their share of every layer at once, which raises the ceiling, but each layer synchronizes over PCIe. vLLM tensor parallelism is designed for identical GPUs, and the GPU count must divide the model's attention heads.
CPU offload keeps some layers in system RAM, where the CPU runs them. Time per token becomes GPU bytes over VRAM bandwidth plus CPU bytes over RAM bandwidth. Dual-channel DDR5-6000 gives a theoretical 96 GB/s (2 channels × 8 bytes × 6000 MT/s), and the DDR5 memory guide covers how channels and speeds combine. A 70B Q4_K_M model with 24 GB on the GPU and 18.5 GB in RAM stays under 5 tokens per second. Offload solves capacity, not speed.
MoE models offload far better. --cpu-moe and --n-cpu-moe N keep expert weights in RAM while attention and the KV cache stay on the GPU, and only active experts are read per token. Recent llama.cpp builds fit layers automatically when -ngl is unset (--fit is on by default); set it explicitly for repeatable results. The commands show a partly offloaded dense model, a MoE model with experts in RAM, and a layer split weighted toward the larger GPU:
llama-server -m Llama-3.3-70B-Instruct-Q4_K_M.gguf -c 16384 -ngl 48 -fa on
llama-server -hf ggml-org/gpt-oss-120b-GGUF -c 32768 -ngl all --n-cpu-moe 24 -fa on
llama-server -m Qwen_Qwen3-32B-Q8_0.gguf -c 32768 -ngl all --split-mode layer --tensor-split 3,2
Power, cooling, PCIe lanes and system RAM
For the GeForce RTX 5090, NVIDIA lists 575 W of graphics power and 1000 W of required system power. The 300 W Max-Q edition of the RTX PRO 6000 keeps the same memory as the 600 W model, which matters for several cards in one chassis. Because decoding waits on memory, test a lower limit with nvidia-smi -pl and keep the setting where generation speed barely moves.
Inference is a sustained load, and two open-air cards in adjacent slots feed each other hot air. Check clocks, temperature and power during a 30-minute run.
GeForce RTX 50 and RTX PRO Blackwell cards use PCIe 5.0 x16. Consumer desktop platforms usually offer one x16 graphics link, which some boards split into x8/x8, and a chipset slot often runs at x4. Layer splits tolerate that; tensor parallelism and model loading prefer x8 or wider, which workstation platforms provide. The reported link generation can drop while the GPU is idle, so check it under load.
Have at least as much system RAM as the largest model file you load, plus room for the OS, and much more if you offload. Model files run from about 5 GB to well over 100 GB, so a fast drive shortens load times; the NVMe SSD buying guide covers that pattern.
nvidia-smi --query-gpu=name,pcie.link.gen.current,pcie.link.width.current,power.draw,memory.used --format=csv -l 1
nvidia-smi dmon -s pucvmt -d 1
sudo nvidia-smi -pl 450
Measure your own throughput
Published numbers rarely match your model file, context, driver and runtime version, so measure and compare with the ceiling formula. In llama-bench, -p sets the prompt length, -n the generated tokens, and -d prefills the KV cache to a depth, which shows how speed falls as a conversation grows. The second command compares a 16-bit and an 8-bit cache:
llama-bench -m Qwen_Qwen3-32B-Q4_K_M.gguf -p 512,4096 -n 128 -d 0,16384 -fa on -r 3 -o md
llama-bench -m Qwen_Qwen3-32B-Q4_K_M.gguf -p 4096 -n 128 -d 16384 -ctk f16,q8_0 -ctv f16,q8_0 -fa on
Then test concurrency. vllm bench serve works against OpenAI-compatible endpoints and reports time to first token, time per output token, inter-token latency and throughput:
vllm serve Qwen/Qwen3-32B-AWQ --max-model-len 32768
vllm bench serve --backend openai --base-url http://127.0.0.1:8000 --model Qwen/Qwen3-32B-AWQ --dataset-name random --random-input-len 4096 --random-output-len 256 --num-prompts 200 --max-concurrency 8
Divide measured single-stream speed by the ceiling. If the ratio is unusually low, look for layers silently offloaded to the CPU, a PCIe link at reduced width, throttling, or a slow backend.
A sizing checklist
- Record parameters, layers, KV heads and head dimension from
config.json. - Choose the quantization: 8-bit if it fits, then Q6_K or Q5_K_M, then 4-bit, and lower only after testing.
- Set context and concurrency, compute the KV cache, and decide whether an 8-bit cache is acceptable.
- Add 1 to 2 GiB per device, plus OS headroom on unified memory.
- Reject platforms whose decode ceiling is below the speed you need.
- For long RAG or coding prompts, weigh compute and measure time to first token.
- Size MoE memory by total parameters and speed by active parameters.
- Check power, cooling, PCIe lanes, RAM and storage before buying a second GPU.
- Benchmark, and close the gap to the ceiling before adding hardware.