What GPU do you need to run a 70B model, and what does it cost?

To run a 70B model quantized to 4-bit you need roughly 48 GB of VRAM — today the cheapest qualifying cloud GPU is the NVIDIA RTX A6000 (48 GB) at $0.57/GPU-hr median. Full FP16 inference needs ~140 GB+, which means a single AMD Instinct MI300X (192 GB) at $2.69/GPU-hr or a multi-GPU split across 2× 80 GB cards.

• Rule of thumb: weights ≈ parameters × bytes per weight (70B × 0.5 B ≈ 35 GB at 4-bit; × 2 B ≈ 140 GB at FP16), plus KV cache and runtime overhead that grow with context length.
• Cheapest 48 GB+ card today: NVIDIA RTX A6000 — $0.57/GPU-hr.
• Cheapest single card that fits FP16 (141 GB+): AMD Instinct MI300X — $2.69/GPU-hr.
• Two 80 GB cards can beat one big card on price — check the per-SKU pages and multiply by GPU count.
• These are inference floors; fine-tuning (even QLoRA) needs meaningful headroom above them.

This answer is computed from live data on every load — per-provider daily medians, per single GPU per hour, pricing tiers never mixed (see methodology). Quote it with attribution: “Data: Price of Compute — priceofcompute.com” (cite).