2× RTX 4090 — LLM Inference Benchmarks

Single-stream LLM inference, measured on a bare-metal 2× RTX 4090 (48 GB total VRAM) machine, both GPUs working in parallel across the model layers.

Same models, same test method as the 1× RTX 4090 results — so the two pages are directly comparable.

How we test

• llama.cpp server, model split across both GPUs
• Single stream, one request at a time (worst case, not batched)
• Three context tiers: 4K, 32K and 128K tokens
• Speed shown is generation (tokens/s); prompt processing runs several times faster

Results — generation speed

Model	Quant	Context	Generation
Llama 3.1 8B Instruct	Q4_K_M	4K	183 tok/s
Llama 3.1 8B Instruct	Q4_K_M	32K	160 tok/s
Llama 3.1 8B Instruct	Q4_K_M	128K	112 tok/s
Qwen3 14B	Q4_K_M	4K	128 tok/s
Qwen3 14B	Q4_K_M	32K	111 tok/s
Qwen3 14B	Q4_K_M	128K	77 tok/s
Gemma 3 12B	Q4_K_M	4K	139 tok/s
Gemma 3 12B	Q4_K_M	32K	121 tok/s
Gemma 3 12B	Q4_K_M	128K	84 tok/s

Notes

• The second GPU roughly 1.8× single-stream generation speed on the same models — near-linear scaling for inference.
• 48 GB total VRAM also opens room for larger models or longer histories at higher KV cache precision.
• Build your machine →