1× RTX 4090 — LLM Inference Benchmarks

Single-stream LLM inference, measured on a bare-metal 1× RTX 4090 (24 GB) machine — the “Single GPU” preset as configured on our pricing page.

All layers offloaded to the GPU, no other workload running on the machine. This is the speed one user gets — not batched multi-tenant throughput.

How we test

• llama.cpp server, all model layers on GPU
• Single stream, one request at a time (worst case, not batched)
• Three context tiers: 4K, 32K and 128K tokens
• Speed shown is generation (tokens/s); prompt processing runs several times faster
• KV cache quantized (Q8_0) to fit long contexts in 24 GB VRAM

Results — generation speed

Model	Quant	Context	Generation
Llama 3.1 8B Instruct	Q4_K_M	4K	102 tok/s
Llama 3.1 8B Instruct	Q4_K_M	32K	89 tok/s
Llama 3.1 8B Instruct	Q4_K_M	128K	61 tok/s
Qwen3 14B	Q4_K_M	4K	71 tok/s
Qwen3 14B	Q4_K_M	32K	62 tok/s
Qwen3 14B	Q4_K_M	128K	43 tok/s
Gemma 3 12B	Q4_K_M	4K	78 tok/s
Gemma 3 12B	Q4_K_M	32K	68 tok/s
Gemma 3 12B	Q4_K_M	128K	47 tok/s

Notes

• A single 4090 comfortably serves 8B–14B class models at Q4 quantization with full-length 128K context — practical for production chat, RAG and extraction workloads.
• Need more? See the 2× RTX 4090 results, or build your machine with up to 4 GPUs.