A token can't come faster than the card reads its weights, so here is the limit for every model.
No rigs on the board yet. All GPUs
Theoretical ceilings, not measured speeds: bandwidth divided by the weights read per token. Real runs land below them. Beside each, the best measured run on a rig with this card.
| Model | Read per token | Theoretical ceiling | Best measured | Share of its rig's ceiling |
|---|---|---|---|---|
| Qwen3.8 Flash-NextQ4_K_MReads 3.4 GB per token | 3.4 GB | 233tok/s max | Not run yet | |
| Qwen3.8 Flash-NextIQ3_SReads 2.8 GB per token | 2.8 GB | 286tok/s max | Not run yet | |
| Qwen3.8 Flash-NextQ8_0Reads 6.0 GB per token | 6.0 GB | 133tok/s max | Not run yet | |
| Qwen3.8 27BQ4_K_MReads 15.5 GB per token | 15.5 GB | 51.8tok/s max | Not run yet | |
| Qwen3.8 27BIQ3_SReads 11.6 GB per token | 11.6 GB | 68.9tok/s max | Not run yet | |
| Qwen3.8 27BQ8_0Reads 27.0 GB per token | 27.0 GB | 29.6tok/s max | Not run yet | |
| Llama 3.3 70BQ4_K_MReads 40.4 GB per token | 40.4 GB | 19.8tok/s max | Not run yet | |
| Llama 3.3 70BIQ3_SReads 30.4 GB per token | 30.4 GB | 26.4tok/s max | Not run yet | |
| Llama 3.3 70BQ8_0Reads 70.6 GB per token | 70.6 GB | 11.3tok/s max | Not run yet | |
| Qwen3 32BQ4_K_MReads 18.8 GB per token | 18.8 GB | 42.6tok/s max | Not run yet | |
| Qwen3 32BIQ3_SReads 14.1 GB per token | 14.1 GB | 56.7tok/s max | Not run yet | |
| Qwen3 32BQ8_0Reads 32.8 GB per token | 32.8 GB | 24.4tok/s max | Not run yet | |
| Qwen3 30B-A3BQ4_K_MReads 1.9 GB per token | 1.9 GB | 423tok/s max | Not run yet | |
| Qwen3 30B-A3BIQ3_SReads 1.4 GB per token | 1.4 GB | 564tok/s max | Not run yet | |
| Qwen3 30B-A3BQ8_0Reads 3.3 GB per token | 3.3 GB | 242tok/s max | Not run yet | |
| Gemma 3 27BQ4_K_MReads 15.7 GB per token | 15.7 GB | 51.0tok/s max | Not run yet | |
| Gemma 3 27BIQ3_SReads 11.8 GB per token | 11.8 GB | 67.9tok/s max | Not run yet | |
| Gemma 3 27BQ8_0Reads 27.4 GB per token | 27.4 GB | 29.2tok/s max | Not run yet |
Mixture-of-experts models read only their active weights, which is why they run so much faster. A rig with several cards has its own ceiling, so the share is measured against that.
Be the first Apple M1 Ultra on the board.
npx toks-gg bench