Best GPU for local AI and LLMs: tokens per second by graphics card

The fastest card at each memory size

Choose the memory first, by the models you want to run, then the fastest card with that much. Figures are for an 8,192-token conversation at the best compression that fits.

8 GB: the RTX 3070 Ti is the fastest. It runs Qwen3.5 9B at Q4_K_M, 49–78 tokens/s. Others with as much: RTX 5060 Ti 8 GB, RTX 5060, RTX 3060 Ti.
12 GB: the RTX 3080 Ti is the fastest. It runs Qwen3 14B at Q4_K_M, 49–78 tokens/s. Others with as much: RTX 3080 12 GB, RTX 5070, RTX 4070 Ti.
16 GB: the RTX 5080 is the fastest. It runs Qwen3 14B at Q6_K, 44–71 tokens/s. Others with as much: RTX 5070 Ti, RTX 4080 SUPER, RTX 4080.
24 GB: the RTX 4090 is the fastest. It runs Qwen3 32B at Q4_K_M, 28–45 tokens/s. Others with as much: Radeon RX 7900 XTX, RTX 3090 Ti, RTX 3090.
32 GB: the RTX 5090 is the fastest. It runs Qwen3 32B at Q6_K, 35–56 tokens/s.

For models bigger than one gaming card

Two cards. Two cards pool their memory: two 24 GB cards hold 70B models at 4 bits. They add room, not speed. Each card’s page says what two of it run.
Ryzen AI Max+ 395 (Radeon 8060S): up to 128 GB shared between processor and graphics, three quarters of it usable for a model, at 256 GB/s: slower than a gaming card, but it holds models none can. It runs Llama 4 Scout 109B-A17B at Q6_K, 10–16 tokens/s.
NVIDIA DGX Spark and other GB10 mini PCs, 128 GB: 128 GB at 273 GB/s in a mini PC, on Linux. It runs Llama 4 Scout 109B-A17B at Q8_0, 8.7–14 tokens/s.
RTX PRO 6000 Blackwell: a workstation card with 96 GB. It runs Mistral Medium 3.5 128B at Q5_K_M, 10–16 tokens/s.

Tokens per second by graphics card

Estimated writing speed for four popular models at 4 bits (gpt-oss as released), in a desktop with the system memory a PC with that card usually has. A real figure is usually within a quarter either way. * Part of the model in system memory: much slower. — Needs more memory than that PC has. Most capable: by parameters; for mixture-of-experts models, which use a few of their experts for each token, the square root of total times active parameters, the usual rule of thumb.

Tokens per second for each desktop graphics card
Graphics	Memory	Llama 3.1 8B	gpt-oss 20B	Qwen3 30B-A3B	Llama 3.3 70B	Most capable it runs well
RTX 5090	32 GB	215	235	230	3.7 *	Qwen3 32B
RTX 4090	24 GB	135	190	180	2.4 *	Qwen3 32B
Radeon RX 7900 XTX	24 GB	135	185	175	2.4 *	Qwen3 32B
RTX 5080	16 GB	135	185	68 *	1.9 *	Qwen3 14B
RTX 5070 Ti	16 GB	130	180	68 *	1.9 *	Qwen3 14B
RTX 3090 Ti	24 GB	125	180	170	1.5 *	Qwen3 32B
RTX 3090	24 GB	115	170	160	1.5 *	Qwen3 32B
RTX 3080 Ti	12 GB	110	75 *	28 *	—	Qwen3 14B
RTX 3080 12 GB	12 GB	110	75 *	28 *	—	Qwen3 14B
RTX 4080 SUPER	16 GB	110	165	65 *	1.9 *	Qwen3 14B
RTX 4080	16 GB	105	160	65 *	1.8 *	Qwen3 14B
RTX 3080 10 GB	10 GB	100	43 *	24 *	—	Gemma 3 12B
Radeon RX 9070 XT	16 GB	99	155	64 *	1.8 *	Qwen3 14B
RTX 4070 Ti SUPER	16 GB	96	150	64 *	1.8 *	Qwen3 14B
RTX 5070	12 GB	92	92 *	42 *	—	Qwen3 14B
Radeon RX 7900 XT	20 GB	89	145	120 *	2.1 *	Muse Glimmer 30B
Radeon RX 9070	16 GB	86	140	62 *	1.8 *	Qwen3 14B
Radeon RX 7800 XT	16 GB	85	140	62 *	1.8 *	Qwen3 14B
Radeon RX 7900 GRE	16 GB	84	140	62 *	1.8 *	Qwen3 14B
Radeon RX 6950 XT	16 GB	83	140	44 *	—	Qwen3 14B
RTX 4070 Ti	12 GB	80	87 *	42 *	—	Qwen3 14B
RTX 4070 SUPER	12 GB	78	86 *	42 *	—	Qwen3 14B
Radeon RX 6900 XT	16 GB	78	130	43 *	—	Qwen3 14B
RTX 2080 Ti	11 GB	77	47 *	23 *	—	Ministral 3 14B
Radeon RX 6800	16 GB	73	125	43 *	—	Qwen3 14B
Radeon VII	16 GB	73	125	39 *	—	Qwen3 14B
Radeon RX 6800 XT	16 GB	72	125	43 *	—	Qwen3 14B
RTX 3070 Ti	8 GB	72	30 *	20 *	—	Qwen3.5 9B
RTX 2080 SUPER	8 GB	68	27 *	18 *	—	Qwen3.5 9B
RTX 4070	12 GB	66	81 *	41 *	—	Qwen3 14B
RTX 5060 Ti 16 GB	16 GB	65	115	58 *	1.8 *	Qwen3 14B
RTX 5060 Ti 8 GB	8 GB	65	44 *	31 *	—	Qwen3.5 9B
RTX 2070 SUPER	8 GB	63	26 *	18 *	—	Qwen3.5 9B
RTX 5060	8 GB	63	43 *	31 *	—	Qwen3.5 9B
RTX 2080	8 GB	63	26 *	18 *	—	Qwen3.5 9B
RTX 3060 Ti	8 GB	62	29 *	20 *	—	Qwen3.5 9B
Radeon RX 7700 XT	12 GB	61	78 *	40 *	—	Qwen3 14B
RTX 2070	8 GB	60	26 *	18 *	—	Qwen3.5 9B
Radeon RX 6700 XT	12 GB	60	62 *	27 *	—	Qwen3 14B
Radeon RX 6750 XT	12 GB	59	62 *	27 *	—	Qwen3 14B
RTX 3070	8 GB	56	29 *	20 *	—	Qwen3.5 9B
RTX 3060 12 GB	12 GB	54	60 *	27 *	—	Qwen3 14B
Radeon RX 9060 XT 16 GB	16 GB	51	97	53 *	1.8 *	Qwen3 14B
Radeon RX 9060 XT 8 GB	8 GB	51	42 *	31 *	—	Qwen3.5 9B
Radeon RX 5700 XT	8 GB	51	26 *	18 *	—	Qwen3.5 9B
Arc B580	12 GB	50	71 *	39 *	—	Qwen3 14B
RTX 5050	8 GB	47	41 *	30 *	—	Qwen3.5 9B
RTX 4060 Ti 16 GB	16 GB	46	89	51 *	1.7 *	Qwen3 14B
RTX 4060 Ti 8 GB	8 GB	46	41 *	30 *	—	Qwen3.5 9B
Radeon RX Vega 64	8 GB	45	25 *	18 *	—	Qwen3.5 9B
GTX 1080 Ti	11 GB	45	41 *	22 *	—	Ministral 3 14B
Radeon RX 6650 XT	8 GB	44	28 *	20 *	—	Qwen3.5 9B
RTX 4060	8 GB	44	41 *	30 *	—	Qwen3.5 9B
RTX 2060 SUPER	8 GB	43	25 *	18 *	—	Qwen3.5 9B
Radeon RX 7600	8 GB	41	40 *	30 *	—	Qwen3.5 9B
Arc A770	16 GB	40	81	49 *	1.7 *	Qwen3 14B
Radeon RX 6600 XT	8 GB	38	27 *	19 *	—	Qwen3.5 9B
Radeon RX 7600 XT	16 GB	38	77	47 *	1.7 *	Qwen3 14B
RTX 3060 8 GB	8 GB	37	27 *	19 *	—	Qwen3.5 9B
Arc A750	8 GB	37	39 *	30 *	—	Qwen3.5 9B
Radeon RX 6600	8 GB	36	27 *	19 *	—	Qwen3.5 9B
Radeon RX Vega 56	8 GB	36	25 *	17 *	—	Qwen3.5 9B
Arc B570	10 GB	35	47 *	32 *	—	Gemma 3 12B
RTX 3050 8 GB	8 GB	34	27 *	19 *	—	Qwen3.5 9B
GTX 1080	8 GB	32	24 *	17 *	—	Qwen3.5 9B
Radeon RX 580 8 GB	8 GB	28	24 *	17 *	—	Qwen3.5 9B
GTX 1070	8 GB	27	23 *	17 *	—	Qwen3.5 9B
GTX 1070 Ti	8 GB	27	23 *	17 *	—	Qwen3.5 9B
Radeon RX 570 8 GB	8 GB	27	23 *	17 *	—	Qwen3.5 9B
RTX 2060	6 GB	18 *	20 *	16 *	—	Gemma 4 E2B
GTX 1660 SUPER	6 GB	18 *	20 *	16 *	—	Gemma 4 E2B
GTX 1660 Ti	6 GB	18 *	20 *	16 *	—	Gemma 4 E2B
RTX 3050 6 GB	6 GB	16 *	21 *	17 *	—	Gemma 4 E2B
GTX 1660	6 GB	16 *	20 *	15 *	—	Gemma 4 E2B
GTX 1060 6 GB	6 GB	13 *	19 *	15 *	—	Gemma 4 E2B
Radeon RX 6500 XT	4 GB	9.4 *	18 *	15 *	—	SmolLM3 3B
GTX 1650 SUPER	4 GB	9.0 *	16 *	14 *	—	SmolLM3 3B
Radeon RX 570 4 GB	4 GB	9.0 *	16 *	14 *	—	SmolLM3 3B
Radeon RX 580 4 GB	4 GB	8.9 *	16 *	14 *	—	SmolLM3 3B
GTX 980	4 GB	8.9 *	16 *	14 *	—	SmolLM3 3B
GTX 1650	4 GB	8.4 *	16 *	14 *	—	SmolLM3 3B
GTX 1050 Ti	4 GB	8.0 *	16 *	14 *	—	SmolLM3 3B
GTX 970	3.5 GB	7.8 *	16 *	13 *	—	Qwen3.5 2B
GTX 1060 3 GB	3 GB	7.2 *	15 *	13 *	—	Qwen3.5 2B

Workstation and data-centre cards

Tokens per second for workstation and data-centre cards
Graphics	Memory	Llama 3.1 8B	gpt-oss 20B	Qwen3 30B-A3B	Llama 3.3 70B	Most capable it runs well
Instinct MI300X	192 GB	320	275	270	47	Mistral Medium 3.5 128B
H100 80 GB (SXM)	80 GB	215	235	230	30	Qwen2.5 72B
RTX PRO 6000 Blackwell	96 GB	205	230	220	28	Mistral Medium 3.5 128B
A100 80 GB	80 GB	140	190	180	18	Qwen2.5 72B
RTX 6000 Ada	48 GB	130	180	175	17	Qwen2.5 72B
Instinct MI210	64 GB	115	170	160	15	Qwen2.5 72B
L40 / L40S	48 GB	110	165	155	14	Qwen2.5 72B
Radeon AI PRO R9700	32 GB	105	160	155	3.7 *	Qwen3 32B
RTX A6000	48 GB	100	155	150	13	Qwen2.5 72B
RTX PRO 4000 Blackwell	24 GB	95	150	145	2.6 *	Qwen3 32B
RTX A5000	24 GB	94	150	140	2.6 *	Qwen3 32B
Tesla V100 32 GB	32 GB	93	150	140	3.6 *	Qwen3 32B
Radeon PRO W7900	48 GB	92	150	140	12	Qwen2.5 72B
A30	24 GB	90	145	140	2.6 *	Qwen3 32B
A40	48 GB	90	145	135	12	Qwen2.5 72B
Instinct MI100	32 GB	86	140	135	3.5 *	Qwen3 32B
Instinct MI50 32 GB	32 GB	78	135	125	3.4 *	Qwen3 32B
Quadro RTX 8000	48 GB	74	130	120	9.4	Qwen3 32B
Instinct MI60	32 GB	72	125	120	3.4 *	Qwen3 32B
RTX A4500	20 GB	70	125	105 *	2.2 *	Muse Glimmer 30B
RTX A4000	16 GB	60	110	59 *	2.0 *	Qwen3 14B
Radeon PRO V620	32 GB	59	110	100	3.2 *	Qwen3 32B
Tesla P100 16 GB	16 GB	52	99	57 *	1.9 *	Qwen3 14B
Arc Pro B60	24 GB	47	91	84	2.3 *	Qwen3 32B
Instinct MI25	16 GB	46	89	54 *	1.9 *	Qwen3 14B
P102-100 (mining card)	10 GB	45	54 *	36 *	1.7 *	Gemma 3 12B
RTX 4000 Ada	20 GB	44	87	76 *	2.1 *	Muse Glimmer 30B
Tesla P40	24 GB	39	79	73	2.2 *	Qwen3 32B
RTX 2000 Ada	16 GB	36	74	49 *	1.9 *	Qwen3 14B
Tesla T4	16 GB	33	69	47 *	1.8 *	Qwen3 14B
Arc Pro B50	16 GB	33	68	47 *	1.8 *	Qwen3 14B
Tesla M40 24 GB	24 GB	27	58	53	2.0 *	Qwen3 14B
Tesla P4	8 GB	24	37 *	30 *	1.6 *	Qwen3.5 9B

Laptops

Laptop graphics are slower than the desktop cards of the same name, by how much depends on the laptop. Without a graphics card, the built-in graphics use the laptop’s memory.

Tokens per second for laptop graphics
Graphics	Memory	Llama 3.1 8B	gpt-oss 20B	Qwen3 30B-A3B	Llama 3.3 70B	Most capable it runs well
RTX 5090 Laptop	24 GB	115	170	160	2.3 *	Qwen3 32B
RTX 5080 Laptop	16 GB	110	165	63 *	—	Qwen3 14B
RTX 5070 Ti Laptop	12 GB	83	86 *	40 *	—	Qwen3 14B
RTX 4090 Laptop	16 GB	76	130	58 *	—	Qwen3 14B
RTX 4080 Laptop	12 GB	61	76 *	38 *	—	Qwen3 14B
RTX 5070 Laptop	8 GB	54	40 *	29 *	—	Qwen3.5 9B
RTX 5060 Laptop	8 GB	52	40 *	29 *	—	Qwen3.5 9B
RTX 3070 Laptop	8 GB	45	39 *	29 *	—	Qwen3.5 9B
RTX 4060 Laptop	8 GB	42	39 *	28 *	—	Qwen3.5 9B
RTX 4070 Laptop	8 GB	41	38 *	28 *	—	Qwen3.5 9B
Arc A770M	16 GB	37	75	45 *	—	Qwen3 14B
Radeon RX 6800M	12 GB	35	57 *	34 *	—	Qwen3 14B
Radeon RX 6600M	8 GB	34	37 *	28 *	—	Qwen3.5 9B
Radeon RX 7600M XT	8 GB	32	36 *	28 *	—	Qwen3.5 9B
GTX 1660 Ti Laptop	6 GB	25 *	32 *	26 *	—	Gemma 4 E2B
RTX 3060 Laptop	6 GB	23 *	31 *	25 *	—	Gemma 4 E2B
RTX 4050 Laptop	6 GB	22 *	31 *	25 *	—	Gemma 4 E2B
Arc B370 / B390 (Core Ultra 300)	32 GB shared	17	39	33 *	—	Ministral 3 14B
Arc 130V / 140V (Core Ultra 200V)	32 GB shared	17	38	32 *	—	Ministral 3 14B
Radeon 880M / 890M (Ryzen AI 300)	32 GB shared	16	35	31 *	—	Gemma 3 12B
Radeon 660M / 680M (Ryzen 6000, 7035)	16 GB shared	15	—	—	—	Qwen3.5 9B
Radeon 740M / 760M / 780M (Ryzen 7040, 8040, 8000G)	32 GB shared	14	32	27 *	—	Qwen3.5 9B
RTX 3050 Laptop	4 GB	14 *	26 *	23 *	—	SmolLM3 3B
Radeon Vega 6 / 7 / 8 (Ryzen 4000, 5000)	16 GB shared	7.7	—	—	—	Gemma 4 E4B
Arc graphics (Core Ultra 100)	16 GB shared	6.0	—	—	—	Gemma 4 E4B
Iris Xe (11th to 13th gen laptops)	16 GB shared	5.2	—	—	—	Gemma 4 E2B

Shared-memory machines and Macs

Tokens per second for shared-memory machines
Graphics	Memory	Llama 3.1 8B	gpt-oss 20B	Qwen3 30B-A3B	Llama 3.3 70B	Most capable it runs well
Apple M5 Max	64 GB shared	76	130	120	9.6	Qwen3 32B
Apple M2 Ultra	128 GB shared	68	120	110	8.6	Llama 4 Scout 109B-A17B
Apple M3 Ultra	128 GB shared	66	120	110	8.4	Llama 4 Scout 109B-A17B
Apple M1 Ultra	128 GB shared	60	110	105	7.6	Llama 4 Scout 109B-A17B
Apple M4 Max	64 GB shared	60	110	100	7.5	Qwen3 32B
Apple M3 Max	64 GB shared	47	92	85	6.0	Qwen3 32B
Apple M2 Max	64 GB shared	47	91	85	5.9	Qwen3 32B
Apple M1 Max	64 GB shared	44	86	80	5.5	Qwen3 32B
Apple M5 Pro	32 GB shared	41	82	77	—	Gemma 4 31B
NVIDIA DGX Spark and other GB10 mini PCs, 128 GB	128 GB shared	41	82	76	5.1	Llama 4 Scout 109B-A17B
Ryzen AI Max+ 395 (Radeon 8060S)	128 GB shared	37	75	70	4.6	Llama 4 Scout 109B-A17B
Apple M4 Pro	32 GB shared	36	74	68	—	Muse Glimmer 30B
Ryzen AI Max 385 / 390 (Radeon 8050S)	128 GB shared	36	73	68	4.5	Llama 4 Scout 109B-A17B
Apple M2 Pro	32 GB shared	28	59	54	—	Qwen3 14B
Apple M1 Pro	32 GB shared	26	56	51	—	Qwen3 14B
NVIDIA Jetson AGX Orin, 64 GB	64 GB shared	24	52	48	3.0	Nemotron 3.5 Bolt 74B-A9B
Apple M3 Pro	32 GB shared	22	48	44	—	Qwen3 14B
Apple M5	16 GB shared	21	—	—	—	Ministral 3 14B
Apple M4	16 GB shared	17	—	—	—	Ministral 3 14B
Apple M2	16 GB shared	16	—	—	—	Gemma 3 12B
Apple M3	16 GB shared	15	—	—	—	Gemma 3 12B
Apple M1	16 GB shared	10	—	—	—	Llama 3.1 8B

For AI images and video, which depend on a card’s arithmetic rather than its memory speed, see Can my PC run Stable Diffusion, FLUX and Wan?

What makes a graphics card good for AI

Memory, first. A model runs fast only when it fits. At 4 bits, 8 GB holds 8B models; 12 GB a 14B; 16 GB gpt-oss 20B; 24 GB Qwen3 30B-A3B and 27B to 32B dense models; 32 GB the same with longer conversations. Two cards pool their memory.
Memory speed, second. To write each token a model reads its weights from memory, so tokens per second follow memory bandwidth more than raw graphics power.
Arithmetic, for reading. How fast a long prompt or document is read follows the card’s arithmetic, where newer NVIDIA cards lead.
Software. LM Studio, Ollama and Jan run on NVIDIA, AMD and Intel cards on Windows; NVIDIA’s is the most widely supported, and AMD and Intel cards run through Vulkan.

Every graphics card

How it is worked out

Writing an answer is limited by memory rather than arithmetic: for every word-piece (token) a model writes, the computer reads every weight that token uses once. So the speed is how fast the memory holding the model can be read, divided by how much is read for each token: the model’s weights at the chosen compression, plus the conversation so far. How much of its memory speed each graphics card reaches is measured, not assumed: it comes from the card’s own result in llama.cpp’s benchmark, which LM Studio, Ollama and Jan are built on. A model that does not fit in the graphics card’s memory spills into system memory, five to fifteen times slower, and one that does not fit in memory at all can be read from the drive as it runs, slower again; the estimate counts each part at its own speed, and a processor with few threads cannot use all of fast memory. Graphics built into a processor use system memory, at its speed. Reading a prompt is limited by arithmetic, so it follows the measured prompt speed.

Already running a model? See if it runs at full speed

A model that is slower than it should be is nearly always partly on the processor rather than all on the graphics card, which is several times slower — and Ollama does that without a word when something else holds the card’s memory. Bottleneck’s free app checks a model run with Ollama, LM Studio, llama.cpp, KoboldCpp or Jan: where it really runs, how fast it writes, and what your PC should manage with it. Nothing leaves your PC. Or check and fix it by hand.

It reads your parts too: the graphics card and how much memory it really has free, how much system memory is installed, the speed it runs at and in how many channels. Memory left in one channel, or below its rated speed, halves how fast a model runs from it, and Bottleneck says so. Its AI models screen then lists what runs on this PC.

Download free for Windows Can my PC run a game?

The free version shows where a model runs and how fast, with no time limit and no account. Pro, $9 once, explains why it is slower than it could be and how to fix it. Compare the versions

Best GPU for local AI: tokens per second by graphics card

The fastest card at each memory size

For models bigger than one gaming card

Tokens per second by graphics card

Workstation and data-centre cards

Laptops

Shared-memory machines and Macs

What makes a graphics card good for AI

Every graphics card

NVIDIA GeForce RTX 50

NVIDIA GeForce RTX 40

NVIDIA GeForce RTX 30

NVIDIA GeForce RTX 20

NVIDIA GeForce GTX

NVIDIA laptop graphics

AMD Radeon RX 9000

AMD Radeon RX 7000

AMD Radeon RX 6000

AMD Radeon RX 5000 and older

Intel Arc

Workstation and data-centre cards

NVIDIA GeForce GTX

AMD Radeon RX 5000 and older

AMD Radeon RX 6000

NVIDIA laptop graphics

AMD and Intel laptop graphics

Graphics built into the processor

Processors with fast shared memory

AI mini PCs

Apple Mac (for comparison)

How it is worked out

Already running a model? See if it runs at full speed