Which AI models can the A40 run?

Every figure is for an 8,192-token conversation (about 6,000 words), at the best compression that fits. Try it in the calculator with your own memory, conversation length and compression, or a second card. For image and video models, see Stable Diffusion, FLUX and Wan on the A40.

Models that run entirely on the card

The fastest way to run a model: all of it in graphics memory. Most people read at 5 to 8 tokens a second, so anything above 10 feels quick.

With help from system memory

In a workstation with 256 GB of DDR5-6400 in eight channels, these fit with part of the model in system memory. The processor reads that part several times more slowly, so they write more slowly the more of them is off the card.

The other 3 need more memory than a workstation with 256 GB of DDR5-6400 in eight channels has; the calculator shows what more memory, or a second card, would add.

Two A40s

Two of them pool 96 GB, enough for 64 of the models entirely on the cards, 7 more than one holds. The most capable that runs well is Qwen2.5 72B at Q4_K_M, 8.4–13 tokens/s. A second card adds memory, not speed: a model that already fits on one runs no faster on two.

Compared with similar graphics

Tokens per second writing Llama 3.1 8B at 4 bits, which every 8 GB card holds, and the most capable model each runs well on its own. Every card ranked.

Estimated tokens per second; a real figure is usually within a quarter either way. * Part of it in system memory. {CAPABLE_NOTE}

How it is worked out

Writing an answer is limited by memory rather than arithmetic: for every word-piece (token) a model writes, the computer reads every weight that token uses once. So the speed is how fast the memory holding the model can be read, divided by how much is read for each token: the model’s weights at the chosen compression, plus the conversation so far. How much of its memory speed each graphics card reaches is measured, not assumed: it comes from the card’s own result in llama.cpp’s benchmark, which LM Studio, Ollama and Jan are built on. A model that does not fit in the graphics card’s memory spills into system memory, five to fifteen times slower, and one that does not fit in memory at all can be read from the drive as it runs, slower again; the estimate counts each part at its own speed, and a processor with few threads cannot use all of fast memory. Graphics built into a processor use system memory, at its speed. Reading a prompt is limited by arithmetic, so it follows the measured prompt speed.