Which AI models can the RTX 3070 Laptop run?

Every figure is for an 8,192-token conversation (about 6,000 words), at the best compression that fits. Try it in the calculator with your own memory, conversation length and compression, or a second card. For image and video models, see Stable Diffusion, FLUX and Wan on the RTX 3070 Laptop.

Models that run entirely on the card

The fastest way to run a model: all of it in graphics memory. Most people read at 5 to 8 tokens a second, so anything above 10 feels quick.

With help from system memory

In a laptop with 32 GB of DDR5-5600, these fit with part of the model in system memory. The processor reads that part several times more slowly, so they write more slowly the more of them is off the card.

The other 26 need more memory than a laptop with 32 GB of DDR5-5600 has; the calculator shows what more memory, or a second card, would add.

Compared with similar graphics

Tokens per second writing Llama 3.1 8B at 4 bits, which every 8 GB card holds, and the most capable model each runs well on its own. Every card ranked.

Estimated tokens per second; a real figure is usually within a quarter either way. * Part of it in system memory. {CAPABLE_NOTE}

How it is worked out

Writing an answer is limited by memory rather than arithmetic: for every word-piece (token) a model writes, the computer reads every weight that token uses once. So the speed is how fast the memory holding the model can be read, divided by how much is read for each token: the model’s weights at the chosen compression, plus the conversation so far. How much of its memory speed each graphics card reaches is measured, not assumed: it comes from the card’s own result in llama.cpp’s benchmark, which LM Studio, Ollama and Jan are built on. A model that does not fit in the graphics card’s memory spills into system memory, five to fifteen times slower, and one that does not fit in memory at all can be read from the drive as it runs, slower again; the estimate counts each part at its own speed, and a processor with few threads cannot use all of fast memory. Graphics built into a processor use system memory, at its speed. Reading a prompt is limited by arithmetic, so it follows the measured prompt speed.