Which AI models can the NVIDIA Jetson AGX Orin, 64 GB run?

Every figure is for an 8,192-token conversation (about 6,000 words), at the best compression that fits. Try it in the calculator with your own memory, conversation length and compression, or a second card.

A Linux board for robots and appliances.

Models that run entirely on its graphics

The fastest way to run a model: all of it in graphics memory. Most people read at 5 to 8 tokens a second, so anything above 10 feels quick.

The other 24 need more memory than 64 GB of memory has; the calculator shows what more memory, or a second card, would add.

Compared with similar graphics

Tokens per second writing Llama 3.1 8B at 4 bits, which every 8 GB card holds, and the most capable model each runs well in its share of memory. Every card ranked.

Estimated tokens per second; a real figure is usually within a quarter either way. * Part of it in system memory. {CAPABLE_NOTE}

How it is worked out

Writing an answer is limited by memory rather than arithmetic: for every word-piece (token) a model writes, the computer reads every weight that token uses once. So the speed is how fast the memory holding the model can be read, divided by how much is read for each token: the model’s weights at the chosen compression, plus the conversation so far. How much of its memory speed each graphics card reaches is measured, not assumed: it comes from the card’s own result in llama.cpp’s benchmark, which LM Studio, Ollama and Jan are built on. A model that does not fit in the graphics card’s memory spills into system memory, five to fifteen times slower, and one that does not fit in memory at all can be read from the drive as it runs, slower again; the estimate counts each part at its own speed, and a processor with few threads cannot use all of fast memory. Graphics built into a processor use system memory, at its speed. Reading a prompt is limited by arithmetic, so it follows the measured prompt speed.