Which device for a local LLM?

Interactive

Which device for a local LLM?

Pick a model and how you will use it. You get the memory it needs and the smallest machines that hold it, from a mini PC to a GPU server.

All options, roughly from small to large investment

• PC with GeForce RTX 5090Too small

  Fast for models that fit into 32 GB. A consumer card: loud, hot, no ECC memory, 575 W total graphics power.

• Mini PC with AMD Ryzen AI Max+ 395Fits · 128 GB

  The cheapest route to a lot of memory: up to 128 GB of 256-bit LPDDR5x-8000 (about 256 GB/s) shared with the Radeon 8060S graphics, 45 to 120 W. Runs llama.cpp, LM Studio and Ollama.

  ≤ 7 tokens/s, upper limit for one user on that machine

• NVIDIA DGX SparkFits · 128 GB

  GB10 superchip with 128 GB unified memory and the full CUDA stack. 273 GB/s limits large dense models; strong for mixture-of-experts models, development and fine-tuning experiments.

  ≤ 8 tokens/s, upper limit for one user on that machine

• Mac Studio M5 MaxFits · 128 GB

  Up to 128 GB unified memory at 460 to 614 GB/s, and quiet. MLX, llama.cpp, Ollama and LM Studio run well on it.

  ≤ 13–18 tokens/s, upper limit for one user on that machine

• Workstation or server with RTX PRO 6000 Blackwell (1–2)Fits · 96 GB

  96 GB ECC GDDR7 at 1,792 GB/s and 600 W per card: the fastest per user. One or two of them with vLLM make a server for a team.

  ≤ 51 tokens/s, upper limit for one user on that machine

• Rack server with 2× L40SFits · 96 GB

  Two 48 GB data-center GPUs with vLLM in front: the other team setup from the article for 30B to 70B models.

• Mac Studio M5 UltraFits · 512 GB

  Up to 512 GB unified memory at 1.2 TB/s, at most 480 W continuous: the simplest way to fit a very large model on a desk.

  ≤ 34 tokens/s, upper limit for one user on that machine

• NVIDIA DGX Station (GB300)Fits · 748 GB

  A deskside machine with the GB300 superchip and 748 GB of coherent memory, 252 GB of it HBM3e.

• GPU server with H200 (up to 8)Fits · 141 GB

  141 GB HBM3e at 4.8 TB/s and up to 700 W per GPU; eight of them give over a terabyte. vLLM splits the model across the GPUs.

• NVIDIA DGX B200Fits · 1,440 GB

  Eight B200 GPUs with 1,440 GB HBM3e and 64 TB/s in total, at about 14.3 kW for the system. Power, cooling and operations become the bigger project.

Planning estimate, not a benchmark. Memory = parameters × bytes per parameter, plus about 20% for the KV cache (long documents or a few people: +50%, a team: +100%, rough factors). Unified-memory machines keep about 10% for the system. The speed limit is memory bandwidth divided by the weights read per token (only the active ones for mixture-of-experts); real numbers are lower. Device figures from the vendors' spec pages, as listed in the article (Ryzen bandwidth calculated from AMD's 256-bit LPDDR5x-8000). Before buying, test your model and prompts on rented GPUs for two weeks.