I want to run a local LLM: which server should I buy?

1. Start with the model, not the box

The model has to fit into fast memory: the VRAM of a GPU or the unified memory of a Mac or an AMD or NVIDIA superchip. The weights take parameters times bytes per parameter: two bytes at 16-bit, one at 8-bit, half a byte at 4-bit. Add about 20 percent for the KV cache at normal context lengths. That gives roughly 5 GB for an 8-billion-parameter model at 4-bit, 19 GB for 32B, 42 GB for 70B and 84 GB for 70B at 8-bit. Mixture-of-experts models are the trap: only a few experts are active per token, but all of them must sit in memory, so a model with about 428 billion parameters needs around 257 GB even at 4-bit.

The second number is memory bandwidth. To produce one token, a dense model reads all of its weights once, so bandwidth divided by the size of the weights is a hard ceiling for single-user speed. A 70B model at 4-bit has about 35 GB of weights: on 1,792 GB/s that ceiling is around 50 tokens per second, on 273 GB/s it is under 8. Real numbers are lower, but the ratio between machines holds, and it explains most of the differences you will read about.

The third number is concurrency. Ten people asking at the same time need ten KV caches and a serving engine that batches their requests. That is where a team server and a personal machine part ways.

2. For one to five people: a quiet machine with a lot of memory

Four options dominate the desk in late 2026, and they trade memory against speed differently.

The Mac Studio with M5 Max offers up to 128 GB of unified memory and 460 to 614 GB/s; with M5 Ultra it goes up to 512 GB and 1.2 TB/s, at a maximum continuous power Apple lists as 480 W. That is the simplest way to fit a very large model on a desk, and it is quiet. MLX, llama.cpp, Ollama and LM Studio all run well on it.

NVIDIA's DGX Spark puts the GB10 superchip with 128 GB of unified memory into a small box. NVIDIA quotes up to 1 petaFLOP of FP4 performance and models up to 200 billion parameters, and you get the full CUDA software stack. Its 273 GB/s of bandwidth is the limit for large dense models, so it shines for development, fine-tuning experiments and mixture-of-experts models.

Mini PCs with AMD's Ryzen AI Max+ 395 share up to 128 GB of LPDDR5x-8000 between CPU and the 40-core Radeon 8060S graphics, at a configurable 45 to 120 W. They are the cheapest route to a lot of memory and run llama.cpp, LM Studio and Ollama.

A workstation with a discrete GPU is the fastest per user: the RTX PRO 6000 Blackwell brings 96 GB of ECC GDDR7 at 1,792 GB/s and 600 W, the consumer RTX 5090 has 32 GB at a total graphics power of 575 W. You get vLLM and everything else that is built for CUDA first.

# Single user on a desktop: Ollama with an OpenAI-compatible endpoint
ollama pull qwen3:32b
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:32b", "messages": [{"role": "user", "content": "Hello"}]}'

3. For a team: an NVIDIA server and vLLM

Once twenty people share one model, the question changes from "does it fit" to "does it stay fast when ten of them ask at once". That is the job of a serving engine with continuous batching and paged KV memory, and vLLM is the reference on NVIDIA hardware. It turns one GPU into a shared service with an OpenAI-compatible API, so every tool that speaks to OpenAI can talk to your server instead.

For a team of around twenty, I usually plan one rack server with one or two data-center GPUs. The L40S has 48 GB of ECC GDDR6 at 864 GB/s and a maximum of 350 W; the RTX PRO 6000 Blackwell doubles the memory to 96 GB. With 96 GB a 70B model at 4-bit fits with plenty of room for parallel KV caches; with 48 GB you are in the 30B class or need two cards. Buy the server from a vendor you can get parts from quickly, with redundant power supplies, ECC system memory, a remote management controller and fast NVMe for the model files.

Llama.cpp is excellent for one person and for unusual hardware; for many simultaneous users, vLLM's batching is what keeps answers fast. The video below compares the two well.

4. For large models and many users: multi-GPU with NVLink

Frontier-class open models and large mixture-of-experts models need hundreds of gigabytes of fast memory, spread across GPUs that talk to each other quickly. The H200 carries 141 GB of HBM3e at 4.8 TB/s and up to 700 W per GPU; eight of them give you over a terabyte. NVIDIA's DGX B200 combines eight B200 GPUs into 1,440 GB of HBM3e with 64 TB/s of total bandwidth, at a maximum of about 14.3 kW for the system. Between those sits the DGX Station, a deskside machine with the GB300 superchip and 748 GB of coherent memory, 252 GB of it HBM3e.

At this size the server is the smaller project. Power feeds, cooling, rack space, a spare-parts plan and someone who can operate it matter as much as the GPUs. vLLM splits a model across GPUs with tensor parallelism, and for mixture-of-experts models with expert parallelism; my article on MiniMax M3 walks through what that means for hosting a 400B-class model.

5. Rent first, then buy

Before any purchase, I run a two-week pilot on rented GPUs: the model you plan to use, your real prompts, your expected number of users. AWS offers H100 instances in its P5 family, Azure the ND H100 v5 series, and Google Cloud publishes its GPU prices per region. A pilot answers the questions no spec sheet can: is the 32B model good enough, or do you really need 70B? How long are your contexts? How many people use it at nine in the morning?

Buying pays off when usage is steady and predictable, when data must not leave your premises, or when the network is air-gapped and renting is simply not an option. Compare a three-year total cost that includes power, cooling, support and the hours someone spends operating it against the hourly rate you would pay for the hours you actually use. Prices change monthly, so take the current numbers from the vendor pages rather than from any article, including this one.

6. What I would buy, by scenario

One developer who wants a private coding assistant and room to experiment: a Mac Studio with 64 to 128 GB, a Ryzen AI Max+ 395 mini PC with 128 GB, or a PC with an RTX 5090 if speed matters more than model size. Ollama or LM Studio, and the endpoint plugged straight into the editor.

A team of about twenty that wants a private chat and search over its own documents: one rack server with one or two RTX PRO 6000 or two L40S, vLLM, a 30B to 70B model at 4-bit or 8-bit, and a gateway in front. Add a second, smaller machine for testing new model versions before they reach everyone.

A regulated company or an air-gapped network: two identical servers rather than one bigger one, so a failure or an update never takes the service down, ECC everywhere, redundant power, remote management on a separate network, and an offline model registry with verified hashes. My articles on local LLMs for sensitive data and on local LLMs in an air-gapped network cover the security side in detail.

7. From box to service in one afternoon

The setup is the same on every tier. Install an Ubuntu LTS release, the NVIDIA driver, CUDA and the NVIDIA Container Toolkit, so containers can use the GPUs. Copy the model files onto the local NVMe and start vLLM, which exposes an OpenAI-compatible API on the internal network. Put a gateway such as LiteLLM in front of it for keys, quotas and routing, connect it to your single sign-on, and log who asked what. A chat interface such as Open WebUI gives everyone a familiar front end, and your applications and document search talk to the same gateway.

The diagram after this step shows the whole path from hardware to users, plus the three hardware tiers side by side.

# Team server: vLLM with an OpenAI-compatible API on port 8000
# (NVIDIA driver and Container Toolkit installed, model files in /srv/models)
docker run -d --name vllm --gpus all -p 8000:8000 \
  -v /srv/models:/models \
  vllm/vllm-openai:latest \
  --model /models/Qwen2.5-32B-Instruct-AWQ \
  --max-model-len 16384

# Smoke test from the gateway host
curl http://gpu-server:8000/v1/models

8. Run it like any other production service

A local model is a service, so it needs what every service needs. NVIDIA's DCGM exporter publishes GPU utilisation, memory, temperature and errors for Prometheus, and a Grafana dashboard next to the gateway's latency and error rates tells you when people start waiting. Alert on GPU memory near its limit, on queue time and on hardware errors, not just on whether the process is up.

Keep the configuration in Git, patch the operating system and drivers on a schedule, test every new model against a fixed set of your own questions before switching, and write down how to fail over to the second machine. Then the hardware question is answered for the next three years, and the conversation can move on to what your team builds on top of it.