vLLM vs Ollama. Pick by your users and your memory.

Both serve an OpenAI-style API, so your app doesn’t care which one is behind it. Start with Ollama, and switch when the queue tells you to.

• Ollama queues. By default it answers one request per model at a time and holds up to 512 more in a queue. Raise OLLAMA_NUM_PARALLEL to run several together, but memory grows with it: Ollama reserves context for every parallel slot.
• vLLM batches. Requests join the running batch as they arrive, and its PagedAttention keeps each conversation’s cache in small blocks of GPU memory, so many fit side by side. More users share the GPU’s work instead of waiting for it.
• The cost of vLLM: by default it claims most of the GPU’s memory up front for that cache, it needs a GPU to shine, and it takes more setup.

Inside a GPU slice, slightly less memory is usable than the slice size, as NVIDIA notes for virtual GPUs. Leave headroom: give vLLM a --gpu-memory-utilization below its default, and check real usage with nvidia-smi. Our VRAM calculator works out a model’s needs in detail.

• Use Ollama. It runs GGUF models on the CPU with no extra setup. vLLM’s CPU backend exists, but its own docs call it basic.
• Size the RAM, not the CPU first. The model has to fit in memory with the system beside it: a Lite 6 for 3B models, a Lite 8 for 8B.
• Keep it to one or two users. On a CPU, every extra request slows the others down.

Ollama

1. 01

   Install and pull a model

   curl -fsSL https://ollama.com/install.sh | sh
   ollama pull llama3.1:8b
   ollama run llama3.1:8b "Say hello in five words"

2. 02

   Allow a few requests at once (optional)

   sudo systemctl edit ollama
   # add under [Service]:
   Environment="OLLAMA_NUM_PARALLEL=2"
   sudo systemctl restart ollama

   Leave OLLAMA_HOST alone: the default keeps the API on 127.0.0.1:11434, private to the server.

3. 03

   Use it from your laptop

   ssh -N -L 11434:127.0.0.1:11434 root@your-server-ip

   Your laptop now reaches the API at http://localhost:11434/v1. OpenAI clients need an API key; Ollama ignores it, so any text works.

vLLM

1. 01

   Install in its own environment

   sudo apt install -y python3-venv
   python3 -m venv ~/vllm && . ~/vllm/bin/activate
   pip install vllm

   vLLM needs Linux, Python 3.11 to 3.14 and an NVIDIA GPU with compute capability 7.5 or newer; the L4 and L40S both qualify. Check your Python with python3 --version first.

2. 02

   Serve a model, privately, with a key

   vllm serve Qwen/Qwen2.5-1.5B-Instruct \
     --host 127.0.0.1 --port 8000 \
     --api-key "$(openssl rand -hex 16)" \
     --gpu-memory-utilization 0.85 \
     --max-model-len 4096

   Print and save the key before you start, or set it from a file. A shorter --max-model-len leaves more memory for users at once.

3. 03

   Use it from your laptop

   ssh -N -L 8000:127.0.0.1:8000 root@your-server-ip

   Point your OpenAI client at http://localhost:8000/v1 with the key.

The full vLLM walkthrough, with a command builder for your GPU memory, is in our vLLM setup tutorial. New to servers? Our SSH key tutorial and SSH hardening guide cover the connection, and the VPS security guide the firewall.

1. 01

   Watch for the queue

   Requests that wait for others to finish are the sign. Raising OLLAMA_NUM_PARALLEL helps until memory runs out.

2. 02

   Pick the same model on Hugging Face

   Ollama names models its own way (llama3.1:8b); vLLM loads them from Hugging Face. Choose a 4-bit AWQ or GPTQ build to stay inside the same memory.

3. 03

   Change the base URL and the model name

   Port 11434 becomes 8000, and the model name becomes the Hugging Face name. Test one request, then move the traffic.