Black FridayCode BLACK150: 150% on your first deposit of $100 or more, up to $500 bonus. New accounts, until 1 Dec, 00:00 UTC.Get the code
LiteKeep
Sign inDeploy a server

Tutorial, 2026

vLLM setup. A private, OpenAI-compatible API on your own GPU.

Most vLLM guides publish the API to the whole internet with no key. Here’s the setup that keeps it yours, sized for the GPU memory you actually have.

  • 10 min read
  • 5 chapters
  • Updated

NVIDIA L4, 24 GB

Forge L4 = one 2 GB slice, 8% of the card’s memory, yours alone. Processing is shared in turns.

NVIDIA L40S, 48 GB

Forge L40S = one 8 GB slice, 17% of the card’s memory, yours alone. Processing is shared in turns.

The short answer

Install vLLM in a Python environment, then run vllm serve with a model name. It downloads the model and serves an OpenAI-compatible API on port 8000. Keep it on 127.0.0.1 with an API key, leave some GPU memory spare on a virtual GPU, and reach it through an SSH tunnel.

The builder below writes the command for your GPU slice and model, or tells you it won’t fit.

vLLM is the open-source server built for serving language models to many users at once. Setting it up is a few commands; getting it right on a small GPU is where most guides stop.

This one covers the whole path: check the GPU, install, pick a model that fits, serve it privately, and call it like the OpenAI API.

In this guide you’ll learn

  • Check the GPU before you install anything
  • Install vLLM in its own environment
  • A command builder sized for your GPU memory
  • Call the API privately, over SSH
  • Run it as a service, or in Docker

Chapter 1

Check the GPU first

vLLM needs Linux, an NVIDIA GPU with compute capability 7.5 or newer (the L4 and L40S both qualify), and Python 3.11 to 3.14.

nvidia-smi
python3 --version

nvidia-smi should show the GPU and its memory: about 2 GB on a Forge L4, about 8 GB on a Forge L40S. On a Forge the NVIDIA driver is already there, so if this works, skip straight to installing. Our GPU guide explains what to look for.

Chapter 2

Install vLLM

vLLM’s docs recommend uv, a fast Python installer. Keep vLLM in its own environment so it can’t clash with system packages.

  1. Create the environment

    curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.12 --seed ~/vllm
source ~/vllm/bin/activate

  2. Install vLLM

    uv pip install vllm --torch-backend=auto
vllm download-kernels

    vLLM’s docs say to run download-kernels after every install or upgrade.

Chapter 3

Build your serve command

Pick your Forge slice and a model. The builder keeps the API on 127.0.0.1 with a key and leaves memory spare, because inside a virtual GPU slightly less is usable than the slice size.

Your GPU and model

Weight sizes are approximate. The cache reserve is our rule of thumb.

Your command

Fits.

About 3.1 GB of weights in a 6.8 GB budget leaves room for the cache. If it runs out of memory, lower --max-model-len first.

export VLLM_KEY="$(openssl rand -hex 16)"; echo "$VLLM_KEY"
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --api-key "$VLLM_KEY" \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192

Deploy a Forge L40S

Chapter 4

Call the API from your computer

The server only listens on 127.0.0.1, so you reach it through SSH.

  1. Open a tunnel

    ssh -N -L 8000:127.0.0.1:8000 root@your-server-ip

  2. List the models

    curl http://localhost:8000/v1/models -H "Authorization: Bearer YOUR_KEY"

  3. Ask it something

    curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Say hello in five words"}]}'

Any OpenAI client works: set the base URL to http://localhost:8000/v1 and the key you generated.

Chapter 5

Keep it running

Two ways to keep vLLM up after you log out.

  1. As a systemd serviceHow to check: Put your serve command in a unit file with the environment’s full path (~/vllm/bin/vllm) and the key in an EnvironmentFile, then enable it.
  2. Or in DockerHow to check: vLLM’s own image is vllm/vllm-openai. Run it with --gpus all and --ipc=host, and publish the port as -p 127.0.0.1:8000:8000 so it stays private. Docker needs the NVIDIA Container Toolkit for GPU access.
  3. Pin the versionHow to check: Install a specific vLLM release, or a specific image tag, so an update never changes your server overnight.

Now it’s your move

vLLM turns a GPU into a private, OpenAI-compatible API. Size the model for your memory, keep the API local, and reach it over SSH.

Deploy a Forge, build your command above, and serve your first model. Not sure vLLM is the right tool? Read vLLM vs Ollama.