Practical Notes: Deploying Large Models with vLLM in a Linux Production Environment

This article is a complete practical record of deploying a local large model using vLLM on a Linux server.

On one hand, it serves as a reviewable note for myself, and on the other hand, I hope to help you, who are also experimenting with local large models, to avoid some pitfalls.

1. Overall Approach: From “Running” to “Online”

Deploying vLLM actually involves four steps:

Prepare the basic environment: GPU drivers, Python, virtual environment
Download the model: Use modelscope to pull the model to the local disk
Use vLLM to provide an OpenAI-compatible interface: Expose the HTTP API through vllm serve
Run it as a daemon: Manage the service with systemd, enabling it to start on boot and automatically restart on failure

At the end of the article, I will briefly discuss: Why can vLLM achieve higher throughput on a single GPU? — The key lies in paged attention and memory management.

2. Environment Preparation

In a production environment, it is not recommended to use the system’s built-in Python directly; it is advisable to use Conda or Python venv to isolate the environment and avoid dependency conflicts.

Operating System: Linux (Ubuntu 20.04/22.04 recommended)
Hardware: NVIDIA GPU (recommended memory ≥ 24GB, depending on model size)
Driver: CUDA 12.1+ (recommended)

Reference documentation: https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html

2.1 Install Basic Tools and GPU Drivers

apt update
apt install -y wget git curl vim

Install a stable NVIDIA driver (e.g., 535 LTS):

# Install 535 Long Term Support version (compatible with CUDA 12.x)
apt install -y nvidia-driver-535

Note: After installation, the server must be restarted. After rebooting, execute nvidia-smi; if the GPU list and CUDA version are displayed, it is successful.

nvidia-smi

If you can see the GPU and CUDA information, it means the driver is ready.

2.2 Install Python3 + venv

apt install -y python3 python3-pip python3-venv

3. Create a vLLM Dedicated Virtual Environment

In a production environment, treat the system Python as read-only, and place all business dependencies in the virtual environment.

# Create a virtual environment named vllm-env
python3 -m venv /root/vllm-env

# Activate the environment
source /root/vllm-env/bin/activate

# Upgrade pip
pip install --upgrade pip

Install vLLM and some dependencies that will be used later:

# Install the core library of vLLM
pip install vllm

# Install model download tools and OpenAI client (for testing)
pip install modelscope openai

Note: If you do not want to execute source every time, you can also write all subsequent commands in the form of the complete path like

/root/vllm-env/bin/python, /root/vllm-env/bin/vllm.

4. Model Download and Management

It is recommended to place the model on a separate data disk, such as /data/models, to avoid filling up the root partition.

4.1 Download Script Example

ModelScope provides a stable domestic download channel.

Create a script vim /root/download_models.py:

from modelscope.hub.snapshot_download import snapshot_download

# Configuration: Download Qwen2-7B-Instruct (dialog version)
model_id = 'Qwen/Qwen2-7B-Instruct'
cache_dir = '/data/models'  # Specify storage directory

print(f"Starting download: {model_id} -> {cache_dir}")
model_dir = snapshot_download(model_id, cache_dir=cache_dir)
print(f"Download finished: {model_dir}")

Execute the download:

python3 download_models.py

After the download is complete, the model directory looks like this:

/data/models/Qwen/Qwen2-7B-Instruct

Remember this path, as it will be used for configuring vLLM later.

5. Service Daemon Deployment

In a production environment, we need to use systemd to manage processes, ensuring the service has the capability for automatic startup and automatic restart on failure.

5.1 Generate an API Key

vLLM does not provide a native key management interface; we need to generate a random string as the key ourselves.

The simplest way: generate a random string to use as a global key:

openssl rand -hex 16
# Example output:
# f4a9c3b2e1d8f7a6b5c4d3e2f1a0b9c8

Please record the generated key, as it will be used in the configuration file in the next step.

5.2 Create Systemd Service File

Create a new file vim /etc/systemd/system/vllm.service:

[Unit]
Description=vllm inference service
After=network.target

[Service]
Type=simple
User=root
Group=root
# Specify the virtual environment path
Environment="PATH=/root/vllm-env/bin:/usr/local/bin:/usr/bin"

# Detailed startup command:
# --host 0.0.0.0: Allow external access
# --port 8000: Service port, customizable (e.g., 8080)
# --api-key: Set access key (replace with the key you just generated)
# --served-model-name: Specify the model name for API calls
# --gpu-memory-utilization 0.95: Recommended value for production, reserving 5% for system overhead to prevent crashes
ExecStart=/root/vllm-env/bin/vllm serve /data/models/Qwen/Qwen2-7B-Instruct \
    --served-model-name qwen2-7b \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 4096 \
    --max-num-seqs 256 \
    --trust-remote-code \
    --api-key "sk-这里换成你刚才生成的 key"

Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Load and start the service:

systemctl daemon-reload

# Set to start on boot and start now
systemctl enable --now vllm

# Check status
systemctl status vllm

6. How to Determine When Port 8000 is “Really Ready”?

Many people encounter a phenomenon the first time:

systemctl status vllm shows running
But curl localhost:8000 cannot connect yet

The reason is quite simple: Starting the Python process ≠ the model has finished loading.

vLLM needs to read several GB or even tens of GB of weights from disk into GPU memory, which can take tens of seconds to several minutes, during which the TCP port is not fully ready.

A reliable approach is to monitor the logs:

journalctl -u vllm -f

Until you see something like:

uvicorn running on http://0.0.0.0:8000

At this point, try curl, and you should be good to go.

7. Simple Self-Check with curl

7.1 Chat Interface

curl http://localhost:8000/v1/chat/completions \
  -H "content-type: application/json" \
  -H "authorization: bearer sk-这里换成你的 key" \
  -d '{
    "model": "qwen2-7b",
    "messages": [{"role": "user", "content": "你好"}],
    "max_tokens": 100
  }'

If you see a normal return, it means:

vLLM is up
The model can load
The API key verification has passed

7.2 View Model List

curl http://localhost:8000/v1/models \
  -H "authorization: bearer sk-这里换成你的 key"

If you can see qwen2-7b in the list, you can consider it done.

8. Common Parameters: How to Adjust for Better Performance?

The core parameters of vLLM are roughly as follows:

Parameter Name	Typical Value	Description
`<span>--gpu-memory-utilization</span>`	0.90 ~ 0.95	Maximum GPU memory utilization. Can be set higher for dedicated cards, but should be conservative for shared environments.
`<span>--max-model-len</span>`	4096 / 8192	Maximum context length, larger values consume more memory.
`<span>--max-num-seqs</span>`	128 / 256 / 512	Maximum number of sequences per iteration, essentially determining how many requests can be served simultaneously.
`<span>--tensor-parallel-size</span>`	1 / 2 / 4	Used for multi-GPU parallelism, e.g., set to 2 when running a large model on two cards.
`<span>--quantization</span>`	awq / gptq, etc.	Whether to enable quantization, requires a quantized model.
`<span>--dtype</span>`	auto / half	Data precision.`<span>half</span>` is fp16, which saves memory; `<span>auto</span>` will prioritize selecting bfloat16 or other suitable precisions.

9. Production Environment Optimization Strategies

In a production environment, we need to adjust parameters based on business goals to balance throughput and latency.

Scenario A: Fast Conversations for End Users (Low Latency)

Goal: When a user says “Hello”, the model must respond immediately without any lag.

Strategy:

Limit the maximum concurrency to ensure each online user gets enough computing power.
Use half-precision (FP16) or quantized models (AWQ) to reduce computation load.

--max-num-seqs 128 \
--dtype half \
--gpu-memory-utilization 0.9

Scenario B: Batch Article Generation in the Background (High Throughput)

Goal: Not concerned whether a single request takes 1 second or 3 seconds, the focus is on completing as many tasks as possible in a given time.

Strategy:

Increase concurrency to maximize vLLM’s Continuous Batching mechanism.
Increase memory utilization to allow more space for KV Cache, enabling larger Batch Sizes.

--max-num-seqs 512 \
--gpu-memory-utilization 0.95

10. Why Can vLLM Achieve Higher Throughput on a Single GPU?

Core Principle: Memory Management and Paged Attention

The “Two Mountains” in GPU Memory

During the inference phase, GPU memory is mainly occupied by two things:

Model Weights: This is fixed. For example, your Qwen2-7B, if it is in FP16 precision, will occupy about 16.8GB of memory. This part is like a “textbook” that must reside in memory.
KV Cache: This is dynamic. The model needs to rely on the computation state of all previous tokens when generating each new token. To avoid redundant calculations, we store these states, which is the KV Cache. This part is like “scratch paper”; the longer the conversation and the more concurrent users, the faster the scratch paper is used up.

vLLM’s “Killer Feature”: Paged Attention

Traditional inference frameworks (like early HuggingFace) allocate GPU memory for KV Cache quite roughly, often reserving a large contiguous space (for example, if you estimate you will say 2048 words, you pre-allocate that much space). This leads to a lot of wasted memory fragmentation — like tearing a whole A4 sheet of paper just to remember a phone number.

vLLM introduces the concept of “paged memory” from operating systems (Paged Attention): it divides GPU memory into small blocks (pages). KV Cache does not need to be stored contiguously; it can be placed wherever there is space. This means memory utilization can approach 100%.

Considering the deployment configuration: --gpu-memory-utilization 0.95

This means you command vLLM to occupy 95% of the total GPU memory. After accounting for the fixed model weights (textbook), all remaining space will be divided into blocks specifically for storing KV Cache (scratch paper).

Thought: In a production environment, suppose your Qwen2-7B model weights occupy 14GB of memory, and your GPU has a total of 24GB (leaving about 10GB for KV Cache). If we optimize (e.g., quantization or parameter adjustment) to increase the space available for KV Cache (from 10GB to 15GB), what do you think would be the most direct improvement for user experience? (Would it allow the model to respond faster? Make the model smarter? Or accommodate more users simultaneously?)

Answer: Increasing the “scratch paper” from 10GB to 15GB can support more users.

In a production environment, we refer to this capability as throughput

The larger the KV Cache = the more “scratch paper” it can hold = the larger the Batch Size.
This is like a bus (GPU); vLLM’s Paged Attention technology arranges the seats (memory) very compactly, avoiding waste, so you can take more people at once (process more user requests concurrently).

This is why vLLM is highly regarded in the industry: it can maximize the service capability of a single GPU without significantly sacrificing the response speed for individual users.

However, production environments do not only pursue “more” but also “stability”, which is why --gpu-memory-utilization 0.95 is capped at 0.95 instead of 1.0 (100%).

What is the remaining 0.05 reserved for? Besides the “textbook” (model weights) and “scratch paper” (KV Cache), the GPU also needs a third space during actual operations — we can call it the “computation temporary area” (Activation Overheads).

Intermediate computation results: Temporary tensors are generated when data is passed between model layers, and these need to be stored in memory.
PyTorch/CUDA overhead: The framework itself and the underlying drivers also require a bit of “operating funds” when running kernel functions.

If you allocate 100% of the space to KV Cache, once a request comes in and starts computing, there will be no place to store these temporary data, and the service will crash. Therefore, reserving 0.05 to 0.1 as a buffer is a “safety airbag” in production environments.

Final Thoughts

This is a complete practical record of deploying a local large model using vLLM:

Using modelscope to pull qwen2-7b-instruct locally
Using vLLM to start an OpenAI-compatible API service
Managing it with systemd for automatic startup and automatic restart on failure
Along the way, I reviewed several key parameters and the principles of paged attention

If you are also experimenting with local large models and want to integrate these capabilities into your system, I hope this practical record can help you avoid some detours.

I will continue to organize content related to large models (focusing on operations) in the future.

If you find this article useful, feel free to bookmark it + give it a thumbs up + share it with friends. You can also discuss in the comments about which models you are currently using and on what hardware, so we can truly utilize large models together.