vLLM

Deployment of vLLM Enterprise Large Model Inference Framework (Linux)

by
Deployment of vLLM Enterprise Large Model Inference Framework (Linux)

Introduction Compared to traditional LLM inference frameworks (such as HuggingFace Transformers, TensorRT-LLM, etc.), vLLM demonstrates significant advantages in performance, memory management, and concurrency capabilities, specifically reflected in the following five core dimensions:1. Revolutionary Improvement in Memory Utilization By utilizing Paged Attention technology (inspired by the memory paging mechanism of operating systems), the KV Cache (Key-Value … Read more

Deploying Multiple LoRA Adapters on a Base Model with vLLM

by
Deploying Multiple LoRA Adapters on a Base Model with vLLM

Source: DeepHub IMBA This article is approximately 2400 words long and is recommended for a 5-minute read. In this article, we will see how to use vLLM with multiple LoRA adapters. We all know that using LoRA adapters can customize large language models (LLMs). The adapters must be loaded on top of the LLM, and … Read more

vLLM Framework Source Code Analysis: Block Allocation and Management

by
vLLM Framework Source Code Analysis: Block Allocation and Management

1. Block Overview A significant innovation of vLLM is the division of the physical layer GPU and CPU available memory into several blocks, which effectively reduces memory fragmentation issues. Specifically, vLLM’s blocks are divided into logical and physical levels, with a mapping relationship between the two. The following diagram explains the relationship between the two … Read more