This article is a complete practical record of deploying a local large model using vLLM on a Linux server. On one hand, it serves as a reviewable note for myself, and on the other hand, I hope to help you, who are also experimenting with local large models, to avoid some pitfalls. 1. Overall Approach: … Read more
Efficiently and conveniently performing large model inference on the Ascend NPU has long been a core challenge faced by domestic developers. Although Huawei officially provides the high-performance MindIE inference engine, its high usage threshold and complex environment configuration have somewhat limited its rapid application and iteration within the broader developer community. This is not only … Read more
Introduction Compared to traditional LLM inference frameworks (such as HuggingFace Transformers, TensorRT-LLM, etc.), vLLM demonstrates significant advantages in performance, memory management, and concurrency capabilities, specifically reflected in the following five core dimensions:1. Revolutionary Improvement in Memory Utilization By utilizing Paged Attention technology (inspired by the memory paging mechanism of operating systems), the KV Cache (Key-Value … Read more
Source: DeepHub IMBA This article is approximately 2400 words long and is recommended for a 5-minute read. In this article, we will see how to use vLLM with multiple LoRA adapters. We all know that using LoRA adapters can customize large language models (LLMs). The adapters must be loaded on top of the LLM, and … Read more
1. Block Overview A significant innovation of vLLM is the division of the physical layer GPU and CPU available memory into several blocks, which effectively reduces memory fragmentation issues. Specifically, vLLM’s blocks are divided into logical and physical levels, with a mapping relationship between the two. The following diagram explains the relationship between the two … Read more