inference framework

Building from Scratch: Implementing Core Logic for Large Language Model Inference in C++

by

In the current era of large language models (LLMs), building an efficient inference framework from scratch allows us to gain a deeper understanding of the underlying logic of AI-generated content. C++, with its close-to-hardware and low-overhead characteristics, has become the preferred language for implementing lightweight LLM inference. This article will integrate the core design ideas … Read more

Deployment of vLLM Enterprise Large Model Inference Framework (Linux)

by
Deployment of vLLM Enterprise Large Model Inference Framework (Linux)

Introduction Compared to traditional LLM inference frameworks (such as HuggingFace Transformers, TensorRT-LLM, etc.), vLLM demonstrates significant advantages in performance, memory management, and concurrency capabilities, specifically reflected in the following five core dimensions:1. Revolutionary Improvement in Memory Utilization By utilizing Paged Attention technology (inspired by the memory paging mechanism of operating systems), the KV Cache (Key-Value … Read more