Next episode preview, a detailed introduction to the scheduling layer in frameworks like vllm and tgi.1. kTransformer proposed by Tsinghua University, flexibly loads expert models onto the CPU during model execution while loading MLA/KVCache onto the GPU.It can deploy the DeepSeek R1 Q4_K_M quantized model (similar to int4 quantization) on 480G memory + 13G video memory, achieving a corresponding speed of 15 tokens/s.2. UnSlothlayer-wise dynamic quantization strategy: dynamically allocates different quantization bit numbers based on the sensitivity of different layers of the model to accuracy. For example, for more critical parts (attention, certain fully connected layers), high precision (4bit, 6bit) is maintained, while for expert layers with a large number of parameters that are relatively insensitive to accuracy (especially in Moe architecture), aggressive quantization to lower bit numbers (2bit, 1.58bit) can be applied.3. The integration of kTransformer and UnSloth.4. The schedule generally introduces continuous batching (dynamic batching), Flash Attention, and Paged Attention optimization techniques internally, significantly improving inference efficiency and performance.