Embedded AI Engineer – Lesson 2 – kTransformer UnSloth Schedule
Next episode preview, a detailed introduction to the scheduling layer in frameworks like vllm and tgi.1. kTransformer proposed by Tsinghua University, flexibly loads expert models onto the CPU during model execution while loading MLA/KVCache onto the GPU.It can deploy the DeepSeek R1 Q4_K_M quantized model (similar to int4 quantization) on 480G memory + 13G video … Read more