In the scenario of large model inference on the edge, balancing low latency and high performance is always a core requirement. The collaborative PD disaggregation architecture of NPU and CPU innovatively addresses the TTFT bottleneck of edge-side inference by deploying the Prefill phase on the NPU, executing the Decode phase on the CPU, and optimizing … Read more
Next episode preview, a detailed introduction to the scheduling layer in frameworks like vllm and tgi.1. kTransformer proposed by Tsinghua University, flexibly loads expert models onto the CPU during model execution while loading MLA/KVCache onto the GPU.It can deploy the DeepSeek R1 Q4_K_M quantized model (similar to int4 quantization) on 480G memory + 13G video … Read more
In the rapidly developing era of large language models (LLMs), FPGAs are ushering in new innovative opportunities due to their unique technical characteristics.Although GPUs and ASICs dominate the training field, FPGAs demonstrate irreplaceable value in inference optimization, edge computing, security and privacy, and task-specific acceleration. This article will delve into the innovative application directions of … Read more
Keywords: V-SEEK, LLM Inference Optimization, RISC-V, SOPHON SG2042, llama.cpp, NUMA Optimization V–SEEK: ACCELERATING LLM REASONING ON OPEN-HARDWARE SERVER-CLASS RISC-V PLATFORMS https://arxiv.org/abs/2503.17422 In recent years, the exponential growth of large language models (LLMs) has relied on GPU-based systems. However, CPUs are gradually becoming a flexible and cost-effective alternative, especially for inference (the phase where the model … Read more