KV cache

Implementation of Edge-side PD Disaggregation in NPU+CPU Heterogeneous Computing

by

In the scenario of large model inference on the edge, balancing low latency and high performance is always a core requirement. The collaborative PD disaggregation architecture of NPU and CPU innovatively addresses the TTFT bottleneck of edge-side inference by deploying the Prefill phase on the NPU, executing the Decode phase on the CPU, and optimizing … Read more

Embedded AI Engineer – KV Cache, Schedule, Unsloth, KTransformer

by
Embedded AI Engineer - KV Cache, Schedule, Unsloth, KTransformer

1. The formula for KV cache and how to calculate itFormula: ceil(max_model_len / block_size) * page_size_bytespage_size_bytes = 2 * block_size * num_kv_heads * head_size * dtype_sizeExplanation:block_size is the size of token chunks, with a default configuration of 16;max_model_len is the maximum input sequence length that the model can processnum_kv_heads refers to the N heads in … Read more