GPU memory

Embedded AI Engineer – KV Cache, Schedule, Unsloth, KTransformer

by
Embedded AI Engineer - KV Cache, Schedule, Unsloth, KTransformer

1. The formula for KV cache and how to calculate itFormula: ceil(max_model_len / block_size) * page_size_bytespage_size_bytes = 2 * block_size * num_kv_heads * head_size * dtype_sizeExplanation:block_size is the size of token chunks, with a default configuration of 16;max_model_len is the maximum input sequence length that the model can processnum_kv_heads refers to the N heads in … Read more

Guide to Calculating GPU Memory Requirements for LoRA and QLoRA Fine-Tuning: Understandable for Beginners

by
Guide to Calculating GPU Memory Requirements for LoRA and QLoRA Fine-Tuning: Understandable for Beginners

I have recently compiled a simple and easy-to-understand guide on the GPU memory requirements for fine-tuning with LoRA and QLoRA, which can help you estimate the memory needed when fine-tuning using LoRA and QLoRA. Below, we will explain step by step, requiring minimal background knowledge.1. What are LoRA and QLoRA? LoRA (Low-Rank Adaptation):This is a … Read more