AdaLoRA

LoRA-PAR Partitioned Training Achieves Half Parameter Usage and SOTA Performance Improvement!

by
LoRA-PAR Partitioned Training Achieves Half Parameter Usage and SOTA Performance Improvement!

Click the card below to follow「AI Vision Engine」public account ( Please add a note: direction + school/company + nickname/name ) Large-scale generative models like DeepSeekR1 and OpenAI-O1 benefit greatly from Chain of Thought (CoT) reasoning; however, improving their performance often requires massive datasets, large model sizes, and full parameter fine-tuning. While Parameter-Efficient Fine-Tuning (PEFT) helps … Read more

P-Tuning and LoRA Fine-Tuning for ChatGLM3-6B

by
P-Tuning and LoRA Fine-Tuning for ChatGLM3-6B

1. Environment Preparation Install Dependencies conda create -n glm3-pfinetune python=3.10 -y conda activate glm3-pfinetune pip install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cu118 pip install transformers==4.36.2 pip install datasets accelerate peft bitsandbytes pip install tqdm Download the GLM3-6B Model Model address (the open-source model is released by Tsinghua KEG):👉 <span>THUDM/glm-3-6b</span> (available on Hugging Face, requires <span>transformers>=4.34</span>) 2. … Read more

Understanding Large Model Fine-Tuning: What Are the Differences Between SFT and LoRA?

by
Understanding Large Model Fine-Tuning: What Are the Differences Between SFT and LoRA?

In the application of large models, “fine-tuning” is a crucial step to adapt general models to specific scenarios — for example, enabling ChatGPT to write product copy or allowing LLaMA to answer industry-specific questions, both of which rely on fine-tuning. However, many people confuse SFT and LoRA, two core technologies: both are based on optimizing … Read more

New Breakthrough in LoRa Technology: High Data Rates Supporting Satellite Networks and Audio/Video Transmission, Empowering AIoT

by
New Breakthrough in LoRa Technology: High Data Rates Supporting Satellite Networks and Audio/Video Transmission, Empowering AIoT

According to a report from Electronic Enthusiasts Network (by Huang Jingjing), at the 2025 LoRa Innovation Forum, Mike Wong, Vice President of Sales for Semtech Asia Pacific, Olivier Beaujard, Senior Director of the Semtech LoRa Ecosystem and Chairman of the LoRa Alliance, and Shahar Feldman, Senior Director of Product Marketing at Semtech, shared exciting insights … Read more

Detailed Explanation of LoRA Stack and ControlNet Stack Node Parameters

by
Detailed Explanation of LoRA Stack and ControlNet Stack Node Parameters

Overview of Node Functions: LoRA Stack: Used for batch loading and applying multiple LoRA models, achieving composite control over generation styles, characters, painting styles, etc. ControlNet Stack: Used for batch loading and applying multiple ControlNet models, achieving precise joint control over image composition, posture, depth, edges, and other dimensions. Both are designed as “stacks”, allowing … Read more

Easily Train LoRA Models with liblib: A Beginner’s Guide!

by
Easily Train LoRA Models with liblib: A Beginner's Guide!

In the fields of AI painting and natural language processing, LoRA models have become favorites among many developers and enthusiasts due to their efficient fine-tuning and resource-saving advantages. However, when it comes to model training, many people feel that the threshold is high and the operation is complex. Today, I would like to recommend a … Read more

Introduction to AdaLoRA: Adaptive Weight Matrix Fine-Tuning for Large Models

by
Introduction to AdaLoRA: Adaptive Weight Matrix Fine-Tuning for Large Models

Introduction: AdaLoRA addresses the issue of manually selecting the low-rank parameter r in LoRA and implements dynamic adjustments to all key parts of the model (including FFN), comprehensively enhancing model capabilities. Issues with LoRA LoRA allows for the original model parameters to remain unchanged while training a “small patch” (low-rank matrix ΔW) that is added … Read more