The LoRA model is a small model of the Stable Diffusion model, achieved by making slight modifications to the standard checkpoint model. They are typically 10 to 100 times smaller than checkpoint models, making them very attractive for those who have a large number of models.
This tutorial is suitable for beginners who have never used the LoRA model before. You will learn what the LoRA model is, how to find them, and how to use them in the AUTOMATIC1111 GUI. Finally, you will find some demonstrations of LoRA models at the end of this tutorial.
# What Is the LoRA Model?
LoRA (Low-Rank Adaptation) is a training technique used to fine-tune Stable Diffusion models.
But we already have other training techniques, such as Dreambooth and text inversion. So what makes LoRA special? LoRA strikes a good balance between file size and training capability. Dreambooth is powerful but has a large model file size (2-7 GB). The text inversion model is small (about 100 KB) but has limited functionality.
LoRA is in between: their file sizes are moderate (2-200 MB), and their training capability is quite good.
Those Stable Diffusion users who love trying various models can tell you that their local storage fills up quickly. Maintaining a collection of models on a personal computer is very challenging due to the large file sizes. LoRA is an excellent solution to the storage problem.
Similar to text inversion, you cannot use the LoRA model alone. It must be used with the model checkpoint file. LoRA modifies the style by applying small changes to the accompanying model file.
# How Does LoRA Work?
LoRA makes minor modifications to the most critical parts of the Stable Diffusion model: the cross-attention layers. This is where images and prompts meet in the model. Researchers found that fine-tuning just this part of the model is sufficient to achieve good training results. The cross-attention layers are indicated in yellow in the Stable Diffusion model architecture below.
The weights of the cross-attention layers are arranged in matrix form. The matrix is like a series of numbers arranged in columns and rows in an Excel spreadsheet. The LoRA model fine-tunes the model by adding its weights to these matrices.
How can LoRA model files be smaller even though they need to store the same number of weights? The trick of LoRA is to decompose the matrix into two smaller (low-rank) matrices. This allows for fewer numbers to be stored. We illustrate this with the following example.
Assume the model has a matrix with 1,000 rows and 2,000 columns. That’s 2,000,000 numbers (1,000 x 2,000) to store in the model file. LoRA decomposes this matrix into a 1,000×2 matrix and a 2×2,000 matrix. This only requires 6,000 numbers (1,000 x 2 + 2 x 2,000), which is 333 times smaller than the original matrix size. That’s why LoRA files are smaller.
LoRA decomposes a large matrix into two small low-rank matrices.
In this example, the rank of the matrix is 2. It is far below the original dimensions, hence it is called a low-rank matrix. The rank can go as low as 1.
But does this cause any problems? Researchers found that performing such operations in the cross-attention layers does not significantly affect the fine-tuning results. Therefore, we are safe.
# How to Find LoRA Models?
Civitai offers a large number of LoRA models. By using the LORA filter, you can see only LoRA models. You may find some similarities among them: female portraits, anime, realistic illustration styles, etc.
(Please note that there is a lot of NSFW (not safe for work) content on Civitai. If you do not want to see unforgettable content, be sure to enable the NSFW filter…)
Hugging Face is another source that provides a library of LoRA models. You will find a wider variety of LoRA models there. However, the number of LoRA models there is not as many as Civitai, and their collection is smaller.
# How to Use LoRA?
In this section, you will find instructions on how to use LoRA models in the AUTOMATIC1111 Stable Diffusion GUI. You can use this GUI on Windows, Mac, or Google Colab.
AUTOMATIC1111 natively supports LoRA. You do not need to install any extensions.
## Step 1: Install the LoRA Model
To install the LoRA model in the AUTOMATIC1111 webui, place the model files in the following folder.
stable-diffusion-webui/models/Lora## Step 2: Use the LoRA Model
To use the LoRA model, include the following phrase in the prompt.
<lora:filename:multiplier>filename is the name of the LoRA model file, excluding the extension (.pt, .bin, etc.).
multiplier is the weight applied to the LoRA model. The default value is 1. Setting it to 0 will disable the model.
How to ensure the filename is correct? Do not manually write this phrase; instead, you should click the model button.
Click on the LoRA tab. You should see a list of installed LoRA models. Click on the model you want to use.
The LoRA phrase will be inserted into the prompt.
That’s it!
## Considerations When Using LoRA
You can adjust the multiplier to increase or decrease the effect. Setting the multiplier to 0 will disable the LoRA model. You can adjust the style effect between 0 and 1.
Some LoRA models are trained using Dreambooth. You need to include a trigger keyword to use the LoRA model. You can find the trigger keyword on the model’s page.
Similar to embeddings, you can use multiple LoRA models simultaneously. You can also use them together with embeddings.
In AUTOMATIC1111, the LoRA phrase is not part of the prompt. It will be removed after applying the LoRA model. This means you cannot use syntax like [keyword1:keyword2:0.8] in the prompt.
# Some LoRA Models
Here are some LoRA models I have selected.
## Shukezouma
The Shukezouma LoRA model showcases a fashionable style with a theme of Chinese ink painting. Shukezouma means that the negative space in the painting (common in Chinese artwork) is so spacious that a horse can pass through.
Prompt:
(shukezouma:0.5) ,<lora:Moxin_Shukezouma:1> , chinese painting, half body, female, perfect symmetric face, detailed chinese dress, mountains, flowers, 1girl, tigerNegative Prompt:
disfigured, ugly, bad, immature
## Akemi Takada (1980s) Style
Akemi Takada is a Japanese manga illustrator. If you love the Japanese anime of the 1980s and 1990s, then this style is for you.
Prompt:
takada akemi, Tifa lockhart as magician, Final Fantasy VII, 1girl, small breast, beautiful eyes, brown hair, smiling, red eyes, highres, diamond earring, long hair, side parted hair, hair behind ear, upper body, stylish dress, indoors, bar 1980s (style), painting (medium), retro artstyle, watercolor (medium) <lora:akemiTakada1980sStyle_1:0.6>Negative Prompt:
(worst quality, low quality:1.4), (painting by bad-artist-anime:0.9), (painting by bad-artist:0.9), watermark, text, error, blurry, jpeg artifacts, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, artist name, bad anatomy
## Cyberpunk 2077 Tarot Card
This LoRA model generates cyber beings and cities in a futuristic cyberpunk style.
Prompt:
cyberpunk, tarot card, close up, portrait, bionic body, cat, young man, perfect human symmetric face, leather metallic jacket, circuit, city street in background, natural lighting, masterpiece <lora:cyberpunk2077Tarot_tarotCard512x1024:0.6>Negative Prompt:
(worst quality, low quality:1.4), (painting by bad-artist-anime:0.9), (painting by bad-artist:0.9), watermark, text, error, blurry, jpeg artifacts, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, artist name, bad anatomy, big breast
AI Painting Practice
-
Master the basic operations and painting techniques of AI painting tools -
Utilize AI painting for commercial monetization and build a side business -
1000+ prompt word gift pack
