LoRA Empowerment: Addressing the Data Scarcity of Large Model Open Platforms

1. Technical Implementation Principles

To understand the technical implementation principles of LoRA, remember an important concept: indirect training. When we talk about training (fine-tuning) large models, we usually refer to “direct training,” which involves fine-tuning all parameters of the large model itself to better adapt to specific generative requirements in certain scenarios. However, this method is too time-consuming and labor-intensive. Therefore, technicians have innovatively thought of an alternative solution: keeping the parameters of the large model itself largely unchanged while simplifying the model’s parameter matrix into two low-rank matrices. By fine-tuning these two smaller matrices, we can indirectly control the generative capabilities of the entire large model.

Compared to direct fine-tuning of the large model, this indirect training can achieve similar effects without affecting the original parameter settings of the large model, making it a cost-effective solution widely adopted at this stage.

LoRA achieves parameter-efficient updates by decomposing the model’s weight matrix into low-rank matrices. Specifically, it decomposes the weight matrix of the large model into two smaller matrices, whose dimensions are much smaller than the original weight matrix. This decomposition significantly reduces the number of parameters that need to be updated, making the training process more efficient.

Training traditional large models requires updating a large number of parameters, which is both time-consuming and resource-intensive. With LoRA, users only need to update the parameters of the low-rank matrices. Since these parameters are far fewer in number than the original model parameters, the update process becomes lighter and faster.

The LoRA model allows users to conduct targeted training using data from specific domains. For example, if a user has data from the medical field, LoRA can enable the model to learn specific terminology, knowledge, and patterns from that field, thereby improving its performance on related tasks. This targeted training can significantly enhance the model’s accuracy and effectiveness in specific tasks.

Users can apply their specific datasets to train the LoRA model. Since LoRA only requires updates to the parameters of the low-rank matrices, the user’s data does not significantly impact the overall structure and performance of the large model. This allows users to train models that meet their needs with a small amount of data without compromising the overall performance of the model.

By using low-rank matrix decomposition, LoRA significantly accelerates the training speed. Users can complete the model training process in a shorter time and quickly carry out multiple iterations for optimization. This efficient training process is very beneficial for applications that require frequent updates and optimizations.

Another important advantage of LoRA is that its update mechanism does not compromise the overall performance of the original large model. The user’s targeted training is limited to updates of the low-rank matrix parameters, while most parameters of the original model remain unchanged. This means users can optimize specific tasks using their data without affecting the performance of the original model.

First, the user loads a pre-trained large language model (e.g., GPT-4). This underlying large model has been trained on a vast amount of general data, possessing extensive language understanding and generation capabilities.

After loading the underlying large model, the low-rank matrix parameters trained through LoRA technology are introduced. This process does not change the core structure and most parameters of the large model but instead overlays the low-rank matrix parameters onto the original model through a parameter adjustment mechanism. Specifically, these low-rank matrix parameters adjust and optimize some of the model’s weights, achieving personalized optimization for specific tasks.

In practice, the output of the large model is the result of the interplay between the parameters of the underlying large model and the LoRA parameters. The original large model provides general language capabilities, while the LoRA parameters optimize specific fields or tasks, allowing the model to perform better on specific tasks. For example, in financial text processing tasks, the LoRA parameters help the model better understand and generate proprietary terminology and knowledge from the financial domain.

During the inference phase (i.e., when the model is used for text generation or understanding), each inference computation of the model incorporates the combined effects of the underlying model and LoRA parameters. The data input by the user is processed through these two sets of parameters, generating high-quality outputs that meet specific domain requirements.

If users find that the performance of certain tasks needs further optimization, they can perform additional fine-tuning training based on the existing LoRA parameters. In this process, users only need to provide more domain data and update the LoRA parameters, without needing to retrain the entire large model. This iterative updating mechanism greatly enhances the training efficiency and flexibility of the model.

2. Can Universal Large Models Solve the “Data Scarcity” Problem?

The technical implementation principles of LoRA have given rise to a commercial model, creating open large model platforms. These platforms can operate the large model base while also opening LoRA (which can be understood as a “small model”) to users, allowing them to upload data to train the desired generative effects with LoRA.

Based on the principles introduced earlier, the training costs required by LoRA are very low, and the volume of required data is also small, making it inherently suitable for self-training by end-users. As a result, the long-standing “data scarcity” problem in the large model industry seems to be thoroughly resolved by allowing users to contribute data through this open platform.

But is this really the case? First, introducing end-users to provide training data is indeed a very good idea, as it objectively achieves “collective wisdom,” mobilizing the data resources held by a wide range of users. Moreover, in this case, since the open large model platform does not directly use the data, it seems to have more justification to claim the “safe harbor” responsibility exemption mechanism in the event of infringement, thereby reducing the risk of using data for the platform.

However, problems still exist. The data contributed by users may not necessarily be data they fully own rights to, and the self-owned LoRA models they train may very well be directed toward infringing purposes. The open mechanism of the LoRA model provided by the platform also relies on the capabilities of the underlying model. As analyzed earlier, it is necessary to overlay LoRA parameters onto the parameters of the large model to guide the large model in content generation. Therefore, the ultimate effect still relies on the underlying large model, making LoRA more like a “trigger” or “catalyst,” thus making it very difficult for platform providers to “wash their hands” of responsibility.

In fact, this open platform model is fundamentally no different from Wikipedia or short video platforms, where content mainly comes from users. The problem is that the technology is provided by the platform, which ultimately forms products and services based on user-generated content and has sufficient control over the content provided by users.

It should be noted that the platform is the most important “handle” for regulating the internet industry in China, without exception. Under this premise, it is unrealistic to rely solely on the operation of an “open platform” to strip away the identity of being a “handle.” Some may argue that they provide a neutral technology to users, and how users use it is unrelated to the platform.

In reality, platforms are unlikely to simply provide technology; otherwise, they would not be called platforms. Generally speaking, platforms will provide scenarios for using the LoRA models trained by users, such as “squares” for sharing content generated by users’ models, and channels or even supporting functional interfaces and agreements for users to open their LoRA models to other users. To encourage deeper user participation, many platforms also incentivize users to upload data and share models through rewards, which all transform technology providers into platform providers.

There are also doubts about whether platforms can avoid these actions. Theoretically, they can, but in terms of business logic, it is unfeasible. The introduction of the LoRA model is aimed at solving the data scarcity problem for platforms training models. The ultimate effect is still to provide users with more accurate generative capabilities. Essentially, it can even be understood as “borrowing the small models of users to serve other users with the large model,” and if users are only allowed to train for their own use, the platform cannot accumulate a large user base to achieve profitability.

It is also important to consider that the data uploaded by users is generally stored on the platform’s own servers. This point is critical and will serve as evidence for the court to determine that the platform has control over the data (this is fundamentally different from the previous cases I represented, where the content was stored on the developer’s own servers, thus the platform could not precisely control the content on the mini-program).

3. Compliance Choices to Avoid Risks

The open large model platform based on LoRA technology is indeed a very good business model that can significantly address the issue of insufficient data while not affecting the generative effects presented to users. Compared to completely self-operated (self-training and providing model services) large model providers, the intellectual property risks from both the data and generation sides are lower. However, for a considerable number of well-known data sources with intellectual property rights, the platform still cannot fully shift responsibility to users. If the rights holders of the data raise infringement claims, the platform may bear infringement liability as a result.

Of course, in the specific determination of infringement, it may also be further subdivided into two main directions: copyright infringement and unfair competition. The former mainly focuses on the generation stage, where generating identical or similar content using well-known copyrighted content may constitute an infringement of the right of information network dissemination; the latter focuses more on the training + generation stage, primarily using well-known copyrighted content to train and generate a large amount of “similar but not identical” content, leading to the dilution or distortion of the original copyright’s recognizability.

The court will likely consider multiple factors, including the platform’s control over training data, the technical implementation principles of the LoRA model, the platform’s profit model, the platform’s incentive strategies for users, the sharing and recommendation mechanisms for user-trained models and generated content, and the platform’s safe harbor complaint mechanisms to determine platform liability. The compliance actions that the platform must take will essentially involve further scrutiny on these dimensions. However, regardless of the circumstances, the possibility of the platform hoping to achieve complete exemption from liability through this model is extremely slim.