MLNLP ( Machine Learning Algorithms and Natural Language Processing ) community is a well-known natural language processing community both domestically and internationally, covering NLP master’s and doctoral students, university professors, and corporate researchers.The vision of the community is to promote communication between the academic and industrial circles of natural language processing and machine learning, as well as enthusiasts, especially the progress of beginners.
This article is reproduced from | Zhihu
Author | FlagAI
Source | https://zhuanlan.zhihu.com/p/533381273
1
Introduction
Meta AI has open-sourced the OPT series models, with the largest model containing 175 billion parameters (access permission required), comparable to GPT-3. The OPT series models include multiple sets of model weights with different parameter sizes, as shown in the figure:
OPT has open-sourced a series of large models, but there is a high technical threshold to actually call these models. To promote the technology of large models, FlagAI allows one-click loading of different versions of the OPT model and will continue to improve it. This article mainly provides a brief introduction to the structure of the OPT model, lists the difficulties encountered in reproducing the OPT model, and provides example code for quickly loading the OPT model using FlagAI. For a more detailed introduction to FlagAI, please refer to the one-click calling solution for large models! The Beijing Zhiyuan Artificial Intelligence Research Institute has launched the NLP large model training framework FlagAI
FlagAI repository address: https://github.com/BAAI-Open/FlagAI
OPT model paper address: https://arxiv.org/pdf/2205.01068.pdf
Github repository address: https://github.com/facebookresearch/metaseq
175B model access application address: https://forms.gle/dag8g7nKiR4o4VZq5
2
Background
OPT stands for Open Pre-trained Transformer Language Models. We all know that GPT-3 has very impressive effects, and there are many interesting demos online; however, its model weights are not open-sourced, and we cannot see its internal principles; this time, Meta AI has directly open-sourced the OPT model with hundreds of billions of parameters, comparable to GPT-3, achieving comparable results to GPT-3 on multiple tasks, whether in zero-shot or multi-shot settings (the left image shows zero-shot, and the right image shows multi-shot):
It is worth noting that in terms of model structure, both OPT and the GPT series models adopt the Transformer-Decoder structure and use a lower triangular mask, thus performing autoregressive predictions from left to right during decoding.
3
Quickly Download and Use the OPT Series Models with FlagAI
Currently, FlagAI has integrated models including opt125m, opt350m, opt1.3b, opt2.7b, opt6.7b, opt13b, and opt30b, with larger parameter models being supported successively. It only takes two lines of code to load directly: if you have already installed FlagAI (pip install -U flagai), you can directly download and use it with the above code. The download process for the opt-13b model is shown in the figure:
The generated result is:
Input:
How about the book The Old Man and the Sea?
Thanks for your question, let me share my thoughts:
Output:
I think that "The Old Man and the Sea" is a great read.
It has many themes which are very interesting but also relatable in some cases (not only for readers).
In conclusion, I would definitely recommend this book to everybody.Loading and Inference for Models Over 10 Billion Parameters
When the model parameters exceed 10 billion, a single card may encounter OOM issues. Therefore, you can use CPU loading/multi-GPU loading methods.
CPU Loading Method
from flagai.model.predictor.predictor import Predictor
from flagai.auto_model.auto_loader import AutoLoader
loader = AutoLoader(task_name="lm",
model_name="opt-30b-en")
model = loader.get_model()
tokenizer = loader.get_tokenizer()
model.eval()
text = "The trophy doesn’t fit in the suitcase because "
predictor = Predictor(model, tokenizer)
out = predictor.predict_generate_randomsample(text,
input_max_length=100,
out_max_length=300,
top_k=30,
top_p=0.9,
repetition_penalty=3.0)
print(f"input is {text} \n out is {out}")CPU Test Results & Resource Usage
The 30b model has 63G, using 36 CPUs and 120G of memory, running for 18 minutes (excluding the time for model download), resulting in the following output.
4
Multi-GPU Loading Method
For a 63G model, at least three 32G V100 cards are needed to load without using fp16. By using model parallelism, the model weights can be loaded onto different graphics cards. FlagAI adopts the operator splitting method of Megatron-LM, and by setting the parameter model_parallel_size, the model can be automatically split across multiple graphics cards. Here, to demonstrate the effect, we set model_parallel_size=4; for more details, refer to the file link below:
FlagAI/opt_30b_en_mutigpu.py at master · BAAI-Open/FlagAI (github.com)
Multi-GPU Test Results & Resource Usage
The main time consumption is in model splitting (18 minutes), and then the model loading and inference took 22 minutes (Note: the machine we tested was shared by multiple users, causing longer loading time, which does not represent the actual level).
Differences Between OPT and GPT-2 Models
The OPT model and the GPT-2 model both use the Transformer-Decoder structure. To maximize code reuse, the following aspects need to be unified:
-
The config configuration files need to be unified: The configuration file fields of the OPT model and the GPT-2 model are different; to share code, the configuration file fields need to be unified;
-
The model structure needs to be unified: Although both are Transformer-Decoder structures, there are still some differences in details; for example, the position encoding of the OPT model is quite special (calculated using the attention mask); the GPT-2 model uses learnable absolute position encoding; the OPT model can modify the position of the layer normalization layer through configuration file fields (before or after the Attention and MLP layers); the dimensionality of the word embedding layer and the hidden dimension of the OPT model can sometimes be different, while in the GPT-2 model, they are the same.
-
A custom pre-trained weight conversion function is needed; due to the significant differences in the pre-trained weight structures of the two models, a custom weight conversion function is required to load the pre-trained weights of both the OPT and GPT series models simultaneously.
By unifying the above points, it is possible to achieve unified invocation of the OPT and GPT models. If you are interested in more detailed technical details on this, please leave a message, and we will share more detailed introductions.
Possible Garbled Issues with OPT125m
Experiments have shown that whether using Hugging Face or FlagAI to load the OPT125m model, there will always be some garbled characters during decoding, as shown in the figure:
The related issue has been submitted to the Transformers repository:
https://github.com/huggingface/transformers/issues/17735
No issues have been found with models of other parameter sizes.
Postscript: This indeed has a problem due to the omission of a final_layer_norm layer, which Hugging Face has now fixed.
https://github.com/huggingface/transformers/pull/17785
5
Conclusion
Meta AI has open-sourced the largest 175B parameter OPT series model, comparable to GPT-3, marking the first time in the industry that such a large parameter scale pre-trained model has been open-sourced, allowing us to better understand the internal workings of large models and validate model parallelism and other acceleration technologies.
FlagAI will continue to update support for large models in the NLP/CV fields, simplifying the download and usage process. We welcome everyone to download and use it, and if you have any questions during the process, feel free to communicate with us.
Technical Exchange Group Invitation
△ Long press to add assistant
Scan the QR code to add the assistant WeChat
Please note: Name – School/Company – Research Direction(e.g., Xiao Zhang – Harbin Institute of Technology – Dialogue System)to apply to join the Natural Language Processing/Pytorch and other technical exchange groups
About Us
MLNLP Community ( Machine Learning Algorithms and Natural Language Processing ) is a grassroots academic community jointly built by scholars in the field of natural language processing, which has developed into a well-known natural language processing community both domestically and internationally. It includes well-known brands such as Ten Thousand People Top Conference Exchange Group, AI Selection Exchange, AI Talent Exchange and AI Academic Exchange , aiming to promote progress between the academic and industrial circles of machine learning and natural language processing and a wide range of enthusiasts.The community can provide an open communication platform for practitioners’ further study, employment, research, and more. Everyone is welcome to follow and join us.
