1. Team: HuggingFace H42. Paper Title: Zephyr: Direct Distillation of LM Alignment3. Paper Link:4. GitHub: https://github.com/huggingface/alignment-handbookTable of Contents: 1. Introduction2. Core Algorithms 2.1 Distilled Supervised Fine-Tuning (dSFT) 2.1.1 Traditional Methods 2.1.2 New Methods 2.1.3 Example Explanation (to aid understanding of the dSFT execution process) 2.2 AI Feedback through Preferences (AIF) 2.2.1 Traditional Methods 2.2.2 New Methods 2.2.3 Specific Steps of AIF 2.3 Distilled Direct Preference Optimization (dDPO) 2.3.1 Objectives 2.3.2 Reward Functions 2.3.3 Direct Preference Optimization (DPO) 2.3.4 Key Observations and Mathematical Formulas 2.3.5 Objective Functions 2.3.6 Training Process 2.3.7 Example Explanation: (to aid understanding of the differences between PPO and DPO)-
Introduction
The goal of this research is to create a smaller language model that aligns more closely with user intent. By applying Distilled Supervised Fine-Tuning (dSFT) and Distilled Direct Preference Optimization (dDPO), along with utilizing preference data from AI Feedback (AIF), the researchers successfully improved the model’s task accuracy and intent alignment. The ZEPHYR-7B model, with 7 billion parameters, has set a new standard in chat benchmarks without the need for manual annotation, surpassing previous models in the MT-Bench test. The advantages of this method include shorter training times and no need for additional sampling, providing a new direction for the development and fine-tuning of open large language models (LLMs). However, the study did not consider model safety issues, such as the potential for harmful outputs.
2. Core Algorithms
The objective of this method is to align an open-source large language model with user intent. In this work, the researchers assume access to a larger teacher model that can be queried through prompt generation. Their goal is to produce a student model, and their method follows stages similar to InstructGPT, as shown in Figure 2.
This figure describes a three-step method to further train and optimize the AI model. Below is a detailed explanation of each step:
This figure describes a three-step method to further train and optimize the AI model, and below is a detailed explanation of each step:
1. dSFT (Distilled Supervised Fine-Tuning) Step:① Generate multi-turn AI dialogues: Start by randomly selecting a prompt from a dataset.② Create a scenario: For example, the AI is asked to create a scenario for a game about space exploration.③ LLM (Language Learning Model) simulates multi-turn user-assistant interactions: The LLM generates multi-turn dialogues for the scenario, simulating real interactions between users and assistants. These dialogues are then used for supervised fine-tuning.
2. AIF (AI Feedback) Step:① Response generation and AI ranking: – Sample a prompt from the dataset, such as, “Describe how to make chocolate brownies.”- Four different language models generate responses to this prompt.- GPT-4 (an advanced language model) scores and ranks these responses.
3. dDPO (Distilled Direct Preference Optimization) Step:① Refining AI preferences: – For the same prompt, such as, “Describe how to make chocolate brownies,” select the best response and another random response. – These two responses are used for direct preference optimization, where the model learns to optimize its preferences based on feedback data. – In this step, the model attempts to understand which response is preferred and adjusts accordingly.
These three steps combine to form a complex AI training and optimization method. First, basic model training is conducted through dSFT, then feedback is collected and the model is scored and ranked through AIF, and finally, the model’s preferences are optimized based on this feedback through dDPO.
2.1 Distilled Supervised Fine-Tuning (dSFT)
2.1.1 Traditional Methods
Completed through supervised fine-tuning (SFT) on a high-quality dataset of instructions and responses.
2.1.2 New Methods
Given the ability to access a teacher language model to generate instructions and responses, we can train directly on these datasets.
① The dSFT method follows the self-instruct protocol, which requires constructing a set of seed prompts representing various subject areas. ② The dataset is constructed through iterative self-prompting, where the teacher model is used to respond to an instruction and optimize this instruction based on the response.③ For each seed prompt, a response is first sampled from the teacher model, then a prompt is used to refine it, and a new instruction is sampled from the teacher model again.④ The ultimate goal is to obtain a dataset containing pairs of instructions and responses, such as: C = {(x1, y1), …, (xj, yj)}⑤ Finally, distillation is performed through the SFT method.
Key Formula-1: Preference Model
The purpose of the formula is to maximize the model’s log-likelihood of the instruction-response pairs in the given dataset C during training. This helps the model learn better how to respond to user prompts, thereby improving the quality of its outputs.
Formula Explanation: (Screenshot from my prepared mind map, WeChat public account does not support LaTex syntax)
2.1.3 Example Explanation (to aid understanding of the dSFT execution process)
(1) Background Assume we are training a simplified language model whose task is to answer weather-related questions. We want to use the dSFT method to further optimize it.
(2) Seed Prompts We start with the following three seed prompts: 1. Describe today’s weather.2. Will it rain tomorrow?3. What is the temperature range for this week?
(3) Iterative Self-Prompting
Step 1:We use an existing teacher model (assumed to be GPT-4) to respond to these seed prompts, with the following responses:
1. For “Describe today’s weather,” the teacher model might respond, “Today is sunny, with a temperature of about 25°C.” 2. For “Will it rain tomorrow?” the teacher model might respond, “There is a 60% chance of rain tomorrow.” 3. For “What is the temperature range for this week?” the teacher model might respond, “The temperature range for this week is from 20°C to 30°C.”
Step 2:Next, based on each seed prompt and corresponding response, we generate new prompts.
1. Based on “Today is sunny, with a temperature of about 25°C,” a new prompt might be, “What are today’s high and low temperatures?” 2. Based on “There is a 60% chance of rain tomorrow,” a new prompt might be, “What is the predicted rainfall for tomorrow?” 3. Based on “The temperature range for this week is from 20°C to 30°C,” a new prompt might be, “Which day is the coldest this week?”
Step 3:These new prompts are again provided to the teacher model, generating corresponding responses. This process can be repeated until the desired dataset size is reached.
(4) Final Dataset
After several rounds of iteration, we might obtain the following dataset:
1. “Describe today’s weather,” “Today is sunny, with a temperature of about 25°C.” 2. “Will it rain tomorrow?” “There is a 60% chance of rain tomorrow.” 3. “What is the temperature range for this week?” “The temperature range for this week is from 20°C to 30°C.” 4. “What are today’s high and low temperatures?” “The high is 28°C and the low is 22°C.” 5. “What is the predicted rainfall for tomorrow?” “The expected rainfall for tomorrow is 5mm.”
6. “Which day is the coldest this week?” “Friday is the coldest, with a predicted low of 20°C.”
This is a simplified example of the dSFT dataset construction process. In practical applications, seed prompts, responses, and new prompts will be more diverse and may involve more complex topics and domains.
2.2 AI Feedback through Preferences (AIF)
2.2.1 Traditional Methods
Human feedback (HF) can provide additional signals for aligning LLMs, where people typically give feedback based on their preferences for LLM responses.
2.2.2 New Methods
For distillation, we use AI preferences from outputs generated by the teacher model (assumed to be GPT-4). The UltraFeedback method uses the teacher model to provide preferences for model outputs.
2.2.3 Specific Steps of AIF (Screenshot from my prepared mind map, WeChat public account does not support LaTex syntax)
This method is fundamentally aimed at obtaining better teacher model feedback through the model’s responses. By comparing responses from multiple models and using the teacher model to score these responses, the system can better understand which responses are preferred. This way, the model can learn human preferences more accurately and train and optimize accordingly.
2.3 Distilled Direct Preference Optimization (dDPO)
2.3.1 Objectives
The goal of dDPO is to optimize a specific model πdSFT to rank expected outputs yw better relative to other outputs yl in a so-called “preference model.” This is essentially a ranking problem, where we want the model to more frequently choose yw over yl.
2.3.2 Reward Functions
The reward function is defined as r(x,y), which determines how well the model’s predictions perform given input x and output y. This function utilizes a model called the “student language model.”
2.3.3 Direct Preference Optimization (DPO)
Traditional techniques use a reinforcement learning method, first training the reward and then sampling to compute updates. However, DPO takes a direct approach: it optimizes the preference model directly from so-called “static data.” Static data refers to historical user behavior data, purchase records, or other information that does not change over time.
2.3.4 Key Observations and Mathematical Formulas
The core of the DPO method is to derive the relationship between two strategies:
Strategy-1: Optimal LLM Strategy π∗
This is the strategy that is most likely to lead to user satisfaction under given conditions.
Strategy-2: Original LLM Strategy πdSFT
This is a baseline strategy that may be formulated based on previous data or experience.
Given the above strategies, the reward function:
2.3.5 Objective Functions
When we insert the reward function into the preference model, we can obtain the following objective function:
This formula essentially states: we want to maximize the likelihood of choosing yw over yl given input x.
2.3.6 Training Process
This section describes how to train the model using the above objective function:
For each AIF triplet (x, yw, yl), the calculation process is as follows:
① Calculate the probabilities of (x, yw) and (x, yl) in the dSFT model;
② Calculate the probabilities of (x, yw) and (x, yl) in the dDPO model;
Calculate the loss based on the above objective function (Formula-1) and perform backpropagation to update the model.
2.3.7 Example Explanation: (to aid understanding of the differences between PPO and DPO)
Assume we are training an agent in a video game whose task is to find treasure in a maze.
(1) Traditional Method: PPO (Proximal Policy Optimization)
-
Strategy:
PPO optimizes the agent’s strategy iteratively to improve its ability to find treasure in the maze.
-
Workflow:
① The agent acts in the maze according to its current strategy, collecting a series of experience data. ② This experience data is used to update the strategy, ensuring that the new strategy does not deviate too much from the old one (this is what “proximal” means). ③ Repeat the above steps until the agent can effectively find the treasure.
(2) New Method: DPO (Direct Preference Optimization)
-
Strategy:
DPO directly optimizes the agent’s preference model from existing static data.
-
Workflow:
① Assume we already have a set of historical data of agents finding treasure in the maze. This data includes various actions taken by the agent in given states and their outcomes. ② DPO does not directly update the strategy but updates a preference model that predicts which action is more preferred by the agent in a given state (or which action is more likely to find treasure). ③ Based on the preference model, the agent can choose the action it deems best. This method’s advantage is that it can learn directly from static data without needing to interact with the environment repeatedly like PPO.
For the experimental content, it is recommended to refer directly to the paper, which contains detailed data comparisons.