Join the professional CV group at Jishi, and interact with 10,000+ visual developers from prestigious institutions like HKUST, Peking University, Tsinghua University, Chinese Academy of Sciences, CMU, Tencent, Baidu, and more!

We also provide monthly expert live streams, real project demand connections, valuable information summaries, and industry technical exchanges. Follow the Jishi Platform public account, reply join group, and apply to join immediately~

This article is translated from Deep Multi-Task Learning – 3 Lessons Learned by Zohar Komarovskyhttps://engineering.taboola.com/deep-multi-task-learning-3-lessons-learned/In recent years, Multi-Task Learning (MTL) has been widely used to solve various business problems at Taboola. In these business problems, a common set of features and deep learning models are used to address MTL-related issues. Here, I would like to share some insights we have learned while working on MTL.

Insight One: Integrating Loss Functions

The first challenge in MTL models: How to define a unified loss function for multiple tasks?The simplest approach is to integrate the loss functions of different tasks and simply sum them up. However, this method has some drawbacks, such as when the model converges, some tasks perform well while others do poorly. The underlying reason is that different loss functions have different scales; some loss functions have larger scales, which affect the performance of those with smaller scales. The solution to this problem is to replace the “simple summation” of multi-task loss functions with “weighted summation.” Weighting can make the scales of each loss function consistent, but it also brings new problems: the weighting hyperparameters are difficult to determine.Fortunately, a paper titled “Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics” adjusts the weighting hyperparameters in the loss functions using “uncertainty” so that the loss functions for each task have similar scales. The Keras version of this algorithm can be found onGitHub:https://github.com/yaringal/multi-task-learning-example/blob/master/multi-task-learning-example.ipynb

Insight Two: Adjusting Learning Rate

In the parameters of neural networks, the learning rate is a very important parameter. In practice, we found that a learning rate of 0.001 works well for task A, while a learning rate of 0.1 works well for task B. Choosing a larger learning rate can lead to dying ReLU in one task, while a smaller learning rate may cause the model to converge too slowly on some tasks. How can we solve this problem? For different tasks, we can use different learning rates. This may sound complicated, but it’s actually quite simple. Generally, the TensorFlow code to train a neural network is as follows:

optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)

Here, AdamOptimizer defines the method of gradient descent, and minimize calculates the gradients and minimizes the loss function. We can customize a minimize function to set an appropriate learning rate for the variables of a specific task.

all_variables = shared_vars + a_vars + b_vars
all_gradients = tf.gradients(loss, all_variables)

shared_subnet_gradients = all_gradients[:len(shared_vars)]
a_gradients = all_gradients[len(shared_vars):len(shared_vars + a_vars)]
b_gradients = all_gradients[len(shared_vars + a_vars):]

shared_subnet_optimizer = tf.train.AdamOptimizer(shared_learning_rate)
a_optimizer = tf.train.AdamOptimizer(a_learning_rate)
b_optimizer = tf.train.AdamOptimizer(b_learning_rate)

train_shared_op = shared_subnet_optimizer.apply_gradients(zip(shared_subnet_gradients, shared_vars))
train_a_op = a_optimizer.apply_gradients(zip(a_gradients, a_vars))
train_b_op = b_optimizer.apply_gradients(zip(b_gradients, b_vars))

train_op = tf.group(train_shared_op, train_a_op, train_b_op)

It’s worth mentioning that such tricks also work well in single-task neural networks.

Insight Three: Evaluating Task A as a Feature for Other Tasks

When we build an MTL neural network, the model’s estimate for task A can serve as a feature for task B. During forward propagation, this process is straightforward because the model’s estimate for A is a tensor that can simply be used as input for another task. However, during backpropagation, there are some differences. We do not want the gradient from task B to affect task A. Fortunately, TensorFlow provides an API tf.stop_gradient. When calculating gradients, we can treat certain tensors as constants rather than variables, so their values are not affected by gradients. The code is as follows:

all_gradients = tf.gradients(loss, all_variables, stop_gradients=stop_tensors)

Once again, it’s worth mentioning that this trick can be used not only in MTL tasks but also in many other tasks. For example, when training a GAN model, we do not need to backpropagate gradients into the generation process of adversarial samples.Thank you for reading, I hope this article is helpful to you! Thank you!-END–*Further Reading

• Sharing Practical Experience: Feeding Your GPU in Deep Learning

• Neural Network Structure Optimization: This Paper Will Help You Easily Train Deep Networks Without Fear of Vanishing or Exploding Gradients

• Amazon Team’s Tuning Secrets: A Collection of Practical Tricks for Image Classification with CNNs

Exclusive Benefits from JishiAI Mobile Application Competition with a prize pool of 400,000, participation guarantees rewards, additional rewards for finalists

Add the Jishi Assistant WeChat（ID: cv-mart）, note:Research Direction-Name-School/Company-City (e.g., AI Mobile Application-Xiao Ji-Peking University-Shenzhen), you can apply to join theAI Mobile Application Jishi Technical Exchange Group. You will also have access tomonthly expert live streams, real project demand connections, job referrals, algorithm competitions,information summaries, and industry technical exchanges, let’s shine our ideas further together~

△ Long press to add Jishi Assistant

△ Long press to follow the Jishi platform forthe latest CV insights