Join the professional CV group at Jishi, and interact with 10,000+ visual developers from top universities and companies such as HKUST, Peking University, Tsinghua University, Chinese Academy of Sciences, CMU, Tencent, Baidu!

We also provide monthly live sharing sessions with experts, real project demand connections, and a summary of valuable information for industry technical exchanges. Follow the Jishi Platform official account, reply join group, and apply to join immediately~

This article is authorized to be reprinted from Zhihu author Anticoder, https://zhuanlan.zhihu.com/p/59413549. Unauthorized reproduction is prohibited.

Background: Focusing only on a single model may overlook potential information that could enhance the target task from related tasks. By sharing parameters between different tasks to a certain extent, the original task may generalize better. Generally speaking, as long as there are multiple losses, it counts as MTL, with some aliases (joint learning, learning to learn, learning with auxiliary tasks).

Objective: To enhance the model’s generalization and performance by balancing training information from the main task and related auxiliary tasks. From a machine learning perspective, MTL can be seen as a form of inductive transfer (prior knowledge), improving model performance through inductive bias (a certain prior assumption about the model). For example, using L1 regularization, we bias the model towards a sparse solution (fewer parameters). In MTL, this prior is provided through auxiliary tasks, making it more flexible and guiding the model to favor some other tasks, ultimately leading to better generalization.

MTL Methods for DNN

• hard parameter sharing (this method is already 26 years old <1993>)

Share some parameters (generally at the lower layers) across all tasks, while using unique parameters at the specific task layer (top layer). In this case, the chance of overfitting with shared parameters is relatively low (compared to non-shared parameters), with the probability of overfitting being O(#tasks). [1]

• soft parameter sharing

Each task has its own parameters, and finally constraints are added to express similarity between the differences in parameters of different tasks. For example, using L2, trace norm, etc.

Advantages and Use Cases

1. implicit data augmentation: Each task has some sample noise, and different tasks may have different noise, ultimately learning multiple tasks can cancel out some noise (similar to the bagging idea, where noise exists in various directions and averaging tends to zero).
2. Some tasks with significant noise or insufficient training samples, high dimensions, may inhibit the model from effectively learning or even learning relevant features.
3. Certain features may be difficult to learn in the main task (e.g., only existing high-order correlations or suppressed by other factors) but can be learned well in auxiliary tasks. Auxiliary tasks can be used to learn these features, methods such as hints (predicting important features) [2].
4. By learning a sufficiently large hypothesis space, the model can perform well on certain new tasks in the future (solving cold start), provided these tasks are homogeneous.
5. As a regularization method to constrain the model. The so-called inductive bias. Alleviate overfitting, reducing the model’s Rademacher complexity (the ability to fit noise, used to measure the model’s capacity).

Traditional MTL Methods (linear models, kernel methods, Bayesian algorithms) mainly focus on two points:

1. Achieving sparsity between tasks through norm regularization.
2. Modeling the relationships between multiple tasks.

1.1 Block-sparse regularization (mixed l1/lq norm)

Objective: Force the model to consider only some features, under the premise that different tasks must be related.

Assume K tasks share the same features and the same number of model parameters. Form a matrix A(DxK), where D is the parameter dimension and K is the number of tasks, aiming for these tasks to use only some features, meaning some rows of A should also be zero. (The simplest idea is to make it a low-rank matrix; or use L1 regularization because L1 can constrain certain features to zero. If we want to make certain rows zero, we can first perform row aggregation and then apply L1 on the aggregated result, for specifics refer to article [3]. Typically, using lq norm to constrain rows (each feature) first, followed by L1 norm is referred to as mixer l1/lq norm.

Development:

1. group lasso [4]: l1/l2 norm, addressing l1/l2 norm non-convexity through trace norm; subsequently, someone proposed an upper bound for using group lasso in MTL [5].
2. When multiple tasks share few common features, l1/lq norm may not perform as well as element-wise norm. Someone proposed combining these two methods, decomposing the parameter matrix into A = S + B, applying lasso to S and using l1/l_infinite for B. [6]
3. distributed version of group-sparse regularization [7].

2.1 Regularization for Learning Task Relationships

When the correlation between tasks is weak, using the above methods may lead to negative transfer (i.e., negative effects). In this scenario, we hope to add prior knowledge that certain tasks are related, while some tasks have poor correlation. Task clustering can be introduced to constrain the model. Different tasks’ parameter vectors and their variances can be penalized. Limiting different models to approach their respective cluster mean vector.

Similarly, for SVM, Bayesian methods can be introduced to pre-specify some clusters, aiming to maximize margin while making different tasks approach their respective cluster centers; [8]

Once clusters are specified, models can be constrained using clustering methods (intra-class, inter-class, and their complexity).

In some scenarios, tasks may not appear in the same cluster, but there may be potential similar structures, such as group-lasso in tree structures and graph structures.

2.2 Other Methods for Learning Task Relationships

1. KNN methods for task clustering. [9]
2. Semi-supervised learning for learning common structures of some related tasks. [10]
3. Multi-task BNN, controlling multi-task similarity through priors, model complexity is high, can use sparse approximation to greedily select samples [11]; Gaussian processes using the same covariance matrix and prior across different tasks (thus also reducing complexity) [12].
4. Using Gaussian priors for each task-specific layer can utilize a mixed distribution of a cluster (pre-defined) to encourage similarity between different tasks [13].
5. Furthermore, sampling distributions through a Dirichlet process allows the model to reflect the similarity between tasks and the number of clusters. Tasks in the same cluster use the same model [14].
6. hierarchical Bayesian model, learning a latent task structure [15].
7. MTL extension of the regularized Perceptron, encodes task relatedness in a matrix. This can then be constrained through different regularizations (e.g., rank) [16].
8. Different tasks belong to different independent clusters, each cluster existing in a low-dimensional space, with each cluster’s tasks sharing the same model. By alternating iterations, learn the distribution weights of different clusters and each cluster’s model weights. Assuming absolute independence between tasks may not be ideal [17].
9. Assuming there is overlap between two tasks in two different clusters, there exists a portion of latent basis tasks. Let the model parameters for each task be a linear combination of the latent basis tasks, constraining the latent basis tasks to be sparse. The overlapping part controls the degree of sharing [18].
10. Learning a small number of shared hypotheses, then mapping each task to a single hypothesis [19].

MTL in DNN

Deep Relation Network [20]

In computer vision, generally share convolutional layers, followed by task-specific DNN layers. By setting priors on task layers, the model learns the relationships between tasks.

Fully-Adaptive Feature Sharing [21]

Starting from a simple structure, greedily and dynamically widening the model to cluster similar models. Greedy methods may not learn a globally optimal structure; each branch for one task cannot learn the complex relationships between tasks.

Cross-stitch Networks [22]

soft parameter sharing, learning the previous layer’s outputs through linear combinations, allowing the model to determine the degree of sharing between different tasks.

Low supervision [23]

Finding better multi-task structures, the lower levels of complex tasks should be supervised by the goals of lower-level tasks (e.g., in NLP, the early layers learn an NER or POS auxiliary task).

A Joint Many-task Model [24]

Pre-setting hierarchical structures for multiple NLP tasks, followed by joint learning.

Weighting losses with uncertainty [25]

Not considering learning shared structures, but considering the uncertainty of each task. By optimizing loss (Gaussian likelihood with task-dependent uncertainty), adjusting the similarity between different tasks.

Tensor factorisation for MTL [26]

Decomposing parameters for each layer into shared and task-specific.

Sluice Networks [27]

A mix (hard parameter sharing + cross-stitch networks + block-sparse regularization + task hierarchy (NLP)), allowing the model to learn which layers and subspaces to share and where to find the optimal representation of inputs.

When different tasks have high correlations and approximately follow the same distribution, sharing parameters is beneficial. But what about tasks with low or no correlation?

Early work involved pre-specifying which layers to share for each pair of tasks, which has poor scalability and a biased model structure; when task correlations decrease or require different levels of reasoning, hard parameter sharing becomes ineffective.

Currently popular methods include learning what to share (outperforming hard parameter sharing); and learning the task hierarchy is also useful when tasks have multi-granularity factors.

Auxiliary Tasks

We focus only on the main task’s objectives but hope to benefit from other effective auxiliary tasks!

Currently, several methods for selecting auxiliary tasks are available

1. Related tasks: Conventional approaches (autonomous driving + road sign recognition; query classification + web search; coordinate prediction + object recognition; duration + frequency).
2. Adversarial: In domain adaptation, related tasks may not be accessible, and adversarial tasks can be used as negative tasks (maximizing training error), for instance, predicting the domain of inputs as an auxiliary task would lead to the main task model learning representations that cannot distinguish between different domains.
3. Hints: Some features mentioned earlier may be difficult to learn in certain tasks, selecting auxiliary tasks for predicting features (e.g., in NLP, the main task is sentiment prediction, and the auxiliary task is whether inputs contain positive or negative words; the main task is name error detection, and the auxiliary task is whether there are names in the sentence).
4. Focusing attention: Making the model pay attention to parts that may be difficult to notice in the task (e.g., autonomous driving + road sign detection; facial recognition + head position recognition).
5. Quantization smoothing: In certain tasks, training targets are highly discretized (artificial scoring, sentiment scoring, disease risk levels), using auxiliary tasks with lower discretization may be helpful because the smoother target makes the task easier to learn.
6. predicting inputs: In some scenarios, certain features may not be selected because they are detrimental to predicting targets, but these features might help the model’s training; in such cases, these features can be used as outputs rather than inputs.
7. Using the future to predict the present: Some features only become available after a decision, for example, in autonomous driving, data about objects is only obtained when the vehicle passes them; in medicine, the effects of drugs are only known after use. These features cannot serve as inputs but can act as auxiliary tasks to provide information to the main task during training.
8. representation learning: Auxiliary tasks often implicitly learn some feature representations that benefit the main task. They can also explicitly learn (using an auxiliary task to learn transferable feature representations, such as AE).

So, which auxiliary tasks are useful?

The assumption behind auxiliary tasks is that they should be somewhat related to the main task and beneficial for its learning.

How to measure the correlation between two tasks?

Some theoretical studies:

1. Using the same features for decision-making.
2. Related tasks share the same optimal hypothesis space (having the same inductive bias).
3. F-related: If the data of two tasks is obtained through a fixed distribution with some transformations [28].
4. Classification boundaries (parameter vectors) are close.

Task similarity is not binary; the more similar the tasks, the greater the benefit. Learning what to share allows us to temporarily overlook theoretical shortcomings, enabling gains even between poorly correlated tasks. However, developing task similarity is crucial for our selection of auxiliary tasks.

MTL Learning Tips

1. Auxiliary tasks with compact and uniformly distributed labels are better (from POS in NLP) [29].
2. The main task training curve stabilizes faster, while auxiliary tasks stabilize more slowly (not yet stable) [30].
3. Different tasks may have different scales, and the optimal learning rate for tasks may vary.
4. The output of one task can serve as input for some tasks.
5. Some tasks may have different iteration cycles, requiring asynchronous training (posterior information; feature selection, feature derivation tasks, etc.).
6. The overall loss may be dominated by certain tasks, requiring dynamic adjustments to parameters throughout the cycle (by introducing some uncertainty, each task learns a noise parameter, unifying all losses) [31].
7. Some estimates can serve as features (e.g., alternating training).

Conclusion

More than 20 years of hard parameter sharing is still popular, and the current hot topic of learning what to learn is also valuable. Our understanding of tasks (similarity, relationships, hierarchy, benefits for MTL) is still quite limited, and we hope for significant developments in the future.

Research Directions

1. learning what to share
2. measurement for similarity of tasks
3. using task uncertainty
4. introducing asynchronous tasks (feature learning tasks), adopting alternating iterative training
5. learning abstract sub-tasks; learning task structures (similar to hierarchy learning in reinforcement)
6. parameter learning auxiliary tasks
7. More…

Note: The learning materials for this article mainly come from _An Overview of Multi-Task Learning in Deep Neural Networks, https://arxiv.org/abs/1706.05098

Reference

[1] A Bayesian/information theoretic model of learning to learn via multiple task sampling. http://link.springer.com/article/10.1023/A:1007327622663

[2] Learning from hints in neural networks. Journal of Complexity https://doi.org/10.1016/0885-064X(90)90006-Y

[3] Multi-Task Feature Learning http://doi.org/10.1007/s10994-007-5040-8

[4] Model selection and estimation in regression with grouped variables

[5] Taking Advantage of Sparsity in Multi-Task Learning http://arxiv.org/pdf/0903.1468

[6] A Dirty Model for Multi-task Learning. Advances in Neural Information Processing Systems https://papers.nips.cc/paper/4125-a-dirty-model-for-multi-task-learning.pdf

[7] Distributed Multi-task Relationship Learning http://arxiv.org/abs/1612.04022

[8] Regularized multi-task learning https://doi.org/10.1145/1014052.1014067

[9] Discovering Structure in Multiple Learning Tasks: The TC Algorithm http://scholar.google.com/scholar?cluster=956054018507723832&hl=en

[10] A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

[11] Empirical Bayes for Learning to Learn

[12] Learning to learn with the informative vector machine https://doi.org/10.1145/1015330.1015382

[13] Task Clustering and Gating for Bayesian Multitask Learning https://doi.org/10.1162/153244304322765658

[14] Multi-Task Learning for Classification with Dirichlet Process Priors

[15] Bayesian multitask learning with latent hierarchies http://dl.acm.org.sci-hub.io/citation.cfm?id=1795131

[16] Linear Algorithms for Online Multitask Classification

[17] Learning with whom to share in multi-task feature learning

[18] Learning Task Grouping and Overlap in Multi-task Learning

[19] Learning Multiple Tasks Using Shared Hypotheses

[20] Learning Multiple Tasks with Deep Relationship Networks http://arxiv.org/abs/1506.02117

[21] Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification http://arxiv.org/abs/1611.05377

[22] Cross-stitch Networks for Multi-task Learning https://doi.org/10.1109/CVPR.2016.433

[23] Deep multi-task learning with low level tasks supervised at lower layers

[24] A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks http://arxiv.org/abs/1611.01587

[25] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics http://arxiv.org/abs/1705.07115

[26] Deep Multi-task Representation Learning: A Tensor Factorisation Approach https://doi.org/10.1002/joe.20070

[27] Sluice networks: Learning what to share between loosely related tasks http://arxiv.org/abs/1705.08142

[28] Exploiting task relatedness for multiple task learning. Learning Theory and Kernel Machines https://doi.org/10.1007/978-3-540-45167-9_41

[29] When is multitask learning effective? Multitask learning for semantic sequence prediction under varying data conditions http://arxiv.org/abs/1612.02251

[30] Identifying beneficial task relations for multi-task learning in deep neural networks http://arxiv.org/abs/1702.08303

[31] Multitask learning using uncertainty to weigh losses for scene geometry and semantics

Recommended Reading:

• Connecting Multiple Visual Tasks with a Versatile Backbone: HRNet

• CVPR 2020 | New SOTA for Multi-task Image Classification, New Algorithm IR-Net Proposed by Beihang University, SenseTime, etc.

• Summary of Multi-task Learning in Deep Neural Networks

Add Jishi Assistant WeChat（ID: cvmart2）, note:Research direction – Name – School/Company – City (e.g., Object Detection – Xiao Ji – Peking University – Shenzhen) to apply to join Jishi technical group for Object Detection, Object Tracking, Face Recognition, Industrial Detection, Medical Imaging, 3D & SLAM, Image Segmentation, etc., along withmonthly expert live sharing sessions, real project demand connections, job referrals, algorithm competitions,valuable information summaries, and industry technical exchanges, let’s illuminate the path of thought further together~

△ Long press to add Jishi Assistant

△ Long press to follow Jishi platform, getthe latest CV insights