Background: Focusing only on a single model may overlook potential information that could enhance the target task from related tasks. By sharing parameters between different tasks to some extent, the original task may generalize better.Generally speaking, as long as there are multiple losses, it counts as MTL, with some aliases (joint learning, learning to learn, learning with auxiliary tasks).

Objective: To improve the generalization and performance of the model by balancing training information from the main task and related auxiliary tasks. From a machine learning perspective, MTL can be seen as an inductive transfer (prior knowledge), which improves model performance by providing inductive bias (a certain prior assumption about the model). For example, by using L1 regularization, we bias the model towards a sparse solution (fewer parameters). In MTL, this prior is provided through auxiliary tasks, which is more flexible and informs the model to lean towards some other tasks, ultimately leading to better generalization.

1 MTL Methods for DNN

• hard parameter sharing (this method is already 26 years old <1993>)

Share some parameters across all tasks (usually at the lower layers), while using unique parameters for specific task layers (upper layers). In this case, the risk of overfitting with shared parameters is relatively low (compared to non-shared parameters), and the probability of overfitting is O(#tasks). [1]

• soft parameter sharing

Each task has its own parameters, and finally constraints are added to the differences between parameters of different tasks to express similarity. For instance, L2, trace norm, etc., can be used.

Advantages and Use Cases

1. implicit data augmentation: Each task has some sample noise, and different tasks may have different noise, ultimately, learning from multiple tasks can mitigate some noise (similar to the idea of bagging, where noise exists in various directions, and averaging will tend towards zero).
2. Some tasks with significant noise or insufficient training samples, or high dimensionality, may not allow the model to learn effectively, or even learn relevant features.
3. Certain features may be difficult to learn in the main task (for instance, existing only in very high-order correlations, or suppressed by other factors), but are easier to learn in auxiliary tasks. These features can be learned through auxiliary tasks, methods such as hints (predicting important features) [2].
4. By learning a sufficiently large hypothesis space, the model can perform well on certain new tasks in the future (solving cold start), provided these tasks are of the same origin.
5. As a form of regularization, constraining the model. The so-called inductive bias. Mitigating overfitting, reducing the model’s Rademacher complexity (the ability to fit noise, used to measure the model’s capability).

MTL in Traditional Methods (linear model, kernel methods, Bayesian algorithms) mainly focuses on two points:

1. Using norm regularization to ensure sparsity across tasks.
2. Modeling the relationships between multiple tasks.

1.1 Block-sparse regularization (mixed l1/lq norm)

Objective: To force the model to only consider certain features, under the premise that different tasks must be related.

Assuming K tasks share the same features and the same number of model parameters, forming a matrix A(DxK), where D is the parameter dimension and K is the number of tasks, the goal is for these tasks to only use some features, meaning certain rows of A must be zero. (The simplest idea is to make it a low-rank matrix; or use L1 regularization, as L1 can constrain certain features to zero. If we want to make certain rows zero, we just need to perform a row aggregation operation first and then apply L1 to the aggregated result, for specifics refer to the article [3]. Typically, using lq norm to constrain rows (each feature) first, and then using L1 norm to further constrain, is known as mixed l1/lq norm.

Development:

1. group lasso [4]: l1/l2 norm, solving the non-convexity of l1/l2 norm through trace norm; later, someone proposed an upper bound for using group lasso in MTL [5].
2. When there are not many common features among multiple tasks, l1/lq norm may not perform as well as element-wise norm. Some have proposed combining these two methods, decomposing the parameter matrix as A = S + B, using lasso on S, and l1/l_infinite on B. [6]
3. distributed version of group-sparse regularization [7].

2.1 Regularization way for learning task relationships

When the correlation between tasks is weak, using the above methods may lead to negative transfer (i.e., negative effects). In this scenario, we hope to increase the prior knowledge that some tasks are related, while others are less related. This can be achieved by introducing task clustering to constrain the model. We can penalize the parameter vectors of different tasks and their variance, limiting different models to converge towards their respective cluster mean vectors.

Similarly, for instance, in SVM, introducing Bayesian methods, to pre-specify some clusters, the goal is to maximize the margin while making different tasks converge towards their respective cluster centers; [8]

Once clusters are specified, model constraints can be applied through clustering methods (intra-class, inter-class, and their own complexity).

In some scenarios, tasks may not appear in the same cluster, but possess potential similar structures, such as group-lasso in tree structures and graph structures of tasks.

2.2 Other methods for learning task relationships

1. KNN methods for task clustering. [9]
2. semi-supervised learning for learning common structures of some related tasks. [10]
3. Multi-task BNN, controlling the similarity of multiple tasks through priors, with a large model complexity, can use sparse approximation to greedily select samples [11]; Gaussian processes by using the same covariance matrix and the same prior among different tasks (thereby reducing complexity) [12].
4. Using Gaussian priors for each task-specific layer, a mixed distribution of a cluster (predefined) can be used to promote similarity among different tasks [13].
5. Furthermore, sampling distributions through a Dirichlet process to reflect the similarity between tasks and the number of clusters. Tasks in the same cluster share the same model [14].
6. hierarchical Bayesian model, learning a latent task structure [15].
7. MTL extension of the regularized Perceptron, encodes task relatedness in a matrix. Subsequently, different regularizations can be applied to restrict it (e.g., rank) [16].
8. Different tasks belong to different independent clusters, each cluster exists in a low-dimensional space, and tasks in each cluster share the same model. Through alternating iterations, learning the allocation weights of different clusters and the model weights for each cluster. Assuming the absolute independence between tasks may not be ideal [17].
9. Assuming there is overlap between two tasks from different clusters, with some latent basis tasks. Let the model parameters for each task be a linear combination of the latent basis tasks, and constrain the latent basis tasks to be sparse. The overlapping part controls the degree of sharing [18].
10. Learning a small number of shared hypotheses, then mapping each task to a single hypothesis [19].

2 MTL in DNN

Deep Relation Network [20]

In computer vision, convolutional layers are generally shared, followed by task-specific DNN layers. By setting priors on task layers, the model learns the relationships between tasks.

Fully-Adaptive Feature Sharing [21]

Starting from a simple structure, greedily and dynamically widening the model to cluster similar models. Greedy methods may not learn a globally optimal structure; each branch may not learn the complex relationships between tasks.

Cross-stitch Networks [22]

soft parameter sharing, learning the outputs of the previous layer through linear combinations, allowing the model to decide the degree of sharing between different tasks.

Low supervision [23]

Finding better multi-task structures, the lower layers of complex tasks should be supervised by lower-level task objectives (e.g., in NLP, the first few layers learn an NER or POS auxiliary task).

A Joint Many-task Model [24]

Pre-setting hierarchical structures for multiple NLP tasks, followed by joint learning.

Weighting losses with uncertainty [25]

Not considering learning shared structures, but considering the uncertainty of each task. By optimizing loss (Gaussian likelihood with task-dependent uncertainty), adjusting the similarity between different tasks.

Tensor factorization for MTL [26]

Decomposing parameters for each layer, for shared and task-specific.

Sluice Networks [27]

A mix of (hard parameter sharing + cross-stitch networks + block-sparse regularization + task hierarchy (NLP)), allowing the model to learn which layers and subspaces to share, and where the model finds the optimal representation of the inputs.

When tasks are highly related and approximately follow the same distribution, sharing parameters is beneficial. But what about tasks that are not highly related or unrelated?

Early work specified which layers to share for each pair of tasks, but this method has poor scalability and severely biased model structures; when task correlations decrease or different levels of reasoning are needed, hard parameter sharing becomes ineffective.

Currently, the popular approach is learning what to share (outperforming hard parameter sharing); also, learning task hierarchies is useful when tasks have multi-granularity factors.

3 Auxiliary tasks

We focus only on the main task objectives, but hope to benefit from other effective auxiliary tasks!

Current selection of auxiliary task methods

1. Related task: Conventional ideas (autonomous driving + road sign recognition; query classification + web search; coordinate prediction + object recognition; duration + frequency).
2. Adversarial: In domain adaptation, related tasks may not be accessible, and adversarial tasks can be used as negative tasks (maximizing training error), for example, an auxiliary task predicting the input domain, leading the main task model to learn representations that cannot distinguish between different domains.
3. Hints: Some features mentioned earlier may be difficult to learn in certain tasks, selecting auxiliary tasks for predicting features (in NLP, the main task is sentiment prediction, and the auxiliary task is whether the inputs contain positive or negative words; for the main task of name error detection, the auxiliary task is whether there is a name in the sentence).
4. Focusing attention: Enabling the model to pay attention to parts that may not be easily noticed in the task (autonomous driving + road sign detection; facial recognition + head position recognition).
5. Quantization smoothing: In certain tasks, the training objective is highly discretized (manual scoring, sentiment scoring, disease risk levels), using auxiliary tasks with lower discretization may be helpful, as a smoother target makes tasks easier to learn.
6. Predicting inputs: In some scenarios, certain features may not be selected because they are detrimental to estimating the target, but these features may actually help with model training; in such cases, these features can be treated as outputs rather than inputs.
7. Using the future to predict the present: Some features only become available after a decision, for example, in autonomous driving, data about objects is obtained only when the vehicle passes them; in medicine, the effects of a drug are known only after it has been used. These features cannot serve as inputs but can be used as auxiliary tasks to provide information to the main task during training.
8. Representation learning: Auxiliary tasks often implicitly learn some feature representations, which to some extent benefit the main task. It can also explicitly learn this (using an auxiliary task that learns to transfer feature representations, such as AE).

So, which auxiliary tasks are useful?

The assumption behind auxiliary tasks is that they should be somewhat related to the main task and beneficial for its learning.

How to measure the similarity between two tasks?

Some theoretical studies:

1. Using the same features for decision-making.
2. Related tasks share the same optimal hypothesis space (having the same inductive bias).
3. F-related: If the data of two tasks is obtained through a fixed distribution with some transformations [28].
4. Classification boundaries (parameter vectors) are close.

Task similarity is not binary; the more similar the tasks, the greater the benefits. Learning what to share allows us to temporarily overlook theoretical shortcomings, as even tasks with poor correlation can yield some benefits. However, developing task similarity is absolutely helpful in selecting auxiliary tasks.

4 MTL Learning Tips

1. Auxiliary tasks with compactly distributed labels are better (from POS in NLP) [29].
2. The main task training curve stabilizes faster, while the auxiliary task stabilizes slowly (not yet stable) [30].
3. Different tasks may have different scales, optimal learning rates for tasks may vary.
4. The output of one task can serve as the input for some other tasks.
5. The iteration cycles of certain tasks may differ, requiring asynchronous training (posterior information; feature selection, feature derivation tasks, etc.).
6. The overall loss may be dominated by certain tasks, necessitating dynamic adjustments to parameters over the entire cycle (by introducing some uncertainty, each task learns a noise parameter, unifying all losses [31]).
7. Some estimates serve as features (alternating training).

5 Conclusion

Hard parameter sharing, which is over 20 years old, is still very popular. Currently, the trending concept of learning what to learn is also very valuable. Our understanding of tasks (similarity, relationship, hierarchy, benefit for MTL) is still very limited, and we hope for significant developments in the future.

Research Directions

1. Learning what to share.
2. Measurement for similarity of tasks.
3. Using task uncertainty.
4. Introducing asynchronous tasks (feature learning tasks), adopting alternating iterative training.
5. Learning abstract sub-tasks; learning task structures (similar to hierarchy learning in reinforcement learning).
6. Parameter learning auxiliary tasks.
7. More…

Note: The learning materials for this article are mainly derived from _An Overview of Multi-Task Learning in Deep Neural Networks, https://arxiv.org/abs/1706.05098.

Download 1: OpenCV-Contrib Extension Module Chinese Version Tutorial

Reply in the "Learning Vision for Beginners" WeChat public account: Extension Module Chinese Tutorial, to download the first OpenCV extension module tutorial in Chinese online, covering installation of extension modules, SFM algorithms, stereo vision, object tracking, biological vision, super-resolution processing, etc., with more than twenty chapters of content.

Download 2: Python Vision Practical Project 52 Lectures
Reply in the "Learning Vision for Beginners" WeChat public account: Python Vision Practical Project, to download 31 visual practical projects including image segmentation, mask detection, lane line detection, vehicle counting, eyeliner addition, license plate recognition, character recognition, emotion detection, text content extraction, facial recognition, etc., to help quickly learn computer vision.

Download 3: OpenCV Practical Project 20 Lectures
Reply in the "Learning Vision for Beginners" WeChat public account: OpenCV Practical Project 20 Lectures, to download 20 practical projects based on OpenCV, achieving advanced learning of OpenCV.

Communication Group

Welcome to join the reader group of the public account to communicate with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions, etc. (will gradually subdivide in the future). Please scan the WeChat ID below to join the group, and note: "nickname + school/company + research direction", for example: "Zhang San + Shanghai Jiao Tong University + Visual SLAM". Please follow the format for notes, otherwise, you will not be approved. After successful addition, you will be invited to relevant WeChat groups based on research direction. Please do not send advertisements in the group, otherwise, you will be removed from the group. Thank you for your understanding~