Source: Deephub Imba
This article is about 1000 words long and is recommended to read in 4 minutes.
This article introduces a simple, flexible, and effective pre-training strategy for ViT.
MAE is a ViT that uses a self-supervised pre-training strategy, masking patches in the input image and then predicting the missing areas for sub-supervision and training.Although this method is both simple and effective, the MAE pre-training objective is currently limited to a single modality—RGB images—restricting its application and performance in practical scenarios where multimodal information is typically presented.
In the new paper MultiMAE: Multi-modal Multi-task Masked Autoencoders, a team from École Polytechnique Fédérale de Lausanne (EPFL) proposes Multi-modal Multi-task Masked Autoencoders (MultiMAE), which is also a pre-training strategy capable of automatically encoding masks and performing multimodal and multitask training. MultiMAE uses pseudo-labels for training, making this framework applicable to any RGB dataset.
The design of MultiMAE is based on traditional Masked Autoencoding but differs in two key aspects:
1. In addition to RGB images, it can optionally accept additional modality information in the input (hence “multi-modal”).
2. Its training objective accordingly includes predicting multiple outputs beyond RGB images (hence called “multi-task”).
Architecturally, the encoder of MultiMAE is a ViT, but each additional input modality has a patch projection layer and a learnable extra global token embedding, similar to the class token of ViT. Therefore, only loading the required input projections and ignoring all other projections allows the MultiMAE pre-training weights to be directly used for standard single-modal ViT.
To perform semantic segmentation patch projection, the authors of the paper replace each class index with a learned 64-dimensional class embedding. Additionally, only a random subset of visible tags is encoded, significantly accelerating computation and reducing memory usage, and MultiMAE multimodal pre-training is performed using three dense input modalities. Each task uses a separate decoder, so the computation of the decoder scales linearly with the number of tasks and only adds minimal cost.
In their study, MultiMAE was pre-trained on three tasks: image classification, semantic segmentation, and depth estimation, with pseudo-labeling on ImageNet-1K, followed by fine-tuning on ImageNet, ADE20K, Taskonomy, Hypersim, and NYUv2 datasets.
Results indicate that when fine-tuning using only RGB, MultiMAE retains the advantages of the conventional MAE, and it can also leverage other modalities such as depth using pseudo-labeled depth or semantic segmentation to enhance performance. The MultiMAE pre-training strategy can significantly improve transfer performance.
The project has also made the code, pre-trained models, and interactive visualizations publicly available on GitHub. The paper MultiMAE: Multi-modal Multi-task Masked Autoencoders can be found at the following address:https://multimae.epfl.ch/
