Predicting multiple real-world tasks within a single model often requires a particularly diverse feature space. Multimodal (MM) models aim to harness the synergistic predictive potential of various data types to create a shared feature space with semantic meaning aligned across inputs (e.g., images, text, sound), despite significant differences in their sizes. Most current MM architectures parallelly fuse these representations, which not only limits their interpretability but also creates a dependency on the availability of modalities. We propose MultiModN, a multimodal modular network that fuses latent representations in sequences of any number, combination, or type of modalities while providing fine-grained real-time predictive feedback on any number or combination of prediction tasks. The composable pipeline of MultiModN is designed to be interpretable and is inherently multitask, robust to the fundamental problem of skewed missingness. We conducted four experiments on several benchmark MM datasets for ten real-world tasks (predicting medical diagnoses, academic performance, and weather) and showed that, compared to the parallel fusion baseline, the sequential MM fusion of MultiModN does not degrade performance. By simulating challenging biases, namely missing not at random (MNAR), this paper demonstrates that, in contrast to MultiModN, the parallel fusion baseline incorrectly learns MNAR and suffers catastrophic failures when faced with different MNAR patterns during inference. To our knowledge, this is the first inherently MNAR-resistant MM modeling approach. In summary, MultiModN provides fine-grained insights, robustness, and flexibility without compromising performance.
https://www.zhuanzhi.ai/paper/d1b365aedeee53e22c2eaec14e77c669
Convenient Access via Zhuanzhi
Convenient Download, please follow the Zhuanzhi WeChat public account (click the above blue Zhuanzhi to follow)
Reply or send a message “MTEM” in the backend to get the download link for “【NeurIPS2023】MultiModN: A Modular Network for Multimodal and Multitask Learning” on Zhuanzhi
Zhuanzhi, a Professional and Trusted AI Knowledge Distribution Platform, accelerates cognitive collaboration! Welcome to register and log in to Zhuanzhi www.zhuanzhi.ai, to access 100000+ AI (AI in military, medicine, public security, etc.) themed knowledge resources!
Scan the QR code to join the Zhuanzhi AI Knowledge Community, to receive the latest AI professional knowledge tutorials and engage with experts!
Click “Read the Original” to learn how to use Zhuanzhi, and to view and obtain 100000+ AI themed knowledge resources