Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery

Introduction

This article is an interpretation of the paper Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot by VCC student Yu Tao. This work comes from the European NAVER laboratory and has been published at the top computer vision conference ECCV 2024.

Project homepage: https://europe.naverlabs.com/blog/whole-body-human-mesh-recovery-of-multiple-persons-from-a-single-image/This work proposes a method called Multi-HMR to recover human meshes of multiple people from a single perspective image, utilizing a new cross-attention module called Human Pose Head (HPH) to predict their full body posture, shape, and spatial location, capable of predicting human meshes with high accuracy. This representation method can play a greater role in subsequent fields such as VR, AR, and human-computer interaction.Note: The images and videos in this article are sourced from the original paper and its project homepage.

I Introduction

Recently, with the popularity of digital humans and virtual worlds, many works have emerged that recover human meshes from images. The commonly used human mesh model is the SMPL model, which is based on a large amount of human scanning data and constructs a triangular mesh model of the human body through low-dimensional pose rotation vectors and shape parameters. This model provides an effective way to represent and generate human morphology, widely applied in tasks such as human pose estimation and motion capture in computer vision and graphics. The mainstream methods currently include two main types: optimization-based and regression-based methods. This article belongs to the regression-based methods, and the task of human mesh recovery still cannot be well applied to real-world scenarios, mainly due to the existing methods’ lack of expressiveness in body posture and inability to accurately estimate a person’s position in camera space, with many methods still focusing only on the recovery of a single person’s mesh.

To address the above issues, this guide presents the Multi-HMR, introducing a new method to recover multiple human meshes from images in a single shot. This method not only captures more expressive body postures but also enhances the representation of facial and hand details, and provides an effective way to handle multiple people in images while utilizing camera parameters to predict a person’s position in camera space. The proposed module applies a pre-trained VIT as the backbone network in feature extraction and performs block-level detection to identify all heads in the image. Additionally, a Human Perception Head (HPH) is proposed to interactively select features of detected individuals with the global feature information of all image blocks through the construction of cross-attention, achieving the prediction of complete posture, shape, and depth parameters. This method can estimate the three-dimensional mesh structure of the human body more effectively and accurately than previous technologies.

II Technical Contributions

The main contributions of this work are as follows:

• Proposed a new dataset for training that enhances the recovery of details (such as hand movements);

• Proposed a new attention network architecture that can combine effective camera parameters and efficiently handle different numbers of people, achieving cross-attention between humans and the entire scene;

• Captures expressive body postures, including facial and hand details.

As shown in the table below, this method provides both more expressive model predictions compared to previous mesh regression methods and a network framework that combines camera parameters to predict multiple people in a single instance. The following will demonstrate the comparative characteristics of Multi-HMR relative to existing methods in multiple aspects.

Table 1 Comparison of Features Between Multi-HMR and Other Mainstream MethodsIII Method Introduction

The Multi-HMR method consists of two parts: human detection and human perception network architecture. The first part aims to extract features from the input image using a basic version of VIT as the backbone network for human detection, while the second part interacts the detected human features with global image features to perform full-body human mesh and depth regression. Additionally, to enhance the ability to predict hand postures, Multi-HMR introduces a new dataset — CUFFS, specifically designed to strengthen hand posture predictions.

The network structure is divided into two parts:

1. Human DetectionIn the input image, a basic version of ViT (Vision Transformer) is used as the backbone network to extract features for human detection. This part follows the CenterNet network paradigm, using the head of the human as the main key point for localization. Specifically, the 3D head key points are defined based on the SMPL-X human model. Through the ViT backbone network, features of the image are first extracted, and the location of the head within the image is predicted. To achieve fine 2D coordinates, the offset within the image block will also be predicted to improve localization accuracy.2. Human Perception Head and Mesh RegressionThis part of the network structure mainly includes two modules: Human Perception Head (HPH) and Parameterization Module (SMPL-X).Human Perception Head (HPH):Utilizes a Transformer structure to process N detected key points through the cross-attention module. Specifically, the query corresponding to the key points is embedded as the query in the cross-attention module, while the key and value of the module come from the features of image blocks extracted by the backbone network. This design allows focusing on key information from all image blocks before performing complete posture, shape, and depth regression predictions for the human body.SMPL-X Parameterization Module:Based on the feature information processed by the HPH module, N human-centered posture, shape, and depth parameters are regressed through a shared MLP. Ultimately, these parameters are transformed into corresponding human triangular meshes using the SMPL-X model.Figure 1 Network Structure DiagramDatasetThe existing synthetic datasets BEDLAM and AGORA provide perfect supervision for the SMPL-X model, including faces and hands. However, in these datasets: i) Most people are seen from a distance, which is not ideal for capturing the subtle details needed for accurate reconstruction of faces and hands, ii) Hand postures lack diversity. Therefore, this method enhances the training data by adding a dedicated, augmented dataset comprising close-up images of single hands in different poses. Sample images of the dataset are shown in Figure 2:Figure 2 Dataset SamplesIV Result Demonstration Below shows the performance of Multi-HMR on multiple datasets. As shown in Figure 3, the top image comes from the EHF and Agora datasets, the middle results come from the MuPoTs and 3DPW datasets, and the bottom results come from the UBody and CMU datasets.Figure 3 Multi-HMR Achieves Good Visualization Results Across Multiple Datasets

Table 2 Comparison of Experimental Results of Different MethodsThis method is compared with the most advanced methods in multi-person mesh recovery, such as ROMP, BEV, and PSVT, as shown in Table 2a. All these multi-person methods are limited to triangular meshes that only include body posture and shape. The model in this article outperforms existing work, showing significant improvements across various metrics. It concludes that Multi-HMR performs well in multi-person full-body mesh recovery. The model’s full-body regression performance is evaluated by comparing it with full-body 3D pose prediction methods. All existing methods are limited to single-person scenarios: they do not consider the detection stage and 3D positions in the scene, but assume predefined 2D bounding boxes around the person of interest. The results in Table 2b indicate that the Multi-HMR method is competitive or even superior compared to previous full-body methods, particularly achieving competitive performance on hands and faces.V Conclusion and Outlook This paper presents the first monocular method for multi-person full-body human mesh recovery. The experiments in this paper show that the method predicts accurately expressed 3D meshes (body, face, and hands) and 3D positions in the scene, outperforming the state-of-the-art techniques for each subproblem. The experiments also indicate that the advantage of Multi-HMR lies in its ability to adapt to camera information when available. Conceptually, Multi-HMR is straightforward: it relies on an architecture with a conventional ViT backbone, thus benefiting from improvements made by a broader range of visual models with minimal effort. Finally, while achieving optimal performance with a large backbone network, this paper demonstrates that using a small backbone network is sufficient to achieve competitive performance, allowing for real-time inference. Future work hopes to expand to larger scenarios, addressing the problem of person-to-person occlusion, making this method more generalizable.VI Thoughts and Discussions Q: In crowded scenes, will the detection module of Multi-HMR result in multiple people’s detection results being under the same image block?A: As observed from the experimental results in this paper, as long as a reasonable resolution (i.e., over 448×448) and appropriate image block size (i.e., 14×14) are used for the image detector, the occurrence of overlapping detection results for people is very rare.Q: What are the main loss terms used in Multi-HMR?A: Multi-HMR predicts three corresponding parameters of the parameterized model SMPL-X, which can then be used to obtain the corresponding triangular mesh, including the corresponding 3D vertex positions and 3D joint positions, which can be projected to obtain the corresponding 2D vertex and joint positions. Additionally, the 2D detection results output during the detection phase will be supervised against the 2D detection maps generated from labeled 2D head points. Therefore, there are three loss terms in total: the loss of 2D detection results, the loss of SMPL-X parameters, and the loss of 3D and 2D points obtained from the SMPL-X parameterized model.The following are open-ended questions, and readers are welcome to leave comments for discussion:

Q: Although Multi-HMR is a regression-based method, it relies heavily on network predictions. If regression methods are used as good initialization results, can optimization methods that utilize some priors for optimization iterations solve the initialization problem of optimization methods and achieve even better results?

— End–

Guide | Yu TaoReview | Zhang QiEditor | Shen Jin, Yu XinquanReferences

[1] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. Conference on Computer Vision and Pattern Recognition (CVPR). 10975-10985, 2019.

[2] Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, Michael J. Black. AGORA: Avatars in geography optimized for regression analysis. Conference on Computer Vision and Pattern Recognition (CVPR). 13468-13478, 2021.

[3] Michael J. Black, Priyanka Patel, Joachim Tesch, Jinlong Yang. Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. Conference on Computer Vision and Pattern Recognition (CVPR). 8726-8737, 2023.

[4] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J. Black, Tao Mei. Monocular, one-stage, regression of multiple

[5] Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, Michael J. Black. Putting people in their place: Monocular regression of 3D people in depth. Conference on Computer Vision and Pattern Recognition (CVPR). 13243-13252, 2022.

[6] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. IEEE International Conference on Computer Vision (ICCV). 14783-14794, 2023.

[7] Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, Yu Li. One-stage 3D whole-body mesh recovery with component aware transformer. Conference on Computer Vision and Pattern Recognition (CVPR). 21159-21168, 2023.

[8] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, Kostas Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. IEEE International Conference on Computer Vision (ICCV). 2252-2261, 2019.

Visual Computing Research Center, Shenzhen University———————————-https://vcc.techThe center is based on disciplines such as computer graphics, computer vision, visualization, robotics, artificial intelligence, and human-computer interaction, dedicated to promoting in-depth cross-disciplinary integration and innovation, focusing on advancing scientific research in areas such as large-scale static and dynamic data acquisition and optimization fusion, multi-scale geometric modeling and image processing, visual content generation and simulation rendering, complex scene reconstruction and recognition understanding, three-dimensional mobile collaborative perception and human-computer interaction, intelligent simulation learning and cognitive reinforcement, and massive information visualization and visual analysis.📫Reprint and Cooperation: [email protected]