Accurate, fast, and open-source—there’s probably nothing better than this.
The paper “MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network” from Middle East Technical University has been accepted at the ECCV 2018 conference, utilizing the Pose Residual Network (PRN) for rapid multi-person pose estimation.
This paper proposes a novel bottom-up multi-person pose estimation architecture that combines a multi-task model with a new assignment method. MultiPoseNet can jointly address human detection, keypoint detection, human segmentation, and pose estimation issues. The innovative assignment method is implemented by the Pose Residual Network (PRN), which receives the results of keypoint and human detection to produce accurate poses by assigning keypoints to human instances. On the COCO keypoint dataset, this pose estimation method outperforms all previous bottom-up methods in terms of accuracy (by 4 points in mAP over the previous best results) and speed; it can match the accuracy of the best top-down methods while being at least 4 times faster. The MultiPoseNet method is currently the fastest real-time pose estimation system, achieving 23 frames per second on a GTX1080TI GPU.
Multi-person pose estimation is divided into bottom-up and top-down approaches.
Bottom-Up Method first detects human parts in the image and then allocates the parts of multiple humans in the image to different human instances, making this method generally faster during testing inference with a smaller model size, but often slightly lower in accuracy due to less detailed modeling of human keypoints.
Top-Down Method separates human detection and keypoint detection, first running a human detector on the image to find all human instances, and then using keypoint detection on each human sub-image. This method is often extremely slow, but the pose estimation accuracy is higher.
The MultiPoseNet multi-task learning architecture efficiently implements human keypoint detection, human detection, and semantic segmentation:
Diagram of the Pose Residual Network (PRN), which is used to allocate each keypoint to the corresponding human.
The backbone network for feature extraction uses ResNet with two Feature Pyramid Networks (FPN), one outputting to the keypoint estimation subnet and the other to the person detection subnet.
Diagram of the keypoint estimation subnet, which takes stacked CNN features as input and outputs keypoint and segmentation heatmaps.
The person detection subnet directly utilizes a modified RetinaNet, allowing the network to detect only humans.
The Pose Residual Network (PRN) is a keypoint collection for each detected human region, learning the mapping from the keypoint collection to a normal distribution of keypoints. The authors refer to this step as residual correction, implemented using a residual multilayer perceptron.
Precision-recall curves on the COCO validation dataset:
Results on the COCO test-dev dataset, where BU denotes the bottom-up method and TD denotes the top-down method.
Performance comparison using different backbone networks.
Experimental results of different PRN models (N: nodes, D: dropout, and R: residual connection) on COCO validation; it can be seen that models utilizing residual connections significantly improve performance:
Results of human detection on the COCO dataset; the original RetinaNet model has 80 classes, but after processing, it only detects humans, leading to a significant performance increase.
Human segmentation is achieved by adding an additional layer to the output layer of the keypoint network, resulting in minimal time cost while reaching the level of DeepLab v2.
Example of image operation results on the COCO test-dev dataset.
Analysis of running speed, model parameters, and detection time as the number of people increases.
On the 1080TI GPU, keypoint and person detections only require 35ms, as PRN is very fast, needing only 2ms, so each additional instance only adds 2ms. When there is only 1 person in the image, it runs at 27 frames per second; even with 20 people, it can still achieve 15 frames per second.
Overall, this article feels more engineering-oriented. Feel free to leave your thoughts on this article.
The code will be open-sourced here:
https://github.com/mkocabas/pose-residual-network/
Click to read the original text at 52cv.net.
Welcome to visit www.52cv.net for more information on computer vision and machine learning technologies.
More exciting recommendations:
PRNet: 3D Face Reconstruction and Dense Alignment
Alibaba proposes the Auto-Context R-CNN algorithm, achieving a new height in Faster RCNN object detection.
Open-source OCR text recognition software Calamari.
The U.S. Intelligence Advanced Research Projects Activity (IARPA) has launched a machine learning challenge for predicting public safety issues (with a total prize pool of $100,000).
A comprehensive guide to getting started in computer vision research.
A comprehensive comparison of Keras and PyTorch: Which deep learning framework is more suitable for beginners?
Long press to follow my computer vision public account:
