Humanoid Robots Finally Have a ‘Panoramic Vision Brain’! How Humanoid Occupancy Solves Perception Challenges?

Have you ever seen a scene where a humanoid robot tries to grab a cup in the kitchen but cannot see the target clearly because its arm blocks the camera? Or when walking in a factory, it nearly falls because the sensors fail to detect a bump on the ground? These ‘perception blind spots’ are the biggest obstacles for humanoid robots to navigate complex real-world scenarios.The paper we are going to discuss today, titled “Humanoid Occupancy: Enabling A Generalized Multimodal Occupancy Perception System on Humanoid Robots”, provides a breakthrough solution — it creates a ‘panoramic vision brain’ for humanoid robots, enabling them to ‘understand’ every detail of the 3D world through multimodal fusion and innovative occupancy representation.01

Why Are Humanoid Robots’ ‘Eyes’ Never Enough?

Humanoid robots must handle three core tasks while working in human environments:grasping objects (manipulation), walking steadily (mobility), and navigating while avoiding obstacles (navigation), but these three tasks have conflicting demands for ‘vision’:

• For manipulation, a ‘close-range high-definition view’ is needed to see the texture and position of the cup;

• For mobility, a ‘ground panoramic view’ is required to detect terrain changes like steps and bumps;

• For navigation, a ‘global spatial view’ is necessary to know the positions of tables and walls to plan paths.

Moreover, traditional perception solutions have two major shortcomings: either they use a single sensor (like a camera), which can be blocked and fail to see depth; or they use representations like BEV (Bird’s Eye View), which only focus on ground projections and miss vertical space information like ceilings and high cabinets.The proposed ‘occupancy representation’ solves this problem — it acts like a ‘3D spatial dictionary’, marking whether each voxel (small cube) is occupied by an object (geometric information) and indicating whether this voxel belongs to a ‘table’, ‘pedestrian’, or ‘wall’ (semantic information), allowing the robot to gain a complete environmental understanding in one go.02

This ‘Vision Brain’: The Golden Combination of 6 Cameras + 1 LiDAR

The dataset named ‘Open X-Embodiment’ has three ‘mosts’:

• The most diverse ‘bodies’: It includes 22 types of robot forms, from single-arm robotic arms (Franka, xArm) to bipedal robots, quadrupedal robots, and even composite robots with mobile bases;

• The most comprehensive ‘skills’: It records 527 types of skills, ranging from ‘picking up a red cup’ to ‘wiring’ and ‘assembling furniture’, covering home and industrial scenarios;

• The most massive ‘experience’: Over 1 million real trajectories, equivalent to allowing the robot to ‘see’ the operations of humans or other robots a million times.

More importantly, this data has been unified into a format that ‘robots can understand’ — regardless of the robot’s actions, they are converted into 7-dimensional coordinates (controlling the position and posture of the end effector), facilitating ‘cross-body’ learning for the model.03

The First Panoramic Occupancy Dataset for Humanoid Robots: 3 Scenes + 13 Semantic Classes

With hardware in place, a ‘textbook’ is also needed for the robot to learn — the paper constructs thefirst dedicated panoramic occupancy dataset for humanoid robots, which is solid in scale and detail:

• Data volume: 180 training clips (200 frames each) + 20 validation clips, covering three core scenarios: home, industrial, and outdoor;

• Annotation accuracy: Not only are dynamic targets (like pedestrians) marked with 3D bounding boxes, but static scenes also undergo ‘point-wise semantic annotation’ — for example, pedestrians are categorized into ‘normal posture’ (only marked with a box) and ‘special posture’ (like bending down, requiring point-wise differentiation between body and occlusion);

• Semantic categories: Different scenes have dedicated ‘dictionaries’, such as 13 categories in home scenes (pedestrians, robots, chairs, tables, etc.), 8 categories in industrial scenes (conveyor belts, static objects, etc.), and 12 categories in outdoor scenes (bicycles, curbs, bushes, etc.), ensuring that robots can ‘recognize the right things’ in different environments.

The generation of ground truth for the dataset is also meticulous: first, multiple frames of static point clouds are stacked and denoised, then dynamic targets are aligned and stitched according to posture, and finally, voxelization is performed directly (without complex Poisson reconstruction), resulting in ‘occupancy + semantic’ labels for each 0.1m×0.1m×0.1m voxel, making robot learning more efficient.04

Core Algorithm: How Does Multimodal Fusion Achieve ‘1+1>2’?

Hardware and data are the foundation, but what truly drives the ‘vision brain’ is the underlying algorithm —HumanoidOcc Multimodal Fusion Network. Its workflow resembles a ‘three-step puzzle’:

Step 1: Extract camera and LiDAR features separately

1. Camera features: Use ResNet50+FPN to extract semantic information from 6 cameras, such as ‘this area contains a red cup’;

2. LiDAR features: Use the PointPillar algorithm to break point clouds into small pillars, encoding depth and geometric information, such as ‘this pillar is 2 meters away from the robot’.

Step 2: Cross-modal attention fusion

The most critical step comes — using ‘deformable attention’ to let the camera and LiDAR ‘supplement each other’: using LiDAR’s BEV features as ‘questions (query)’ and camera features as ‘answers (key/value)’, accurately finding the camera area corresponding to each LiDAR point, preserving LiDAR’s depth accuracy while supplementing the semantic details from the camera. Additionally, the algorithm considers camera distortion to ensure that no geometric deviations occur after fusion.

Step 3: Temporal fusion + 3D prediction

To cope with dynamic scenes (like moving pedestrians), the network caches 1 frame of historical BEV features, aligns them to the current coordinate system, and then stitches and fuses them to enhance the perception stability of moving objects. Finally, the 2D BEV features are converted into 3D voxels, outputting a 200×200×24 occupancy grid (spatial range X/Y±10 meters, Z-1.5 meters to 0.9 meters), perfectly covering the activity space of humanoid robots.05

Experimental Results: 40.5M Parameters Outperforming Large Models

The paper validates the system’s performance through 3600 real experiments, with impressive core results:

• Lightweight and efficient: HumanoidOcc has only 40.5M parameters, which is more than half smaller than similar models (BEVDet 75.3M, FBOcc 76.8M), yet achieves an mIoU (3D semantic accuracy) of 55.73 and a rayIoU (depth axis accuracy) of 61.32 with dual-frame input (current + 1 historical frame), surpassing the larger parameter BEVFusion11;

• Ablation experiments prove the effectiveness of key designs:

• Distortion handling:Instead of performing image undistortion, ‘distortion-aware projection’ is used during fusion, resulting in an mIoU that is 0.51 percentage points higher than the undistortion approach;

• Temporal frame count:Using 2 frames (current + 1 historical) is optimal; increasing the number of frames leads to performance degradation due to accumulated pose errors;

• Modal fusion:The mIoU of camera + LiDAR fusion (55.73) is higher than using the camera (50.37) or LiDAR (48.61) alone, proving ‘1+1>2’.

06Future: Enabling Humanoid Robots to ‘See More and Walk Further’The paper not only provides a complete perception solution but also points out future directions: for example, combining 3D Gaussian splatting, NeRF, and other technologies to achieve omnidirectional reconstruction, expanding the dataset to cover more scenarios, allowing this ‘vision brain’ to adapt to humanoid robots from different brands.Perhaps soon, we will see humanoid robots accurately grasping tableware at home, flexibly avoiding workers in factories, and stably crossing steps outdoors — and all of this starts with this perception system that ‘understands the 3D world’.Paper Information

Paper Title: Humanoid Occupancy: Enabling A Generalized Multimodal Occupancy Perception System on Humanoid Robots

Paper Link: https://arxiv.org/abs/2507.20217

Project Homepage: https://humanoid-occupancy.github.io

Follow 【Embodied Intelligent Manufacturing】, and we will continue to analyze the cutting-edge breakthroughs in robot learning in the next issue~