Background Introduction
In today’s rapidly advancing industrial, logistics, and service robotics sectors, “enabling robots to navigate safely and efficiently through complex crowds” has become a crucial step for enterprises towards intelligence.
Compared to the navigation environment of self-driving cars on highways,the real-life scenario of “mixed traffic of people and vehicles” is much more uncontrollable:
-
Pedestrians may suddenly cross;
-
The layout of the area is irregular (such as parks, campuses, squares, streets);
-
Dense obstacles, numerous dynamic elements, and significant occlusion.
However, the mainstream datasets (such as KITTI, nuScenes, Waymo) are designed for autonomous driving,collected from clean and orderly road environments, and almost do not include information on dense obstacles and complex crowds at close range. This leads to:
✅ Robots “see unclear” → Unable to effectively avoid obstacles✅ Inaccurate prediction of pedestrian behavior → Causes sudden stops or collisions✅ Enterprises have to build their own data → High costs, long cycles, and unstable results
Especially forcleaning robots, security robots, and delivery robots, which are low-speed mobile devices, the perception range is concentrated within 5 meters,traditional solutions cannot meet the urgent need for “near-field safety perception” at all.
The image shows how robots can “see” their surrounding environment through multiple sensors in real scenarios, including distant vehicles and nearby pedestrians. Compared to traditional single-view solutions, this “surround vision” significantly reduces blind spots and greatly enhances the safety of robot decision-making.
AI Methods
To enable robots to “see clearly and move steadily” in complex environments, the research team has developed a brand newindustry-level data platform — RoboSense, providing enterprises with real, usable, and deployable AI navigation data support.
✅ 1. What is RoboSense?
RoboSense is a large-scale multimodal dataset specifically designed for near-field perception tasks of robots, featuring the following core characteristics:
| Capability | Description |
|---|---|
| 📦 Data Scale | Covering 133,000 frames of images, 1.4 million 3D bounding boxes, and 216,000 trajectories, far exceeding traditional datasets like KITTI. |
| 🎯 Comprehensive Perception | Equipped with cameras + fisheye + LiDAR multiple sensors,360° no blind spots |
| 👀 Closer Distance | The number of near-field obstacles within 5 meters in the samples is 18 to 270 times that of other datasets. |
| 🧠 Multi-task Support | Providing6 types of task benchmarks including detection, tracking, prediction, and occupancy mapping, assisting in the full-process iteration of algorithms. |
✅ 2. In what scenarios can it be used?
RoboSense is not just a dataset, but areal toolbox for AI + robotics deployment. Typical scenarios it targets include:
-
Industrial Cleaning Robots: Smart obstacle avoidance and edge walking in factories and shopping malls to avoid collisions or missed cleaning;
-
Campus Delivery Robots: Automatically planning routes in universities and scenic areas, recognizing changes in crowd density, and autonomously avoiding obstacles;
-
Supermarket Service Robots: Efficient movement in crowded environments to achieve functions such as guidance, stocking, and security;
-
Unmanned Patrol/Security Devices: Stably identifying suspicious targets and continuously tracking them at night or under occluded conditions.
The common characteristics of these scenarios are:low speed, near-field, dynamic, and complex — which is precisely the problem RoboSense is designed to solve.
The image shows the number of annotated obstacles within a 5-meter range, with RoboSense exceeding nuScenes by dozens of times, indicating its suitability for real industrial, retail, and campus scenarios for robot applications.
Module Details
The advantages of RoboSense are not only reflected in data scale and scenario diversity but also in itshighly modular perception system design — enterprises can build their own robot perception solutions flexibly, like assembling building blocks.
✅ 1. Multi-sensor Layout: Perceiving the World Like Human “Multi-sensory”
RoboSense adopts a “triple perception system”, combining camera + fisheye lens + LiDAR to achieve a 360° surround view:
-
Standard Camera: Provides high-definition details for identifying distant targets;
-
Fisheye Lens: Ultra-wide angle, focusing on near-field low-position targets (such as children and pets);
-
LiDAR: Constructs three-dimensional spatial perception, accurately measures distances, and identifies obstacles behind occlusions;
-
Auxiliary Sensors (GPS / IMU / Ultrasonic): Provide positioning, stability, and near-ground information.
These devices are installed on the front, back, sides, and top of the robot,forming a blind-spot-free “perception ring”, ensuring that even when navigating through crowds, there is no fear of being obstructed.
✅ 2. Perception-Fusion-Prediction: The Three Steps of the Robot’s “Brain”
RoboSense supports enterprises in building a complete navigation perception system, including the following three modules:
🧠 Module One: Perception Detection
-
Task: Identify the position and shape of surrounding people, vehicles, and bicycles;
-
Support: 3D detection, 2D detection, multi-view reconstruction;
-
Representative Models: PointPillar (lightweight), Transfusion-L (Transformer strong model), etc.
🧠 Module Two: Multi-sensor Fusion
-
Task: Treat the results of different sensors as a **”whole”**;
-
Technology: Bird’s Eye View (top-down fusion), Transformer attention mechanism;
-
Benefits: The fused perception is more accurate, with a higher recognition rate.
🧠 Module Three: Motion Prediction and Mapping
-
Task: Predict where pedestrians or vehicleswill go in the next 3 seconds, andplan passable areas;
-
Use: Helps robots to navigate in advance, avoiding low-level mistakes like “bumping and then stopping”.
The image shows the positions of various sensors installed on the robot, and how perception data from different angles is fused into a unified coordinate system. Enterprises can adjust sensor positions or quantities based on this to achieve more cost-effective deployments.
Modular Model Structure
To verify the practicality of the RoboSense dataset in real industrial scenarios, the research team conducted systematic benchmark tests involving multiple mainstream algorithms and various tasks, including:3D object detection, multi-target tracking, motion prediction, occupancy prediction, etc.
The results indicate:RoboSense is not only suitable for AI algorithm research but also for direct deployment in enterprise robot products, achieving “clear vision, precise avoidance, and stable movement”.
✅ 1. Stronger Detection Performance: Accurate Vision for Effective Avoidance
-
Near-field Detection Performance: For obstacle detection accuracy within 5 meters, the models supported by RoboSense data achieve 36.9% AP (industry high standard), more thantwice that of traditional datasets;
-
Better Fusion Sensor Effects: After fusing camera + fisheye + LiDAR, the detection accuracy is significantly higher than that of single-modal models,solving occlusion and near-field target miss detection issues.
The chart shows the detection performance of different models and sensor combinations within the “nearest 5 meters” range;
-
Explanation:Near-field perception is the most critical yet challenging point in enterprise robot navigation, and RoboSense precisely addresses this pain point.
✅ 2. Improved Motion Prediction Accuracy: Avoiding Obstacles 3 Seconds in Advance, Preventing “Bumping and Then Stopping”
-
In the task of predicting the future 3 seconds of pedestrian or vehicle trajectories, the PnPNet model based on RoboSense achieved:
-
Average error of less than 1 meter;
-
Miss detection rate as low as 17%, significantly better than traditional constant-speed models;
-
Compared to the “static position assumption” or “constant speed assumption”,the accuracy of AI predicted paths improved by nearly 40%.
Demonstrating the performance comparison of different prediction methods (traditional vs AI);
-
Practical Value:Applicable to delivery robots and supermarket guide robots, avoiding children, elderly people, or suddenly turning crowds.
✅ 3. Occupancy Prediction and Spatial Mapping: Planning Passable Paths to Enhance Mobility Efficiency
-
RoboSense introduces a 3D occupancy prediction mechanism, allowing robots to “understand the environment” and determine which areas are:
-
Passable,
-
Occupied by obstacles,
-
Or out of sight;
-
In the occupancy mapping within 2 meters, the mIoU reaches 48.2% (BEV perspective), outperforming existing visual solutions.
-
Demonstrating the performance of occupancy prediction at different ranges (0–2m, 2–5m, 5–12.8m);
-
In simple terms: Robots no longer just “take the long way around” but can “navigate through openings like experienced drivers”, enhancing efficiency.
(Reference: RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments)
Who Are We?
RoboSense is not just a research dataset, but adeployable, implementable, and profitable AI toolbox:
-
Solving Core Pain Points: Focusing on the three major industry challenges of “near distance, complex environments, and low-speed movement”;
-
Directly Accelerating Product Launch: Enterprises can directly train models using the dataset, reducing the time for self-built data and annotation,cutting development cycles by 50%;
-
Cost Reduction, Efficiency Doubling: More precise near-field obstacle avoidance capabilities meanless human-machine intervention, fewer downtime incidents, and lower operational costs;
-
Connecting to Real Application Scenarios: Covering extensive application scenarios for ground robots in supermarkets, campuses, factories, and scenic areas,transforming general algorithms into practical solutions.
💡 For Example:
A delivery robot originally could only operate on fixed routes and would stop and alarm when encountering obstructions.
🔁 After upgrading to a navigation model trained based on RoboSense, it can flexibly navigate around temporary obstacles (such as crowds or fallen objects),increasing passage efficiency by 43%, saving a significant amount of manual operational intervention and supporting round-the-clock operation.
What does this mean?It means your robot team can reduce staffing by two people, double the delivery orders, and directly double profits.
RoboSense demonstrates the immense potential of visual AI in industrial navigation, but true implementation still requiresintegrated design from data, algorithms to scenarios.
What Can We Do?
We are Pydance Technology, focusing on customized development of AI solutions.
We do not just “sell models” but genuinely solve frontline issues for enterprises using AI. In the past, we have served:
Image grading in the tobacco industry;
Wall crack detection at construction sites;
Industrial defect identification;
Classification and recognition of scenic images;
Medical image structured auxiliary judgment……
Our philosophy is: “Design structure for real scenarios, optimize models for application results”.
If you encounter similar issues in your industry regarding “image recognition, classification, and detection”, feel free to message us, and let’s discuss your needs to see how AI can truly help.
Pydance Technology
Company Name
– Shanghai Pydance Technology Co., Ltd.
– Yunnan Pydance Technology Co., Ltd.
Company Address
– Building 14, Pudong International Talent Port, Huan Ke Road, Pudong New District, Shanghai, AI Station
– Room 2209, Block B, Dianchi Times, Xishan District, Kunming City, Yunnan Province
Contact Email:[email protected]
