This article is published in “China Aerospace” Issue 5, 2022
Beijing Spacecraft Design Institute
Wang Yaobing, Ma Chao
Artificial intelligence is a strategic technology leading the development of future space science and technology. This article elaborates on the current applications and future demands of artificial intelligence in the field of space robotics, and analyzes the issues faced by space robots using artificial intelligence technology in conjunction with the characteristics of aerospace engineering tasks, hoping to provide a reference for the research and application of artificial intelligence in the field of space robotics.
Artificial intelligence refers to the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and enhance human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve the best results. The “New Generation Artificial Intelligence Development Plan” released in 2017 in China pointed out that artificial intelligence has become a new focus of international competition and a new engine for economic development, emphasizing the need to build a space robot support platform in the “Basic Support Platform” section.
Space robots are special robots used in outer space beyond the Earth’s atmosphere. Compared to ground robots, their characteristics include working in unmanned, remote, harsh, and unknown space environments, thus requiring higher adaptability to the environment and intelligent autonomy. From this perspective, the organic combination of space robots and artificial intelligence is an inevitable trend.
Space robots are generally divided into two categories: on-orbit operation robots and planetary exploration robots. On-orbit operation robots refer to space robots that perform various operational tasks in a microgravity orbital environment, including but not limited to: free-flying robots, manned space station / space laboratory internal and external operation robots, unmanned space service station operation robots, etc. These robots provide various on-orbit operation services, such as target capture, target transfer and release, on-orbit assembly, on-orbit service, on-orbit manufacturing, and assisting astronauts in extravehicular activities. Planetary exploration robots refer to space robots that perform tasks on extraterrestrial bodies such as the moon, planets, and small celestial bodies, including but not limited to: unmanned / manned rovers, planetary exploration robots, planetary surface construction robots, etc. These robots usually have wheeled or legged mobility systems, and are generally equipped with operational robotic arms. The tasks they perform generally involve both mobility and operation aspects, such as planetary surface patrolling, sample collection and recovery, scientific experiments, planetary base construction, and assisting astronauts in exploration, etc.
Due to the differences in the tasks performed, the two types of robots have slightly different demands for artificial intelligence. The demands for artificial intelligence in on-orbit operation robots mainly include: operational target/object feature recognition and measurement, decision-making and planning for operational tasks, and motion planning and control for operational arms/hands; the demands for artificial intelligence in planetary exploration robots mainly include: environmental measurement, positioning and mapping, motion planning and control, sample screening, collection, packaging, and recovery, emergency handling for unexpected situations, etc. Overall, the demand for artificial intelligence in space robots mainly focuses on four aspects: perception, planning/navigation, control, and human-machine interaction.
1. Current Application Status at Home and Abroad
(1) Autonomous Perception
1. Space Target Pose MeasurementCurrently, the information perceived by on-orbit operation robots is mainly visual information. After the robot acquires the target image, it processes the image for feature recognition and extraction, obtaining the real-time pose of the target relative to the robot. Space robots such as the ETS-VII satellite robot, the International Space Station robotic arm (SSRMS+SPDM), the Japanese module robot on the International Space Station (JEMRMS), and the Orbital Express (OE) satellite robot, as well as China’s space station robotic arm, all use traditional image processing algorithms to autonomously complete the measurement of target pose information.Currently, AI-based space target pose measurement technology has not yet been applied in active space robots, but it is a hot topic in academic research. Numerous aerospace agencies are actively participating, for example, researchers from the Japan Aerospace Exploration Agency (JAXA) have conducted research on the pose estimation of freely floating targets under rapidly changing lighting conditions, proposing a robust pose estimator based on convolutional neural networks (CNN) that can directly estimate the 3D position of key points in real-time. In 2019, the European Space Agency (ESA) organized a challenge focused on satellite pose estimation, in which nearly 50 participating teams all used deep learning methods, demonstrating the application potential of artificial intelligence algorithms in space target measurement. In 2025, the ESA-supported ClearSpace-1 project plans to launch a satellite equipped with a robotic arm to perform orbital debris removal tasks (Figure 1), using deep learning algorithms to improve the reliability of pose estimation and reduce the impact of complex lighting environments on computational results.
Figure 1 ClearSpace-1 Mission Schematic2. Planetary Surface Topography Measurement and AssessmentMeasuring and assessing planetary surface topography based on visual information is a prerequisite for planetary exploration robots to carry out patrolling tasks. In 1997, the “Sojourner” Mars rover landed on Mars and operated in remote control mode: ground personnel completed topography measurement, assessment, and path selection tasks, periodically sending the location information of target points along the designated path to the “Sojourner,” which autonomously completed the movement between target points. During the movement, the “Sojourner” needed to identify and avoid unexpected events based on sensor information such as speed, acceleration, rotation speed, and distance.
In 2003, the “Spirit” and “Opportunity” rovers were launched, supporting both remote control mode and autonomous work mode, including autonomous topography assessment, path selection, and mileage estimation. The topography assessment module of “Spirit” and “Opportunity” used images from multiple stereo vision cameras to calculate topographic feature parameters, generating 3D information and forming a 2D global map, based on which autonomous path selection and obstacle avoidance planning were completed, as shown in Figure 2.
Figure 2 Topography Assessment and Path Selection of “Spirit” and “Opportunity”In 2012, the “Curiosity” rover landed on Mars, and compared to “Spirit” and “Opportunity,” the software and hardware onboard “Curiosity” were further enhanced, enabling it to autonomously complete localization, mapping, path selection, and mileage calculation based on image information obtained from multiple sensors, and supporting long-distance autonomous navigation. Similar to “Curiosity,” the “Perseverance” rover, launched in 2020, used a more powerful autonomous navigation system—ENav (Enhanced AutoNav), employing artificial intelligence technology extensively in topography measurement, assessment, and path planning.3. Extraterrestrial Surface Sample Identification and Selection
Mobile robots, during the planetary surface patrolling tasks, acquire a large number of visual images, which usually need to be sent back to the ground for processing, greatly reducing the efficiency of planetary exploration. To address this, NASA’s Jet Propulsion Laboratory (JPL) developed the AEGIS (Autonomous Exploration for Gathering Increased Science) autonomous system, which enables autonomous identification and prioritization of scientific targets on the Mars rover, and can call upon other cameras for high-precision observations of targets, thereby obtaining more valuable scientific images, as shown in Figures 3 and 4. The AEGIS system has been successfully deployed on the “Opportunity,” “Curiosity,” and “Perseverance” rovers.
Figure 3 AEGIS Marking and Prioritizing High-Value Targets
Figure 4 AEGIS Calling Panoramic Camera for High-Value Target Details(2) Intelligent Planning/Navigation1. Robot Motion PlanningSpace robots such as the ETS-VII satellite robot, the International Space Station robotic arm, the Japanese module robot on the International Space Station, the Orbital Express satellite robot, and China’s space station robotic arm mainly perform motion planning through ground commands, remote control, and visual servoing, while artificial intelligence technology has not yet been applied in orbit. AI-based task planning and motion planning for space robots have been a hot research topic in academia. For instance, for on-orbit capture tasks related to space debris or abandoned spacecraft, researchers have conducted studies using machine learning algorithms to optimize the motion trajectory of robots to solve a series of issues such as autonomous collision avoidance during capture, reducing joint torque, minimizing satellite attitude disturbance, and adapting to target uncertainty. For multi-robot space systems, researchers have attempted to reduce the dependence of motion planning on complex dynamic models through reinforcement learning algorithms.2. Autonomous Navigation of Planetary Exploration RobotsAutonomous navigation is a relatively mature area of artificial intelligence application in space. NASA’s “Spirit” and “Opportunity” rovers achieved autonomous navigation capabilities through software upgrades, while “Curiosity” and “Perseverance” further enhanced mobility planning autonomy.
After landing on Mars, “Spirit” and “Opportunity” significantly improved their autonomous capabilities by injecting the “Onboard Autonomous Science Investigation System (OASIS)” software. OASIS can autonomously discover potential high-value scientific targets in navigation camera images according to predetermined criteria, mark and prioritize them, and autonomously modify the rover’s established program to move closer to the target, as shown in Figure 5. OASIS can also be used to autonomously discover, capture, and mark incidental events, such as the formation of dust devils or changes in clouds, as shown in Figure 6. Without the automatic discrimination function of OASIS, the likelihood of capturing such images would be greatly reduced.
Figure 5 MER Autonomously Determining the Exploration Target and Moving Closer
Figure 6 “Opportunity” Captured Dust Devil
“Curiosity” adopted a new autonomous navigation software system, AutoNav, thus gaining long-distance autonomous navigation capabilities, as shown in Figure 7. “Perseverance,” based on the AutoNav system, is equipped with a large number of artificial intelligence technology-based ENav systems, further enhancing autonomous navigation capabilities. On February 8, 2022, NASA announced that “Perseverance” moved autonomously for 243.3m, setting a record for autonomous movement of a rover on the Martian surface, as shown in Figure 8.
Figure 7 NASA’s AutoNav Software Demonstration
Figure 8 “Perseverance” Achieving Rapid Movement with ENavNASA has also addressed the safety and efficiency issues of autonomous navigation by classifying the Martian surface terrain. JPL developed the SPOC (Soil Property and Object Classification) software for Martian surface terrain analysis, which has been applied to the analysis of landing point passability and prediction of rover slip rates in the “Mars 2020” mission. JPL has built a large database AI4Mars using the Mars images obtained from “Spirit,” “Opportunity,” and “Curiosity” to classify and label terrains in images using deep learning techniques, providing fundamental data for autonomous navigation.(3) Intelligent Control1. On-Orbit Robot Operation ControlSo far, the space robots in service on orbit mainly use remote control or teleoperation modes, while also supporting autonomous control modes under visual guidance, but the application is relatively limited.
In 2011, the “Discovery” space shuttle delivered the first humanoid robot, Robonaut-2 (R2), to the International Space Station, to assist and eventually replace astronauts in performing tasks inside and outside the space station. During the on-orbit testing period, R2 mainly operated under ground remote control and on-orbit teleoperation, verifying autonomous control modes on some simple tasks, such as autonomously recognizing tools like wrenches and screwdrivers, as well as switches, buttons, and valves. Based on this, it autonomously completed simple operations such as grasping and handing over, and autonomously judged the execution effects of various operations through visual images, as shown in Figure 9.
Figure 9 R2 On-Orbit Testing—Task Panel OperationApplications of artificial intelligence in autonomous control of space robots are limited, but related theoretical research has been continuously developing, including improving the tracking capability and motion accuracy of space robots in free space using artificial intelligence technology, and training control strategies for robots aimed at space target capture tasks through reinforcement learning.2. Mobile Control of Planetary Exploration RobotsCurrently, all on-orbit planetary exploration robots are wheeled robots, which autonomously control the direction and speed of their wheels to move along a given path. The “Curiosity” rover, due to severe damage to its left front wheel, has added a terrain-adaptive wheel speed control algorithm through software upgrades. This algorithm can autonomously allocate wheel speeds based on real-time suspension and attitude measurement values and the rover’s rigid body kinematics model, coordinating the motion and forces among the wheels.Furthermore, while moving, the rover can also detect unexpected events such as power drops, wheel sinking, mechanism jamming, and body tilting in real time, as well as hardware failures of the computer, camera, suspension, drive components, heating and temperature measuring devices, and inertial measurement units (IMU). When related parameters exceed set thresholds, “Curiosity” can automatically stop moving, capture and send images, and wait for further instructions, thereby enhancing its autonomous survival capability.(4) Human-Machine Interaction1. Humanoid Robot Human-Machine Interaction
R2’s human-machine interaction mode mainly relies on remote control and teleoperation, while also supporting astronauts to interact with the robot through helmets, microphones, and data gloves to control the robot to perform complex operations such as tying ropes, assembling bolts, and manipulating soft objects. R2 can also accept voice commands from astronauts to perform operations such as delivering tools to astronauts, as shown in Figure 10.
Figure 10 Astronauts Controlling R2 in Teleoperation Mode
In August 2013, Kirobo, a conversational robot developed by the University of Tokyo, was launched, becoming the first conversational robot to enter space. Kirobo extensively uses artificial intelligence technology, possessing capabilities such as voice recognition, natural language processing, facial recognition, and emotion recognition, as shown in Figure 11.
Figure 11 Kirobo Interacting with Astronauts on the International Space Station2. Other Robot Human-Machine Interactions
The German Aerospace Center (DLR) launched the artificial intelligence assistant CIMON (Crew Interactive Mobile Companion) in June 2018, which supports astronauts in performing daily tasks on the International Space Station. This robot supports voice interaction with astronauts, having capabilities such as listening, seeing, speaking, understanding, and flying. CIMON can assist astronauts in completing daily tasks through voice prompts and can help astronauts deal with emergency events when they occur, as shown in Figure 12.
Figure 12 CIMON Interacting with Astronauts on the International Space Station(5) Summary of Application StatusIn summary, artificial intelligence has made substantial achievements in the field of space robotics, such as in information perception, effectively supporting human identification and measurement of scientific targets on the Martian surface; in planning and navigation, significantly enhancing the autonomous survival capability of rovers and improving planetary exploration efficiency; in robot control, achieving coordinated motion control of ultra-redundant degree-of-freedom robot systems; and in human-machine interaction, enabling multimodal interaction through voice, facial expressions, and gestures in orbit, making the “astronaut assistant” a reality.
2. Future Application Demands
As human space activities continue to deepen, the application scope of space robots is constantly expanding, and the application scenarios are becoming increasingly complex. Manned spaceflight is the traditional application field for space robots, with typical tasks including space station operation support and human-robot joint planetary exploration. For space station operations, on one hand, robots are needed to replace astronauts in daily tasks such as cleaning, inspection, and experiments; on the other hand, to reduce the risks of astronauts’ extravehicular activities or to lower the difficulty of extravehicular tasks, robots need to replace or assist astronauts in performing extravehicular tasks. In human-robot joint exploration, manned mobile robots need to be used to expand the exploration range of astronauts and provide support for sample identification, auxiliary exploration, and emergency handling.In the field of unmanned deep space exploration, space robots, with their large operational range, long duration, and strong operational capabilities, have become the most effective tools for planetary surface exploration. Future deep space exploration will focus on activities such as landing/attachment on various extraterrestrial bodies, multi-terrain patrolling and data collection, sample collection and storage, in-situ resource utilization, and construction of unmanned research stations, where various space robots will play supporting roles.On-orbit service is another major area of space robot application. With the powerful perception and execution capabilities of space robots, humans can widely carry out activities such as upgrading and restructuring active spacecraft, repairing malfunctioning spacecraft, and capturing and removing space debris. In particular, the recently much-anticipated on-orbit manufacturing and assembly technology for spacecraft components will enable humans to break through transportation constraints and construct super-large spacecraft.The emergence of these new tasks places higher demands on the autonomy and intelligence level of space robots, which can be summarized in the following four aspects: space intelligent perception, space intelligent planning and control, space intelligent interaction, and space swarm intelligence.(1) Space Intelligent Perception. The goal of intelligent perception is to use data obtained from various sensors, through intelligent methods such as machine learning and data mining, to complete the recognition and measurement of the space environment, providing sufficient, high-precision, and real-time information for the planning, control, and evaluation of space robots. Specific demands include 3D reconstruction of the operational/exploration environment, identification of controlled target parameters, recognition and measurement of samples, task evaluation and health monitoring, etc.(2) Space Intelligent Planning and Control. The main goal of intelligent planning and control is to autonomously complete task decomposition, path selection, and trajectory planning based on task objectives, self-state, time, and resource constraints, forming control inputs for execution agencies and controlling them to complete tasks. Specific demands include autonomous selection and prioritization of exploration targets, planning for on-orbit construction/station building tasks, autonomous navigation and movement control, autonomous motion planning, motion control, and force control, etc.(3) Space Intelligent Interaction. Intelligent interaction refers to using interaction methods such as voice, facial expressions, gestures, VR/AR, etc., to recognize and process information through intelligent methods, achieving efficient information transfer between humans and robots. Specific demands include intelligent companion robots for space stations, multimodal interactions between robot astronauts and human astronauts, and intelligent interactions between planetary surface manned mobile robots, exploration robots, and human astronauts.(4) Space Swarm Intelligence. Space robot swarm intelligence refers to complex intelligent behaviors exhibited by multiple space robots gathered within a certain space through organic interaction, coordination, and control among themselves. Specific demands include tasks such as coordinated flight execution by groups of space robots, on-orbit collaborative assembly, planetary surface formation exploration, and planetary surface base construction. In future space missions, the swarm intelligence of robot systems will have tremendous application value.3. Issues to be Resolved
Although it has become a consensus in the industry that artificial intelligence is an effective means to enhance the intelligent autonomy of space robots, based on the reliability and test coverage requirements of aerospace missions, several key issues still need to be addressed before artificial intelligence technology can be truly applied in engineering.
(1) Non-deterministic Boundary Adaptation Issues. Space robots are always in unknown and complex space environments during their mission cycles. Taking the lunar polar region as an example, it has complex lighting, numerous shadow areas, rugged terrain, and unknown geological conditions, and artificial intelligence algorithms must possess adaptability to non-deterministic boundaries.(2) Continuous System Learning Dimension Issues. The state-action space in robot systems is often continuous, and current reinforcement learning algorithms require discretization of the state space. If the discretization is too coarse, it results in significant errors, causing the learning algorithm to fail to converge or yield poor results after convergence; if the discretization is too fine, it encounters the curse of dimensionality, causing the algorithm to fail to converge and unnecessary consumption of computational resources, thus an appropriate granularity should be chosen based on the task situation.
(3) Small Sample Learning Overfitting and Reliability Issues. At present, the efficiency of machine learning relies on the support of large data resources. If the learning data is too sparse, overfitting issues may arise. However, compared to ground data, space missions, especially deep space exploration tasks, are “small-scale” tasks, with limited effective data available for training generated during the task process.
(4) Learning Convergence and Learning Efficiency Issues. Currently, the convergence and learning efficiency of learning algorithms remain bottlenecks that limit the further application of machine learning. Most machine learning methods involve a process of continuously adjusting strategies based on agent outputs and environmental feedback; however, these algorithms often struggle to converge or do so slowly in practical cases, resulting in low learning efficiency.
(5) Simulation Model and Physical Prototype Transfer Deviation Issues. Generally, the main reason for the transfer deviation between simulation models and physical prototypes is often insufficient accuracy of the simulation model. Overly precise modeling must consider the real-time nature of numerical solutions, while insufficient real-time performance will also increase the degree of deviation in the transfer between simulation models and physical prototypes.
(6) Machine Learning Cold Start Issues. Machine learning has long faced cold start issues, especially in the early learning process where function estimates may have significant deviations, increasing learning difficulty and training time. Currently feasible solutions include introducing supervised learning during the early learning phase and improving algorithm performance and on-orbit computing capabilities as software and hardware permit.
(7) Machine Learning Black Box Issues. The existence of black box characteristics severely affects system reliability and will be a key limiting factor for the adoption of intelligent methods in aerospace missions.
Firstly, model interpretability issues. The composition of neural networks may contain hundreds of thousands or even millions of weight coefficients, bias terms, and various nonlinear activation functions, which have no actual physical significance from a physical perspective.Secondly, problem localization issues. Unlike traditional control ideas, intelligent control uses one or more “end-to-end” networks. Designers do not understand the reasons for the values of learned weight coefficients or bias terms, and when strategies do not converge or the environment changes, it often leads to the need for redesign and retraining due to the inability to locate and isolate error parts.
Thirdly, hardware environment dependency issues. The black box nature of neural networks leads to greater dependence on deployment in actual hardware and training in real environments. However, on-orbit validation for space missions is costly, and generally, only limited validation can be achieved in ground simulation environments, which cannot meet the learning model’s dependence on real hardware environments.
Fourthly, behavior unpredictability issues. The black box nature of neural networks leads to unpredictability in action values; if situations arise that have not been covered in training, unpredictable results will occur, making it impossible for designers to guarantee execution outcomes.Fifthly, code verifiability issues. Programs in aerospace missions need to undergo source code testing and code reviews, but the source code of neural networks consists of numerous weight coefficient matrices, bias term matrices, and activation functions, making it impossible to explain the neural network from within; only structural and logical aspects of the code can be reviewed.(8) Machine Learning Testability Issues. Machine learning methods face two key issues regarding testability: firstly, due to the complexity of neural networks, the workload for testing neural networks is substantial, making it difficult to achieve complete coverage testing; secondly, due to the nonlinear characteristics of neural networks, the behavior of untested neural networks is entirely unpredictable..
Wang Yaobing
Researcher at Beijing Spacecraft Design Institute, PhD supervisor, Director of the Beijing Key Laboratory of Space Intelligent Robot Systems Technology and Application. Engaged in research on space robots, spacecraft structures and mechanisms for a long time.
Journal Brief
“China Aerospace” (CN11-2801/V, ISSN1002-7742) was founded in 1987, published monthly, supervised by China Aerospace Science and Technology Corporation, and hosted by the China Academy of Space Technology (China Aerospace 12th Institute). It is an authoritative comprehensive scientific and technological journal with significant influence in the aerospace field.The journal has won the second and third prizes for National Excellent Journals once each, the first prize for Aerospace Science and Technology Journals four times, and the “Double Hundred” journal award from the Chinese Journal Matrix.
Submission Scope
Research on key technologies of aerospace transportation systems, construction of space infrastructure systems, key technology tackling for manned spaceflight, lunar and deep space exploration missions and key technology research, technology experimental satellites and key technology verification, construction of space application service systems, major breakthroughs in the forefront of space science, construction of space environment monitoring systems and disaster early warning, etc.
Contact Information
Phone: 010-68372041Email: [email protected]Address: Room 210, No. 14, Fucheng Road, Haidian District, Beijing
