Author Introduction: Liu Shaoshan, co-founder of PerceptIn. PhD in Computer Science from the University of California, Irvine, with research interests in intelligent perception computing, system software, architecture, and heterogeneous computing (FPGA, GPU). Currently, PerceptIn focuses on core SLAM technology for augmented reality, virtual reality, and robotics, as well as its implementation and optimization on smart hardware. Before founding PerceptIn, he worked at Baidu’s US R&D center.
Editor: Zhou Jianding ([email protected])
This article is an original piece from “Programmer” and may not be reproduced without permission. For more exciting articles, please subscribe to the 2016 issue of “Programmer” at http://dingyue.programmer.com.cn
Ten years ago, Microsoft founder Bill Gates expressed his vision for the future in his article “A Robot in Every Home”: robots will enter every household like personal computers, serving humanity. With the rapid development of artificial intelligence and smart hardware in recent years, by 2016, the author firmly believes that various technologies have matured, and intelligent robots will soon enter the commercialization era. Gates’ vision is likely to be realized within 5 to 10 years.
To give robots intelligence, we must first endow them with perceptual capabilities. Perceptual computing, especially vision and deep learning, typically involves a large amount of computation and high performance requirements. However, robots are limited by battery capacity, leaving a low amount of energy available for computation. In addition, as perception algorithms continue to evolve, we need to constantly update the perceptual processors of robots. Compared with other processors, FPGAs have low power consumption, high performance, and programmability, making them very suitable for perceptual computing. This article first analyzes the characteristics of FPGAs, then discusses the acceleration and energy savings of perception algorithms using FPGAs, and finally talks about the support of robot operating systems for FPGAs.
FPGA: High Performance, Low Power Consumption, Programmable
Compared with other computing platforms such as CPUs and GPUs, FPGAs possess characteristics of high performance, low power consumption, and hardware programmability. Figure 1 introduces the hardware architecture of FPGAs, which mainly consists of three parts: input-output logic, primarily used for communication between the FPGA and other external components such as sensors; computational logic components, mainly used to construct computational modules; and a programmable interconnect network, primarily used to connect different computational logic components to form a calculator. During programming, we can map computational logic to hardware, adjusting the network connections to link different logic components together to complete a computational task. For example, to accomplish a task of image feature extraction, we would connect the FPGA’s input logic with the camera’s output logic to allow the image to enter the FPGA. Then, we connect the FPGA’s input logic with multiple computational logic components, allowing these components to extract feature points from each image area in parallel. Finally, we can connect the computational logic components with the FPGA’s output logic to summarize and output the feature points. Thus, it can be seen that FPGAs typically hardwire the data flow of algorithms and execution instructions into hardware logic, avoiding the Instruction Fetch and Instruction Decode operations of CPUs.
Figure 1 FPGA Hardware Architecture
High Performance
Although the frequency of FPGAs is generally lower than that of CPUs, they can achieve a high degree of parallelism in hardware calculators. For instance, a typical CPU can only handle 4 to 8 instructions at a time, while using data parallelism on FPGAs can allow processing of 256 or more instructions simultaneously, enabling FPGAs to handle a much larger data volume than CPUs. Furthermore, as mentioned above, FPGAs generally do not require Instruction Fetch and Instruction Decode, reducing these pipeline processes and saving considerable computation time.
To provide the readers with a better understanding of FPGA acceleration, we summarize a study conducted by Microsoft Research in 2010 on the FPGA acceleration of the BLAS algorithm. BLAS is a low-level library for matrix operations widely used in high-performance computing, machine learning, etc. In this study, the researchers analyzed the acceleration and energy consumption of BLAS using CPUs, GPUs, and FPGAs. Figure 2 compares the execution time of the GaxPy algorithm per iteration on FPGAs versus CPUs and GPUs, showing that both GPUs and FPGAs achieved a 60% acceleration relative to CPUs. The results displayed are for small matrix operations; as the matrix size increases, the acceleration ratio of GPUs and FPGAs relative to CPUs becomes more pronounced.
Figure 2 GaxPy Algorithm Performance Comparison (Unit: Microseconds)
Low Power Consumption
FPGAs have a significant power consumption advantage over CPUs and GPUs for two main reasons. First, there is no Instruction Fetch and Instruction Decode in FPGAs; in Intel’s CPUs, due to the use of CISC architecture, the Decoder alone accounts for 50% of the chip’s power consumption; in GPUs, Fetch and Decode consume about 10% to 20% of energy. Secondly, the main frequency of FPGAs is much lower than that of CPUs and GPUs, typically below 500 MHz, while CPUs and GPUs operate between 1 GHz and 3 GHz. This substantial frequency difference leads to much lower energy consumption in FPGAs compared to CPUs and GPUs.
Figure 3 compares the energy consumption of FPGAs versus CPUs and GPUs during the execution of the GaxPy algorithm per iteration. It can be seen that the energy consumption of CPUs and GPUs is similar, while the energy consumption of FPGAs is only about 8% of that of CPUs and GPUs combined. This shows that FPGA computation is 60% faster than CPUs, while the energy consumption is only 1/12 of that of CPUs, providing a significant advantage, especially in energy-constrained situations, where using FPGAs can greatly extend battery life.
Figure 3 GaxPy Algorithm Energy Consumption Comparison (Unit: Millijoule)
Programmable Hardware
Since FPGAs are hardware programmable, they can iteratively update hardware logic compared to ASICs. However, FPGAs are often criticized because writing algorithms into FPGA hardware is not an easy process; the technical barrier is much higher than programming on CPUs and GPUs, and the development cycle is significantly longer.
Figure 4 Comparison of Traditional FPGA Development Process and C-to-FPGA Development Process
Figure 4 shows the comparison between the traditional FPGA development process and the C-to-FPGA development process. In the traditional FPGA development process, we need to translate the algorithm written in C/C++ into hardware language based on Verilog line by line, and then compile the Verilog to write the logic into hardware. With the development of FPGA technology in recent years, the technology to compile directly from C to FPGA has gradually matured and is widely used in Baidu. In the C-to-FPGA development process, we can add Pragmas in the C/C++ code to indicate which computational kernels should be accelerated, and then the C-to-FPGA engine automatically compiles the code into hardware. Based on our experience, using the traditional development process, completing a project takes about six months, while using the C-to-FPGA development process, a project can be completed in about two weeks, achieving more than a tenfold increase in efficiency.
Acceleration of Perceptual Computing on FPGAs
Next, we will mainly introduce the acceleration of robotic perceptual computing on FPGAs, particularly in feature extraction and position tracking calculations (which can be considered the eyes of the robot), as well as deep learning calculations (considered the brain of the robot). Once a robot has eyes and a brain, it can move in space and locate itself while recognizing objects it sees.
Feature Extraction and Position Tracking
The main algorithms for feature extraction and position tracking include SIFT, SURF, and SLAM. SIFT is an algorithm for detecting local features, which extracts feature points from an image and provides descriptions of their scale and orientation for matching image feature points. The SIFT feature matching algorithm can handle matching problems between two images under translation, rotation, and affine transformations, boasting strong matching capabilities. The SIFT algorithm consists of three major steps: 1. Extracting key points; 2. Attaching detailed information (local features) to key points, known as descriptors; 3. Finding several pairs of matching feature points by comparing two sets of feature points (with corresponding feature vectors), thus establishing correspondences between the objects. The SURF algorithm is an improvement over SIFT, primarily increasing the execution efficiency of SIFT by utilizing integral images and Haar derivatives. SLAM, or Simultaneous Localization and Mapping, aims to create a map of the environment while the robot moves and simultaneously determine the robot’s position within that map. This technology allows robots to localize themselves without relying on external signals (WIFI, Beacons, GPS), making it particularly useful in indoor positioning scenarios. The localization method primarily utilizes Kalman filters to fuse information from various sensors (images, gyroscopes) to infer the current position of the robot.
To help readers understand the acceleration and energy savings provided by FPGAs for feature extraction and position tracking, we will focus on a research study from UCLA on accelerating feature extraction and SLAM algorithms on FPGAs. Figure 5 shows the acceleration ratios of FPGAs relative to CPUs when executing SIFT feature-matching, SURF feature-matching, and SLAM algorithms. After using FPGAs, SIFT and SURF feature-matching achieved acceleration factors of 30 and 9, respectively, while the SLAM algorithm achieved a 15-fold acceleration. Assuming images enter the processor at a rate of 30 FPS, the perceptual and localization algorithms must complete processing an image within 33 milliseconds, which places significant pressure on CPUs. After using FPGAs, the entire processing flow has been accelerated by more than ten times, making high frame rate data processing feasible.
Figure 5 Perceptual Algorithm Performance Comparison (Unit: Acceleration Ratio)
Figure 6 illustrates the energy savings of FPGAs relative to CPUs and GPUs when executing SIFT, SURF, and SLAM algorithms. After using FPGAs, SIFT and SURF achieved energy savings ratios of 1.5 and 1.9, respectively, while the SLAM algorithm achieved a 14-fold energy savings ratio. Based on our experience, if a robot uses a multi-core mobile CPU to run this set of perceptual algorithms, the battery will deplete in about 40 minutes. However, if FPGAs are used for computation, the mobile battery can last over 6 hours, achieving about 10 times overall energy savings (since the computational load of SLAM is much higher than that of feature extraction).
Figure 6 Perceptual Algorithm Energy Consumption Comparison (Unit: Energy Savings Ratio)
In summary, using FPGAs for visual perception and localization computations not only improves the perceptual frame rate, making it more accurate, but also saves energy, allowing computations to last for several hours. Once the perceptual algorithms are determined and the demand for chips reaches a certain volume, we can also design FPGA chips as ASICs to further enhance performance and reduce energy consumption.
Deep Learning
Deep neural networks are neural networks with at least one hidden layer. Similar to shallow neural networks, deep neural networks can model complex nonlinear systems, but the additional layers provide a higher level of abstraction, thus enhancing the model’s capabilities. In recent years, Convolutional Deep Neural Networks (CNNs) have made significant progress in computer vision and automatic speech recognition. In the visual domain, Google, Microsoft, and Facebook have continuously refreshed recognition rate records in the ImageNet competition. In speech recognition, Baidu’s DeepSpeech 2 system has significantly improved vocabulary recognition rates compared to previous systems, reducing the word recognition error rate to around 7%.
To help readers understand the acceleration and energy savings provided by FPGAs for deep learning, we will focus on a collaborative study between Peking University and the University of California on accelerating CNN algorithms using FPGAs. Figure 7 shows the time comparison between FPGAs and CPUs when executing CNNs. When running one iteration, using a CPU takes 375 milliseconds, while using an FPGA only takes 21 milliseconds, achieving an acceleration ratio of about 18. If this CNN computation has real-time requirements, such as needing to keep up with camera frame rates (33 milliseconds/frame), then CPUs would not meet the computation requirements. However, with FPGA acceleration, CNN calculations can keep up with camera frame rates and analyze each frame.
Figure 7 CNN Performance Comparison (Unit: Milliseconds)
Figure 8 illustrates the energy consumption comparison between FPGAs and CPUs when executing CNNs. When performing one CNN computation, using a CPU consumes 36 joules, while using an FPGA only consumes 10 joules, achieving an energy savings ratio of about 3.5. Similar to SLAM calculations, using FPGA acceleration and energy savings makes real-time deep learning computations easier to run on mobile devices.
Figure 8 CNN Energy Consumption Comparison (Unit: Joules)
FPGA and ROS Robot Operating System Integration
The previous sections discussed the acceleration and energy savings of FPGAs for perceptual algorithms, highlighting the significant advantages of FPGAs in perceptual computing compared to CPUs and GPUs. This section introduces the current usage of FPGAs in the robotics industry, particularly their integration with the ROS robot operating system.
The Robot Operating System (ROS) is a software development framework specifically designed for robotics. It provides services similar to an operating system, including hardware abstraction, low-level device control, common functionality execution, inter-process messaging, and package management. It also offers tools and libraries for acquiring, building, writing, and executing multi-machine fusion programs. The primary design goal of ROS is to improve code reuse in the field of robotics research and development. ROS is a distributed processing framework (also known as Nodes), allowing executable files to be designed separately and loosely coupled at runtime. These processes can be encapsulated in packages and stacks for easier sharing and distribution. ROS also supports a joint system of code libraries, facilitating collaboration. ROS is currently widely used in various robots and is gradually becoming the standard operating system for robots. In the 2015 DARPA Robotics Challenge, more than half of the competing robots used ROS.
With the development of FPGA technology, more and more robots are incorporating FPGAs, and there is an increasing demand within the ROS community for FPGA compatibility. One example is the Sandia Hand robotic arm from the Sandia National Laboratories. As shown in Figure 9, the Sandia Hand uses FPGAs to preprocess information from the camera and the robot’s palm, then passes the preprocessed results to other computation nodes in ROS.
Figure 9 ROS Support for FPGA in Sandia Hand
To enable connectivity between ROS and FPGAs, the Sandia Hand employs the Rosbridge mechanism. Rosbridge connects ROS with non-ROS programs through a JSON API. For example, a ROS program can connect to a non-ROS network frontend via the JSON API. In the design of the Sandia Hand, a ROS Node connects to the FPGA processor via the JSON API, allowing the FPGA to transmit data and initiate computation instructions, then retrieve the computation results from the FPGA.
Rosbridge provides a communication mechanism for the connectivity between ROS and FPGAs; however, under this mechanism, ROS Nodes cannot run on FPGAs, and the connection via the JSON API incurs some performance overhead. To achieve better coupling between FPGAs and ROS, recent researchers in Japan proposed the design of ROS-Compliant FPGAs, allowing ROS Nodes to run directly on FPGAs. As shown in Figure 10, in this design, the FPGA implements an input interface that can directly subscribe to ROS topics, allowing data to seamlessly flow into the FPGA computation unit. Additionally, an output interface is implemented on the FPGA, enabling ROS Nodes on the FPGA to directly publish data for other ROS Nodes subscribing to that topic to use the data produced by the FPGA. In this design, developers can seamlessly integrate their FPGA calculators into the ROS-compliant FPGA framework, connecting with other ROS Nodes effortlessly.
Figure 10 FPGA as Part of ROS
Recently, in communications with the Open Source Robotics Foundation, which operates ROS, it has been found that an increasing number of robotic developers are using FPGAs as computational units for sensors and controllers, leading to a growing demand for FPGA integration into ROS. It is believed that ROS will soon produce a tightly coupled solution with FPGAs.
Looking to the Future
FPGAs have low power consumption, high performance, and programmability, making them very suitable for perceptual computing. Particularly in energy-constrained situations, FPGAs exhibit significant performance and energy consumption advantages over CPUs and GPUs. Additionally, as perceptual algorithms continue to evolve, we need to continually update the perceptual processors of robots. Compared to ASICs, FPGAs also have the advantage of hardware upgradability and iteration. For these reasons, the author firmly believes that FPGAs will be one of the most important chips in the era of robotics. Due to the low power consumption characteristics of FPGAs, they are well-suited for data preprocessing tasks for sensors. It can be foreseen that the close integration of FPGAs and sensors will soon become widespread. Subsequently, with the continuous optimization of visual, speech, and deep learning algorithms on FPGAs, FPGAs will gradually replace GPUs and CPUs as the main chips in robots.
Recommended Reading from “Programmer”:
-
Interview with “Father of PHP” Rasmus Lerdorf
-
Exploring Core Technologies of Autonomous Driving: Optical Radar
-
Detailed Explanation of Taobao’s Big Second System Design
-
Seven-Year Alibaba Veteran Talks about the Growth of Newcomers
-
Critique of Spark: Summary of Issues with Version 1.6
-
How to Write C Language in 2016
-
Practice and Experience of Highly Available Systems at Dianping
-
Application of Topological Data Analysis in Machine Learning
For more exciting articles, please scan the QR code below to subscribe to the 2016 issue of Programmer:
