Abstract
Lattice Semiconductor has launched a sensor hub based on FPGA technology, providing support for the development of intelligent robots. It features flexible I/O interfaces and supports parallel computing near the sensors, enabling connections with multiple sensors and actuators while achieving low-latency and low-power computation. This sensor hub can undertake low-level real-time computing tasks specific to certain sensors within CPU and GPU systems, allowing them to focus on higher-level intelligent task processing. Additionally, Lattice’s solution offers advantages such as small size, low power consumption, and no need for cooling systems (like fans), making it highly suitable for robotic applications in factory environments.
Introduction
For over half a century, autonomous mobile robots have been collaborating with humans in industrial manufacturing. Since the development and deployment of the world’s first industrial robot in the 1950s, companies have begun to assign tedious or dangerous tasks to these machines, allowing employees to focus on more intricate work. Today, the application of such robots is no longer limited to the industrial sector but has expanded into various verticals including healthcare, retail, and agriculture.Moreover, advancements in technology—especially in the fields of artificial intelligence (AI) and machine learning—have given rise to a new generation of smarter robots. These robots are no longer confined to repetitive tasks but can perform much more complex operations. For instance, with capabilities such as computer vision and autonomous mobility, these robots can handle a range of tasks from product assembly and quality inspection to advanced threat detection and response.In short, they have become a vital asset in augmenting the workforce, ensuring extremely high precision while possessing limitless productivity potential. However, as companies’ demands for robotic assistants continue to rise, the design complexity of such systems is also increasing exponentially.
Growing Design Challenges
Intelligent robots equipped with AI technology require more sensors and actuators than their predecessors, including cameras, LiDAR, radar, inertial measurement units (IMU), motor encoders, and pressure sensors. Additionally, these robots must perform more complex computational tasks in real-time, such as 3D vision processing, simultaneous localization and mapping (SLAM), and point cloud calculations.Therefore, building such systems requires support in two areas: first, equipping more I/O interfaces to accommodate the large number of sensors; second, enhancing the performance of processing modules (such as CPU, GPU, or NPU) to achieve more complex computational functions. However, there is a contradiction between these two demands. To achieve higher performance, processing modules need to be manufactured using advanced process technologies—by shrinking transistor sizes and integrating more transistors. However, under advanced processes, the design difficulty of I/O interfaces significantly increases. For example, even with advanced processes, the size of I/O interfaces cannot be reduced simultaneously, leading to relatively higher costs; furthermore, the difficulty of supporting multiple voltages in advanced processes also increases. This results in high-performance processing modules’ I/O interfaces often being “integrated and simplified,” supporting only standard high-bandwidth interfaces like PCIe and Gigabit Ethernet, rather than accommodating low-bandwidth, generalized non-standard interfaces. This situation often prevents designers from relying solely on processing modules to achieve connections with the various sensors required by the system.Moreover, even if a certain computing module can provide sufficient or compatible I/O interfaces to meet connection needs, the practice of directly transmitting large amounts of raw sensor data to that computing module is not energy-efficient. More importantly, traditional computing modules like CPU and GPU have not been optimized for the real-time processing capabilities required by intelligent robots. For instance, if critical tasks such as sensor fusion are handed over to the CPU, it can introduce latency and significantly reduce system efficiency.Fortunately, hardware designers and developers are focusing on innovative products to help fill these technological gaps.
FPGA: A Highly Valuable Hardware Solution
FPGA is one of the products that can solve the aforementioned problems. This type of flexible semiconductor device can serve as a bridge between sensors, actuators, and CPU. With real-time, near-sensor computing capabilities, FPGA can undertake low-level, sensor-specific tasks, supporting the development of smarter and more responsive robots.After the first layer of data processing through FPGA, the data is transmitted to the CPU via standard high-bandwidth channels. By splitting the processing tasks of intelligent robots in this way, FPGA can offload part of the processing load from the CPU, thereby saving energy for higher-level computations such as trajectory planning, clustering analysis, and object detection. This allows the CPU to focus on optimization and decision-making tasks that are difficult to accomplish at the hardware level.For example, this architecture can help developers overcome several challenges:■ Interconnect:FPGA provides more I/O interfaces, and the protocols and voltage levels of these interfaces can be freely customized. Developers can connect and control more sensors and actuators using interfaces such as Ethernet, SPI, HDMI, MIPI, I2C, UART, and even GPIO, thereby supporting non-standard protocols with cycle-level precision. This expands the options available to developers to meet the needs of different application scenarios.■ Power Consumption: FPGA can achieve hardware-based parallel computing near robot sensors. By processing sensor data in real-time locally and then transmitting the processed data to computing modules like CPU, FPGA helps reduce the overall system energy consumption.■ Latency:FPGA can significantly accelerate critical tasks such as sensor fusion—tasks that integrate data from various sensors like cameras and LiDAR to construct a comprehensive environmental view, thereby enhancing the robot’s accuracy in judgment and decision-making efficiency. For example, if the VLP16 LiDAR sensor transmits 384 sets of distance data every 1.32 milliseconds via the network, FPGA can process this data in just 0.32 milliseconds, achieving a computation speed of 100 million operations per second.These advantages provide designers with the flexibility to integrate a large number and variety of sensors into intelligent robots; while pushing the performance limits of robots, they can also effectively control energy consumption and latency, meeting system constraints.
Lattice Solutions
To demonstrate the performance of the sensor hub using Lattice FPGA, we built a proof-of-concept demonstration system. This demonstration system can simultaneously process data streams from cameras, LiDAR, and radar within a single Lattice Avant™ FPGA. Figure 1 shows the block diagram of this proof-of-concept system.
Figure 1: Lattice Avant FPGA Sensor Hub Functional Concept Demonstration Block DiagramCamera and LiDAR data are transmitted to FPGA in the form of UDP packets via Ethernet. The LiDAR data is processed by the LiDAR processing module: first, the UDP packets are unpacked to extract the distance information from each laser emitter; then, mathematical calculations convert the distance information into point cloud data. This mathematical computation uses 32-bit single-precision floating-point numbers (FP32), and we generated register transfer level (RTL) code from reference C code using high-level synthesis (HLS) tools. Subsequently, the point cloud is mapped to a 2D plane based on the given input perspective for display on the output screen. This functionality is also generated through HLS tools from reference C code. The bounding box matching module serves to fuse the machine learning results output by the camera sensor with data from other sensors.The image data output from the camera is first captured and then downsampled for human detection tasks based on machine learning. The machine learning/vision engine (ML/VE) module runs a neural network to detect the upper body of a person and outputs the corresponding bounding box. This functionality is implemented through the sensAI solution suite.For radar data, the radar’s analog-to-digital converter (ADC) output is transmitted via the LVDS bus. In this proof-of-concept, the radar is configured in a 4-channel mode, with each channel’s data needing to undergo Range FFT (Fast Fourier Transform), Doppler FFT, CFAR, and Angle FFT processing sequentially, ultimately outputting the coordinates and speed information of multiple targets. During development, we used Matlab tools to debug radar parameters and validated the effectiveness of algorithms and parameters using real data captured by the radar in Matlab. After parameter validation, we generated FFT and other processing modules through the Simulink HDL workflow.The bounding boxes output by the human detection neural network are fused with the output data from LiDAR and radar to enhance detection accuracy and decision reliability. Specifically, we calculate the area in the LiDAR point cloud that matches each detection bounding box (i.e., corresponding to a person), thereby determining the actual distance to each person; similarly, by identifying radar points corresponding to a person, we can not only obtain the person’s movement speed but also further confirm that the target is a “real human” (rather than a scaled human image), enhancing detection credibility. All this information is transmitted to the RISC-V CPU, which completes the post-processing and implements the user interface through OpenCV (see Figure 2). This sensor fusion technology can support various intelligent application scenarios: for example, in surveillance systems, there is no need to continuously run the camera and AI/ML modules; instead, the radar can first detect whether there are moving targets, and only when a target is detected does it activate the AI/ML module to process camera data—this can significantly reduce the energy consumption associated with AI/ML computations. Another application scenario is the virtual safety fence in factories: traditional radar-based safety fence sensors need to first “understand” the environment to filter out reflection noise generated by surrounding structures; otherwise, it is difficult to distinguish human signals from the reflection noise of other structures. However, human detection via camera-based AI/ML can solve this problem: the human detection neural network outputs the area occupied by a person (i.e., the region of interest, ROI), and the radar only needs to focus on processing data within that area. This solution eliminates the cumbersome process of measuring and filtering static reflection noise, and even if the environment changes, the AI/ML-based region of interest detection can adapt.The processing times for LiDAR and radar are as follows:■ For the VLP16 LiDAR (which sends 384 sets of distance information via 1 UDP packet every 1.32 milliseconds): we use 1 processing engine (generated via HLS), and at a clock frequency of 100 MHz, it takes only 0.32 milliseconds to process each packet. If higher throughput is needed, multiple processing engines can be deployed in parallel.■ For the TI FMCW 2243 radar: we configured it in a 4-channel mode, sending 1 frame of data every 40 milliseconds. If using 1 processing engine (i.e., processing the 4 channels sequentially), the processing time is 6.5 milliseconds; if independent processing engines are deployed in parallel for each channel, the processing time can be reduced to 1.6 milliseconds at a clock frequency of 100 MHz.
Figure 2: Fusing the bounding box output of the human detection neural network with LiDAR and radar data
Conclusion
Lattice’s FPGA-based sensor hub offers numerous advantages for the development of intelligent robots. It features flexible I/O interfaces that enable the connection and data aggregation of multiple sensors, as well as control of multiple actuators. The hardware-based parallel computing conducted near the sensors allows for low-latency, low-power operations and can undertake low-level real-time computing tasks specific to sensors within CPU, GPU, and NPU systems. This characteristic presents two application possibilities: one is to use lower-end computing modules to reduce BOM costs; the other is to allocate more resources of the computing module to achieve higher-level functions (rather than handling low-level computations), thereby enhancing system performance.In terms of the development process, this solution can integrate high-level synthesis (HLS), MATLAB/Simulink tools, hardware-in-the-loop technology, and optimized RTL code. Last but not least, Lattice’s solution is characterized by its small size, low power consumption, and no need for cooling systems (like fans), making it highly suitable for robots in dusty factory environments and intelligent sensors housed in plastic enclosures.