Click I Love Computer Vision to star and get CVML new technologies faster
This article is authorized for reprint by Machine Heart (WeChat public account: almosthuman2014), and secondary reproduction is prohibited.
Selected from tryolabs.com
Author: Juan Pablo
Translated by Machine Heart
Contributors: Panda
Edge computing is crucial for the booming development of the Internet of Things. Recently, Tryolabs, a machine learning and data science consulting company, released a benchmark report comparing the NVIDIA Jetson Nano, Google Coral development board (with built-in Edge TPU), and Intel Neural Compute Stick, all designed for machine learning. The results show that the NVIDIA Jetson Nano is the undisputed winner in both inference time and accuracy. They also provided results for the Raspberry Pi 3B and NVIDIA 2080ti GPU for reference.
Why Is Edge Computing Necessary? Humans generate and collect data at unprecedented levels. Devices in our pockets can generate massive amounts of data, such as photos, GPS coordinates, audio, and various personal information that we unintentionally leak. Moreover, we not only generate personal-related data but also collect unknown data from many other sources, such as traffic and travel control systems, video surveillance units, satellites, smart cars, and countless other smart devices. The trend of data growth is already established and will continue to grow exponentially. In terms of data points, the International Data Corporation (IDC) predicts that the total amount of data in the world will grow from 33 ZB in 2019 to 175 ZB in 2025, with an annual growth rate of 61%. Although we have been processing data, initially in data centers and then in the cloud, these solutions are not suitable for high-demand tasks with large data volumes. The performance and speed of networks continuously push new limits, leading to a demand for new solutions. Now is the starting point for edge computing and edge devices. This report is a benchmark evaluation of five new types of edge devices. We used different frameworks and models to test which combinations performed best. We will focus on the performance results of edge machine learning.What Is Edge Computing? Edge computing involves placing data processing tasks on devices at the edge of the network, as close to the data source as possible. This computing method enables real-time data processing at very high speeds, which is essential for many complex IoT solutions with machine learning capabilities. On this basis, edge computing can alleviate network pressure, reduce energy consumption, enhance security, and improve data privacy. Using this new paradigm, dedicated hardware and software libraries optimized for edge machine learning come together to create cutting-edge applications and products, enabling large-scale deployment. The biggest challenge in building such applications comes from audio, video, and image processing tasks. Evidence shows that deep learning technology has been very successful in overcoming these difficulties.Implementing Edge Deep Learning Take autonomous vehicles as an example. Autonomous vehicles need to quickly and continuously analyze incoming data to interpret their surroundings and take action within milliseconds. This time constraint means we cannot rely on the cloud to process data streams; instead, we must process locally. However, local processing has a drawback: the hardware is not as powerful as supercomputers in the cloud, and we cannot compromise on accuracy and speed. Solutions to this problem involve either using more powerful and efficient hardware or simplifying the complexity of deep neural networks. To achieve the best results, a balance must be struck between the two. Therefore, the real question is:
To maximize the accuracy and speed of deep learning algorithms, which edge hardware and what type of network should we combine?
In our quest for the best combination of the two, we will compare various current top edge devices with different deep neural network models.
New Edge Device Benchmark Testing We explore the most innovative use cases. Here, we will measure real-time inference throughput through a single image classification task to obtain an approximate frames-per-second processing rate. Our specific approach is to evaluate the top-1 inference accuracy across all categories on a specific subset of the ImagenetV2 dataset and compare the results with certain convolutional neural network models. We also experimented with different frameworks and optimized versions.Hardware Accelerators Although significant work has been done over the past few years to enhance existing edge hardware, we chose the following new devices for experimentation:
-
NVIDIA Jetson Nano
-
Google Coral Development Board
-
Intel Neural Compute Stick
-
Raspberry Pi (reference upper limit)
-
NVIDIA 2080ti GPU (reference lower limit)
The experiments will include the Raspberry Pi and NVIDIA 2080ti to compare the tested hardware with well-known systems, where the Raspberry Pi is an edge device and the NVIDIA 2080ti GPU is commonly used in the cloud. This lower limit is straightforward; we at Tryolabs will design and train our own deep learning models. Thus, we have plenty of computing power available. So, we certainly used it. To determine the lower limit of inference time, we ran tests on an NVIDIA 2080ti GPU. However, since we only used it as a reference, we only ran tests using the unoptimized baseline model. As for the upper limit, we chose the reigning champion: the most popular single-board computer: Raspberry Pi 3B.Neural Network Models This benchmark primarily includes two networks: the well-known Resnet-50 and Google’s brand new EfficientNet released this year. For all benchmarks, we used publicly available pretrained models and ran them on different frameworks. For NVIDIA Jetson, we tried TensorRT optimization; for Raspberry Pi, we used TensorFlow and PyTorch variants; for Coral devices, we implemented the S, M, and L versions of the EfficientNet model for the Edge TPU engine; for the Intel Neural Compute Stick, we used Resnet-50 compiled with the OpenVINO toolkit.Datasets Since all models were trained on the ImageNet dataset, we used the ImageNet V2 MatchedFrequency. It contains 10,000 images divided into 1,000 categories. We ran inference once on each image, saved the inference time, and then calculated the average. We computed the top-1 accuracy for all tests as well as the top-5 accuracy for specific models. Top-1 accuracy: This is the regular accuracy, meaning the model’s answer (the highest probability answer) must match the exact expected answer. Top-5 accuracy: This means any of the top five highest probability answers from the model matches the expected answer. It is important to note that when comparing results, for faster device-model combinations, we ran tests encompassing the entire dataset, while for slower combinations, we only used a subset of the dataset. Results and Analysis The following graphs illustrate the metrics obtained from the experiments. Due to significant differences in inference times between different models and devices, the average inference time is presented in logarithmic form.
First Place in Inference Time:Jetson Nano In terms of inference time, the combination of ResNet-50, TensorRT, and PyTorch on the Jetson Nano won. This combination processed an image in 2.67 milliseconds, or 375 frames per second. This result is surprising as it exceeds NVIDIA’s published inference speed by ten times. The discrepancy in results is likely because NVIDIA used TensorFlow, not PyTorch. Second Place in Inference Time:Coral Development Board Second place goes to the Coral Development Board paired with EfficientNet-S. It completed image processing in 5.42 seconds, or 185 frames per second. This result is similar to Google’s reported speed of 5.5 milliseconds per image, processing 182 frames per second. Although this combination is quite fast, the accuracy is poor. We cannot ascertain the exact validation set used by Google for their reported accuracy, but we suspect their image preprocessing transformation methods differ from ours. Because quantized 8-bit models are very sensitive to image preprocessing, this could significantly impact the results.First Place in Accuracy:Jetson Nano The best accuracy result comes from the combination of Jetson Nano with TF-TRT and EfficientNet-B3, achieving 85% accuracy. However, these results are relative, as some models used larger datasets than others during training. It can be seen that when we input smaller datasets into the model, the accuracy is higher; while using the full dataset results in lower accuracy. This result is due to the fact that we did not randomize the smaller dataset, so the images did not achieve reasonable balance. Availability of Hardware Accelerators Regarding the availability of these devices, developers noted some significant differences. When it comes to selecting and deploying precompiled models and frameworks, Jetson is the most flexible. The Intel Neural Compute Stick follows closely, as it offers good libraries, many models, and excellent projects. Additionally, the second generation of this compute stick has significant improvements over the first generation. The only downside is that their extensive software library, OpenVINO, only supports Ubuntu 16.04 and does not support newer versions of Linux operating systems. Compared to Jetson and Intel Compute Stick, Coral devices have some limitations. If you want to run unofficial models on it, you must convert them to TensorFlow Lite and then quantize and compile them for the Edge TPU. Depending on the model, this conversion may not be possible. Nevertheless, we expect Google to improve future versions of this device.Conclusion This research is based on our exploration of the current best edge computing devices designed for deep learning algorithms. We found that the Jetson Nano and Coral Development Board perform very well in terms of inference time. In terms of accuracy, the Jetson Nano also performs excellently, although this result is relative. Overall, the Jetson Nano is the undisputed winner. However, it must be noted that due to the different designs of the Jetson Nano and Coral, we could not test the same models on both. We believe each device has its own optimal scenario, depending on the specific tasks to be accomplished.
Original link:https://tryolabs.com/blog/machine-learning-on-edge-devices-benchmark-report/
Model Compression and Application Deployment Group
Follow the latest cutting-edge lightweight model design, compression, pruning, and AI inference technology. Scan the code to add CV君 to pull you into the group. (If you are already friends with CV君’s other accounts, please message directly)
(Please be sure to indicate: Deployment)
Friends who like to communicate on QQ can add the official QQ group: 805388940.
(I won’t be online all the time, so please forgive me if I can’t verify promptly.)
Long press to follow I Love Computer Vision