The development resources provided by NVIDIA mostly belong to the library or API level, including CUDA, CUDNN, CuFFT, CuBLAS, TensorRT, etc. Developers with a solid foundation in C++/Python programming languages are required to leverage the parallel computing advantages of GPU/CUDA, which somewhat limits the popularity of applications related to parallel computing.
DeepStream is a suite of intelligent video analytics that embodies NVIDIA’s cutting-edge technology, integrating deep neural networks and other complex processing tasks into a streaming processing pipeline for near real-time analysis of video and other sensor data.
DeepStream plays a core role in many urban management projects for video analysis. The left side of the image above shows millions of cameras deployed in homes, streets, parking lots, shopping centers, warehouses, and factories. Through intelligent technology based on deep learning, specific information is quickly extracted and sent back to the control center, providing a security monitoring mechanism within designated areas and improving overall operational efficiency.
Current applications include access control, loss prevention, automatic checkout, monitoring, security, automatic inspection (QA), package classification (smart logistics), traffic control/engineering, and industrial automation.
To address the efficiency issues in the aforementioned complex scenarios, not only must the performance of each link be maximized, but it is also crucial to balance each node in the workflow, as this involves the allocation of overall performance.
The diagram below illustrates the standard workflow of DeepStream, showing the eight steps executed on a “single device.” Each step requires corresponding computational resources for processing, and any mishandling in one segment will affect overall performance, making “balance” a critical consideration.
Below is a brief introduction to the key focus of each step (from left to right):
-
COLLECT: This can interface with various data sources, including cameras (CSI, USB, GigE interfaces, etc.), video/image files from storage devices (hard drives, SSDs, etc.), and data coming from network routers.
-
DECODE： This process traditionally consumes a lot of CPU resources, but many NVIDIA devices now provide hardware decoders (DECODERS) and algorithms that can quickly decode different input sources/video formats, offering high compatibility and significantly reducing the CPU load.
-
PRE-PROCESS： In this stage, most tasks require extensive format conversions (e.g., RGB to BGR, HSV color space, etc.) and mathematical transformations (e.g., Hough transform, Canny detection, etc.) using OpenCV and NVIDIA MultiMedia API. These computations exhibit significant parallelization characteristics, which CUDA/GPU excels at. The green section above this block indicates the important role of CUDA.
-
TRACK INFERENCE： The tracking inference environment is the most performance-impacting part of the entire DeepStream, heavily relying on the performance enhancement capabilities of NVIDIA TensorRT inference engine in FP16/INT8. TensorRT is a crucial underlying acceleration API for edge computing applications, which is not a deep learning framework itself but supports most commonly used neural networks and models trained by frameworks, converting these models into TensorRT accelerated engines, making it very powerful.
-
ENCODE： This encodes the results after inference, including the object positions (bounding boxes), categories, and confidence levels, into formats for subsequent visualization.
-
COMPOSITE： This combines the results of various AI computations (vehicle type recognition, color recognition, license plate recognition, etc.) for the next stage of data analysis.
-
ANALYZE DATA： The focus here is on “what aspects the user needs to analyze.” Currently, a prominent case is the traffic and pedestrian statistics in specific locations.
-
VISUALIZE： The visual output can adjust the number of displays based on the GPU display capabilities of the usage environment. Currently, NVIDIA devices can output multiple 4K quality images.
The above is the workflow for completing a standalone application. If satisfied with a single device application, the technology provided in the diagram below can assist you in aggregating specific (object) information obtained from dispersed DeepStream application endpoints to the control center via internet technology, and even allow the control center to issue action commands to designated (groups of) DeepStream devices, which represents a complete architecture of AIOT.
For the actual deployment of IVA applications/services, remote management and control of applications are crucial. The DeepStream SDK can run on any cloud and edge, making it a powerful SDK for handling IoT demands, such as effective bidirectional messaging between edge and cloud, security, intelligent logging, and over-the-air AI model updates.
Combining the above technologies enables the hosting of intelligent video analytics applications online and allows for easy deployment and upgrading of DeepStream application environments.
Therefore, DeepStream not only pursues extreme performance on a single machine but also adds supporting technologies for implementing AIOT applications, mainly including the following two parts:
1. Bidirectional Communication and Cloud Services：
(1) Supports IoT integration interfaces such as Kafka, MQTT, and AMQP.
(2) Supports turnkey features for AWS IoT and Microsoft Azure IoT.
(3) Communication security mechanisms: Provides bidirectional TLS authentication based on SSL certificates and encrypted communication based on public key authentication.
(4) Uses bidirectional IoT messaging to trigger specific event logging using DeepStream.
(5) Through bidirectional messaging between edge and cloud, greater control over use cases can be added, such as remote triggers for event logging, changing operational parameters and application configurations, or requesting system logs.
2. Application Deployment and Management：
(1) OTA over-the-air updates: Seamless updates of the entire application or individual AI models from any cloud registry.
(2) High-performance DeepStream cloud-native applications can be built using NVIDIA NGC containers.
(3) Using DeepStream, applications can be deployed at scale and managed using Kubernetes and Helm Charts.
(4) The intelligent logging feature in DeepStream applications allows for selective recording at the edge to save valuable disk space, enabling faster search capabilities. Cloud-to-edge messaging can quickly trigger recordings from the cloud.
The diagram below shows the software stack of DeepStream, where the underlying CUDA-X part is responsible for processing computational performance on a single machine, while the middle DEEPSTREAM SDK part focuses more on the two aspects of “communication” and “deployment.” The top layer shows the main interfaces currently provided, which are Python and C/C++.
The best news for beginners is that even without knowledge of Python or C/C++ development languages, one can easily use this video analysis tool. The next article will guide you through experiencing the powerful capabilities of DeepStream in the NVIDIA Jetson Nano 2GB development kit without any programming code.