Click the blue text above to follow us
WeChat public account:OpenCV Developer Alliance
Follow for more knowledge on computer vision and deep learning
What do deep learning engineers do?
Deep learning has transformed many industries, and deep learning engineers have become high-paying professions. However, there is also a growing trend of internal competition. In the past, it was enough to know how to train models, but now most deep learning engineer positions require proficiency in both model training and deployment.
It is no exaggeration to say that in the future, only deep learning engineers who can train, deploy, and are proficient in C++ model deployment will continue to have high salaries. Otherwise, according to this trend of internal competition, deep learning engineers who can only train models but cannot deploy them will inevitably face increasing challenges from later developers, and their previously accumulated advantages will disappear. Therefore, mastering skills from data labeling to model training to model deployment is a basic requirement for deep learning engineers and an inevitable trend for the future.
Steps for deep learning engineering
Deep learning engineering consists of two main stages: data preparation, model training, and model deployment.
Deployment scenarios for deep learning models
The main deployment scenarios for deep learning models include:
Cloud deployment scenarios
This is mainly based on cloud servers and distributed services, where companies need to pay for cloud server computing and storage costs. The advantage is that it is easy to scale and convenient for quick deployment and use of model algorithms in multiple locations and nodes; the disadvantage is that compared to edge deployment, the latency is relatively high, reliability is lower, and data security is challenged, not fully utilizing the computing power of edge devices.
Edge (PC-side) deployment scenarios
An ideal choice for high-performance applications, highly customizable (built with application-specific components), and flexible pricing (components can be chosen based on applications). The advantage is that costs are controllable, data security is guaranteed, and low latency and high reliability make it widely adopted in fields such as machine vision and security monitoring, relying on industrial computers + graphics cards to provide computing power for model deployment, supporting defect detection, security monitoring, and automated production, etc. The disadvantage is that this solution still cannot be widely adopted in some cost-sensitive industries.
Edge (ARM, FPGA, and inference boards, smart cameras)
Edge and terminal deployment is an important scenario, typically various AI boxes, including Intel’s NUC boxes, the latest AlxBorad boards, NVIDIA’s Jetson series boards, RK series boards, Raspberry Pi, etc., which have been used to realize various smart devices. Their advantages are low cost, low power consumption, significant savings on peripheral hardware computing power, while being highly reliable and secure, suitable for scenarios with less stringent computing power requirements, supporting various lightweight model deployments. The disadvantage is that it requires deep learning developers to have a high level of skill, as they need to quantize models, and different boards support different model deployment frameworks with varying toolchain software.
Mainstream deep learning deployment frameworks
Deep learning developers should at least master one deep learning model deployment framework, currently, the main frameworks for deep learning model deployment are OpenVINO, TensorRT, ONNXRUNTIME, which are the three mainstream frameworks.
OpenVINO comes from Intel, and the latest version is OpenVINO2023.1
TensorRT comes from NVIDIA, and the latest version is TensorRT8.6
ONNXRUNTIME comes from Microsoft, and the latest version is 1.15.x
For model acceleration and inference on Intel CPU/GPU, the best choice is OpenVINO
For model acceleration and inference on NVIDIA GPU, the best choice is TensorRT
The best choice for high compatibility across different hardware platforms is ONNXRUNTIME
Master the three mainstream deep learning model deployment frameworks to achieve optimal performance for inference on different hardware platforms such as CPU, GPU, AMD, etc. OpenCV Academy has launched a systematic learning roadmap for OpenVINO, TensorRT, and ONNXRUNTIME deep learning deployment. “A craftsman must sharpen his tools before doing his work”, as a deep learning engineer, learn deployment before working; it’s not too late to start now!
Deep learning model deployment roadmap video course
Choose the deep learning model deployment framework to learn based on your needs, all supporting C++ and Python SDK code demonstrations, supporting mainstream model deployment and acceleration techniques such as object detection, instance segmentation, semantic segmentation, etc. Deep learning developers can choose the deep learning model deployment framework they want to learn based on their needs.
