The Importance of Edge AI (Endpoint AI)

1. Edge computing can effectively reduce the consumption of computational resources and alleviate the pressure on cloud computing centers or servers.

2. Edge computing can reduce data transmission latency. Especially in scenarios where real-time performance is required, such as autonomous driving and real-time translation, data transmission delays can severely impact user experience.

3. Edge computing helps protect user privacy. In scenarios involving sensitive data, such as healthcare and financial transactions, protecting user privacy is particularly important.

4. Edge computing also promotes the personalization and customization of large models. Customization based on scene requirements can achieve more accurate predictions and decisions.

5. Deploying large models on endpoint devices can also promote deep integration of large models with endpoint devices, stimulating new application scenarios and innovations. By combining the sensors and cameras of endpoint devices, large models can achieve more precise environmental perception and object detection; by integrating the voice recognition and natural language processing capabilities of endpoint devices, large models can enable smarter human-machine interactions and intelligent assistants.

Three Processing Techniques for Endpoint Models: Pruning, Distillation, Quantization

• Pruning

Model pruning is a model compression technique aimed at reducing model size, computational load, and memory usage while preserving the original model’s performance as much as possible.

The basic principle is to check the weights or other parameters in the model one by one. If a parameter has little impact on the model’s performance, it is set to 0, making the model sparser.

• Distillation

Knowledge distillation is a machine learning technique aimed at transferring the knowledge of a large, complex model (often referred to as the teacher model) to a smaller, more efficient model (commonly referred to as the student model).

The core idea of knowledge distillation is to transform the outputs of the teacher model (which usually include class probabilities) into softened probability distributions, serving as the training target for the student model.

• Quantization

Quantization of large models is a technique that converts the weights and activations of large neural network models from floating-point numbers (FP32) to low-bit-width integers (such as INT8, INT4, etc.). The purpose of this technique is to reduce the model’s storage requirements and computational costs while minimizing performance loss, allowing these models to run on resource-constrained devices.

Challenges in Object Detection

1. Small Object Detection

Definition: Small objects are defined as those that occupy a very small proportion of pixels in an image. These targets are often difficult to detect because they provide limited feature information and are easily obscured by background noise.

Examples: Distant people or vehicles, small animals in aerial images, etc.

2. Occluded Object Detection

Definition: An occluded object is one that is partially or completely obscured by other objects.

Challenges: Occlusion reduces the amount of information available for identifying the target, altering the shape, color, and other features, which increases the difficulty of recognition.

3. Blurred Object Detection

Definition: When a target is in motion or the camera is moving, it may result in a blurred effect in the image.

Challenges: Blurriness can lead to unclear object boundaries and loss of texture features, thereby affecting detection accuracy.

4. Other Difficult Situations

Low contrast targets: Targets with little color or brightness difference from the background.

Lighting changes: The appearance of the same target can change under different lighting conditions.

Viewpoint variation: Observing the same target from different angles may lead to changes in its appearance.

Dense object detection: When multiple targets are closely arranged together, it becomes difficult to separate individuals..

Object Detection Process

1. Two-stage detectors (such as Faster R-CNN): First generate candidate regions, then extract features, classify, and perform bounding box regression for each region.

2. Single-stage detectors (such as YOLO, SSD): Directly detect across the entire image, omitting the candidate region generation process, thus improving detection speed.

• Candidate Region Generation

Use candidate region generation networks (such as Selective Search, EdgeBoxes, or RPN) to generate a series of possible regions containing targets.

In some modern frameworks, such as YOLO (You Only Look Once), predictions are made directly on the entire image, skipping the candidate region generation step.

• Feature Extraction

Utilize Convolutional Neural Networks (CNN) to extract features from candidate regions.

Feature maps capture different levels of information from the image, including edges, textures, etc.

• Classification and Localization

Use classifiers to categorize each candidate region, determining whether it contains a specific class of target.

Simultaneously, compute bounding box regression to adjust the location and size of the candidate regions, making them closer to the actual targets.

• Post-processing

Run Non-Maximum Suppression (NMS) to remove overlapping bounding boxes, retaining the highest-scoring detection results.

Filter out low-confidence detection results based on a set threshold.

• Output Results

Finally, output each detected target’s category, location (given in the form of bounding boxes), and corresponding confidence scores.

Mainstream Object Detection Algorithm Frameworks

• R-CNN Series

R-CNN (Regions with CNN features): This was one of the earliest works to use deep learning for object detection. It first uses selective search (Selective Search) to generate candidate regions, then extracts CNN features for each region, performing classification and bounding box regression. Due to its large computational load, it has a slower processing speed.

Fast R-CNN: Improved upon R-CNN by sharing convolutional computations for all candidate regions, speeding up detection.

Faster R-CNN: Introduced the Region Proposal Network (Region Proposal Network, RPN), which automatically generates candidate regions using convolutional networks, further improving efficiency.

Mask R-CNN: Builds on Faster R-CNN by adding a segmentation module, capable of simultaneously performing object detection and instance segmentation tasks.

• YOLO (You Only Look Once)

YOLO v1: Proposed an end-to-end single-stage detection method that treats the detection problem as a regression problem, directly mapping from the input image to bounding box coordinates and class probabilities. YOLO v1 is known for its speed advantage, but earlier versions had relatively low accuracy.

YOLO v2/v3/v4/v5: Subsequent versions continuously improved the network architecture, feature fusion strategies, modular design, and data augmentation techniques, enhancing detection accuracy and speed.

YOLO v6/v7: Further optimized the model structure, incorporating anchor-free designs and more efficient training methods, improving real-time detection performance while maintaining high accuracy.

YOLOv8/v9:: Provides SOTA object detection, supports multi-scale model adaptation for different needs, employs efficient network modules to enhance feature extraction capabilities, optimizes positive sample allocation and loss functions to improve detection accuracy and speed.

YOLOv10: Utilizes dual label assignment and consistent matching metrics to achieve NMS-free YOLO training, enhancing inference efficiency and improving overall efficiency and accuracy through lightweight design, decoupled downsampling, ranking-guided blocks, large kernel convolutions, and self-attention.

• SSD (Single Shot MultiBox Detector)

SSD: Similar to YOLO, SSD is also a single-stage detector, but it uses feature maps of different scales to detect targets of various sizes. It generates default boxes by adding additional convolutional layers at different levels and classifies and localizes these boxes.

• Other Notable Frameworks

RetinaNet: Proposed focal loss to address the class imbalance issue, improving the detection performance of small objects.

EfficientDet: Combines network structure design, scaling techniques, and composite loss functions to achieve efficient and accurate object detection.

CenterNet: A center-point-based detection method that simplifies the detection process and improves detection speed.

Deformable DETR: An end-to-end object detection model based on Transformer with good generalization capabilities.

The Concept of Open Domain Object Detection

• Traditional Closed-Domain Object Detection

Traditional closed-domain object detection is typically trained on specific datasets, such as COCO or Pascal VOC, which contain a predefined list of categories. Additionally, traditional closed-domain object detection models are designed to recognize only these predefined categories and assume that only known categories will appear in test images.

In closed-domain object detection, the main metrics for evaluating model performance include:

lAverage Precision (AP): Used to evaluate the detection accuracy of the model for a specific category, typically calculated by plotting the precision-recall curve.

lMean Average Precision (mAP): Calculates the average of all categories’ APs, serving as an important metric for overall performance.

• Open Domain Object Detection:

The goal is to detect in the presence of unknown categories, meaning the model needs to handle new categories that were not seen during training. This requires the model to have a certain degree of generalization ability, capable of responding reasonably when facing new categories, such as identifying them as unknown categories rather than incorrectly classifying them as known categories. Open domain detection typically uses more complex datasets that may contain a large number of samples from unknown categories.

In addition to the traditional AP and mAP metrics, the following metrics must also be considered for a comprehensive evaluation of model performance:

lRejection Rate (RR): Measures the model’s ability to refuse classification in the presence of unknown categories. A higher rejection rate indicates that the model is better at recognizing unknown categories.

lOpen-Domain Accuracy (ODA): Combines recognition accuracy for known categories with the ability to reject unknown categories.

lAnomaly Detection Ability (ADA): Evaluates the model’s ability to identify unknown categories or abnormal situations.

lUnknown Class Recognition Rate (UCRR): Measures the model’s ability to correctly identify unknown categories and classify them as unknown.

Challenges in Open Domain Object Detection

Generalization Ability: Generalization ability refers to the performance of machine learning models on unseen data. In an open domain scenario, models often need to handle various unknown categories, requiring them to accurately recognize and classify data within known categories while also responding reasonably when encountering unknown categories.

Data Imbalance: Open domain datasets often contain a large number of unknown category instances, which can lead to the model being biased towards predicting unknown categories. Data imbalance refers to the phenomenon where there is a significant difference in the number of samples for different categories in the training dataset. This imbalance issue is particularly pronounced in open domain datasets, as they typically contain a large number of instances from unknown categories that do not belong to known classifications. This imbalance can lead machine learning models to favor predicting more common categories, including unknown categories, thereby affecting the model’s recognition accuracy for known categories.

Evaluation Standards: Determining an appropriate evaluation method to measure the model’s handling of unknown categories is a challenging issue. In open domain classification tasks, models must accurately recognize and classify instances of known categories while also reasonably handling instances of unknown categories. Due to the diversity and unpredictability of unknown categories, traditional evaluation metrics (such as accuracy, precision, recall, and F1 score) may be insufficient to comprehensively assess model performance in this regard. Evaluating a model’s handling of unknown categories requires considering various key factors, including anomaly detection performance, classification performance, overall performance, uncertainty estimation, and robustness testing.

Computational Resources: Computational resources are crucial for open domain object detection tasks, as processing large-scale datasets is involved during both training and use. To further accelerate training speed and handle larger datasets, distributed training methods are often employed, which require specialized distributed computing frameworks (such as TensorFlow’s Distribution Strategies, PyTorch’s DistributedDataParallel, etc.) and corresponding network infrastructure support. In the inference phase, for certain scenarios, such as real-time object detection in autonomous vehicles, processing may need to occur at the device end (edge side), which requires the device to have a certain level of computational capability or rely on edge computing services for additional support.

Anomaly Detection: Anomaly detection is an important challenge in open domain object detection tasks, especially when facing unknown categories or rare situations. In open domain object detection tasks, models must not only identify common known categories but also distinguish between normal and abnormal situations. This is because open domain scenarios often include diverse backgrounds and unknown objects that may not have appeared in the training dataset. For instance, in a surveillance camera system, besides recognizing common objects like pedestrians and vehicles, it may also need to detect sudden appearances of atypical objects, such as animals, abandoned items, or other unusual events. Since the model was only exposed to a limited number of known categories during training, when encountering unknown categories, it may misclassify them as one of the known categories or fail to provide any meaningful prediction results.

Model Design: Designing a model architecture that can recognize both known and unknown categories in open domain object detection tasks is a challenging task. Open domain object detection tasks require models to accurately recognize known categories that appeared in the training data while also being capable of recognizing unknown categories or abnormal situations.

Drone-Borne Endpoint AI

1. Open Object Detection (such as detecting traffic accidents on roads, detecting large trucks on overpasses, etc.)

2. Difficult Object Detection (small targets, blurred, multi-angled, etc.)

3. High real-time requirements, needing to form a business loop with positioning, flying, and photographing

4. Lightweight models, less than 6T

   This article is for learning and exchange purposes only. If there is any infringement, please contact the author for deletion.