Click on the above “Beginner’s Visual Learning”, select to add Star or “Pinned”
Heavyweight content delivered first-hand
Introduction Traditional deep learning-based object detection networks often resize images during the data preprocessing stage to achieve a uniform size and scale in the feature maps. The resizing aims to facilitate model propagation and fully connected classification. However, resizing inevitably leads to deformation and information loss of the objects in the image. This drawback is particularly evident for small targets with linear shapes and few pixels (e.g., distribution towers).
To address this issue, the author proposes to abandon the resizing operation. Instead, the author introduces a positional encoding multi-head cross-attention mechanism. This allows the model to capture contextual information and learn from multiple representation subspaces, effectively enriching the semantic information of distribution towers. Additionally, the author enhances the spatial pyramid pooling by reshaping three pooled feature maps into a new unified feature map while reducing computational burden. This method allows images of different sizes and scales to generate feature maps of uniform dimensions, which can be used for feature map propagation. The author’s SCAResNet integrates these improvements into the backbone network ResNet. The author evaluated SCAResNet using the image dataset of transmission and distribution infrastructure from Duke University.
Source: Intelligent Bookworm For academic sharing only, please contact for removal if infringed
I Introduction
Detecting transmission and distribution towers is crucial for the safe and reliable operation of the power grid, as the location and quantity of these towers are key parameters in designing the topology of the power network and planning its expansion. The combination of remote sensing (RS) and deep learning techniques as a widely adopted object detection method has multiple advantages, including the ability to quickly cover large areas, reduce human errors, and improve detection accuracy. Convolutional neural networks (CNNs) have become the main method in deep learning because they can automatically learn discriminative features from raw image pixels, capturing complex spatial relationships between image regions and achieving better detection performance. Mainstream object detectors based on CNNs can be classified into anchor-based and anchor-free categories. Anchor-based detectors use predefined anchor boxes to predict object positions and sizes, providing better accuracy but may be affected by biases in anchor box design. Anchor-free detectors directly regress the center and size of the objects without using anchor boxes.
However, detecting transmission and distribution towers is challenging due to their small size and lack of obvious features, especially since distribution towers often appear as black linear objects in remote sensing images. Small objects typically have a smaller spatial footprint and lower contrast with their surrounding environment, making them harder to detect. Furthermore, existing object detectors often tend to favor larger objects when allocating positive and negative samples, resulting in most small targets not receiving positive sample allocation. Therefore, detection performance significantly decreases when detecting small objects, such as transmission and especially distribution towers.
Currently, many works are dedicated to addressing the difficulties in small object detection. Multi-scale feature learning involves learning different sizes of objects separately, mainly addressing the insufficient discriminability of small object features. Feature pyramid-based methods primarily use low-level spatial and high-level semantic information to enhance object features. To address the issue of small objects not being allocated to samples, Xu et al. proposed a Gaussian receptive field-based label assignment (RFLA). As a label assignment method specifically designed for small objects, RFLA draws inspiration from normalized Gaussian Wasserstein distance and effective receptive fields, utilizing Gaussian modeling of ground truth boxes and receptive fields to re-measure the distance between them for label assignment. This method improves the probability of label assignment for small objects and yields good results for both anchor-based and anchor-free detectors.
However, current research on the impact of data preprocessing on small object detection is limited. In the data preprocessing stage, the traditional approach is to resize the data before inputting it into the object detection network to achieve a uniform size and scale for images of different sizes and scales. This practice aims to alleviate the computational burden on the model, and more importantly, to ensure that feature maps can generate feature vectors of the same length. This is an indispensable prerequisite for subsequent fully connected classification. Unfortunately, for small objects, the loss of pixels during resizing in data preprocessing has a more serious impact on extracting features of small objects than on large objects.
As shown in Figure 1, compared to the resized images of transmission towers with rich features, the resized images of distribution towers with small black linear shapes lose more distinctive features and even become a blurred black dot. This is extremely detrimental to subsequent feature extraction, as these small objects lose valuable features from the very beginning.
Therefore, the author proposes to abandon the traditional resizing data preprocessing step. Instead, during the propagation process of the backbone network, the author initially performs positional encoding multi-head cross-attention on feature maps of different sizes and scales. This allows the model to learn the global contextual information of the entire image in multiple sub-representation spaces, enabling the model to learn richer feature information about the location of distribution towers and their surrounding environment in remote sensing images. Subsequently, three rounds of pooling are applied to feature maps of different sizes and scales. The flattened feature vectors of the pooled feature maps are reshaped into new feature maps of the same size and scale. The backbone network SCAResNet designed by the author integrates the aforementioned innovative modules into ResNet, achieving promising results on the image dataset of power transmission and distribution infrastructure released by Duke University.
II Methods
SCAResNet combines the positional encoding multi-head cross-attention (Positional-Encoding Multi-head CCA) and SPPRCSP module based on ResNet, as illustrated in Figure 2.
Positional-Encoding Multi-head CCA Module
In the process of detecting transmission and distribution towers, one challenge is the limited availability of rich features, especially for distribution towers that often appear as small black linear shapes. To mitigate the issue of insufficiently distinct features for distribution towers, the author adopts a cross-attention mechanism (Criss-Cross Attention, CCA) improved based on self-attention to capture contextual information and correlations from long-term dependencies, thus obtaining towers with richer semantics.
As shown in the cross-attention block module in Figure 3, CCA replaces the full-image correlation matrix transformation in self-attention with two consecutive row-column correlation matrix transformations, significantly reducing the number of parameters required by self-attention while maintaining high accuracy.
However, single-head attention processes input data with a single representation, which may limit the model’s ability to capture the complete complexity of the data. In contrast, multi-head attention projects input data into multiple representation subspaces, allowing the model to capture different aspects of information and integrate them into the final output. Therefore, multi-head CCA can allocate multiple attention focuses to different points of interest, as shown in Figure 3, including the surrounding environment of the image and global context, such as transmission lines, residential blocks, and roads. This comprehensive learning approach helps determine which environmental features are more likely to be associated with the presence of distribution towers, thereby assisting in detection.
When calculating attention, the influence of different position keys is not considered, so the purpose of preprocessing absolute position encoding is to add positional information in the attention mechanism to capture positional relationships. The author employs a positional encoding method based on sine and cosine functions, which encodes positional information into periodic high-dimensional vectors that can be easily added to the input data.
SPPRCSP Module
After the positional encoding multi-head CCA module comprehensively extracts contextual information related to distribution towers and retains information, feature maps of different sizes and scales will pass through the author’s designed Spatial Pyramid Pooling Reshape Cross-Stage Part (SPPRCSP) module. This module ensures that the output feature maps are uniform in size and scale while avoiding accuracy loss and reducing the computational cost of subsequent tasks (such as fully connected classification).
Spatial Pyramid Pooling (SPP), proposed by He et al., divides feature maps of various sizes and scales into a set of pyramid layers with fixed-size grids (usually 3). Features within each grid area are pooled independently. Finally, the pooled features from each grid area of the pyramid layer are concatenated to form a fixed-length feature vector. However, this feature vector cannot propagate in a network organized by feature maps; it can only serve as input to a classification network. Therefore, the author introduces the Spatial Pyramid Pooling Reshape (SPPR) layer to enable feature vector propagation within the network.
As shown in the SPPR module in Figure 4, in the early stages of the SPPR layer, the steps of the SPP layer are kept unchanged, with the distinction being the final reshape of the concatenated feature vector. Unlike the standard SPP layer, to successfully reshape the concatenated feature vector into a feature map, the levels of the pooled feature maps in the SPPR layer cannot be arbitrarily specified; they must satisfy the following equation:
Where , , and are the levels of the three pooled feature maps and the reshaped feature map. , , and are arbitrary positive integers. In the author’s study, , , and are determined as , , and .
Once the hierarchy of the pooled feature maps is established, the author can infer the parameters of the pooling layer, including the pooling kernel, stride, and padding values. In the original SPP paper’s equations, during the pooling process of feature maps, the padding size can exceed half the pooling kernel size. This situation may lead to truncation of edge information and loss of valuable features, thereby hindering the model training process. Therefore, the author proposes the following revised equation to calculate the parameters of the SPPR pooling layer:
Where is the judgment value calculated using equation (2). represents the height or width of the input feature map. represents the level of the pooled feature map.
If equation (3) is satisfied, then:
If equation (3) is not satisfied, then:
Due to the lack of resizing, the feature map dimensions from high-resolution remote sensing images are larger. This, in turn, imposes a significant computational burden on the training process. To address this issue, the author introduces a Cross-Stage Part (CSP) structure and DSEConv module.
As shown in Figure 4, the feature maps entering the SPPRCSP structure undergo channel compression to reduce parameters. To compensate for the accuracy loss caused by channel compression and pooling of feature maps, detailed information is integrated into the output results of the SPPR layer via skip connections. Furthermore, the author replaces standard convolutions with depthwise and pointwise convolutions to reduce parameters and FLOPs, while also introducing squeeze-and-excitation attention to help select channels that are more advantageous for small object detection from a global perspective.
III Experiment Results
Dataset
The author used the Electric Transmission and Distribution Infrastructure Images (ETDII) dataset for experiments, which is a publicly available dataset from Duke University. The sources of this dataset include various providers such as CT ECO, USGS, LINZ, and SpaceNet. It consists of 494 image blocks from six countries: the USA, Sudan, New Zealand, Mexico, China, and Brazil, each with unique terrain types, with a resolution of 0.3 meters. The detection categories include distribution towers (DT) and transmission towers (TT), providing 16,418 and 1,385 ground truth boxes for DT and TT, respectively.
According to the size classification standards of the standard COCO dataset, targets smaller than or equal to 32 pixels are considered small, targets larger than 32 pixels but less than or equal to 96 pixels are considered medium, and targets larger than 96 pixels are considered large. In the ETDII dataset, there are 12,713 small targets, of which 6,342 are smaller than or equal to 32 pixels; there are 4,723 medium targets; and 367 large targets.
Experiment Settings
All experiments were conducted on a computer equipped with an NVIDIA GeForce RTX 3060 laptop GPU. Model training was based on PyTorch, with core code built on MMdetection. Pre-trained models from ImageNet were used.
All models were trained using the Stochastic Gradient Descent (SGD) optimizer, with momentum set to 0.9, weight decay set to 0.0001, and a batch size of 16, for a total of 100 training epochs. The initial learning rate was set to 0.001 and decayed at the 60th and 80th epochs. The author used SCAResNet with 50 layers for experimentation. All other parameter settings were the same as those for the RFLA-based detector.
Ablation Study
The author evaluated the impact of different modules on the performance of SCAResNet through ablation experiments to verify the effectiveness of each module. The results of the ablation experiments are presented in Table 1.
Iii-C1 Effect of Positional-Encoding Multi-head CCA
In the author’s experiments, the detector based on RFLA equipped with the positional encoding multi-head CCA module improved the mAP by 0.7%, mAP by 1.4%, mAP by 0.2%, mAP by 1.0%, and mAP by 1.7% compared to the detector using only RFLA.
Iii-C2 Effect of SPPRCSP
In the author’s experiments, the detector based on RFLA combined with the SPPRCSP module improved the mAP by 1.7%, mAP by 2.5%, mAP by 1.1%, and mAP by 1.9%, but decreased by 3.4% compared to the standalone RFLA-based detector in mAP.
Iii-C3 Effect of Combining Positional-Encoding Multi-head CCA and SPPRCSP
The detector based on RFLA, which combined the advantages of the positional encoding multi-head CCA and SPPRCSP modules, improved the mAP by 1.3%, mAP by 4.3%, mAP by 2.1%, and mAP by 1.4% compared to the detector based solely on RFLA.
Contrast Experiments
To verify the performance of SCAResNet in detecting transmission and distribution towers under different detector architectures, the author conducted comparative experiments by replacing the backbone networks of Cascade R-CNN, Faster R-CNN, and FCOS with the author’s designed SCAResNet on the ETDII dataset.
Table 2 presents the experimental results, and Figure 5 illustrates the precision-recall performance of each detector.
It can be seen that the detector based on SCAResNet shows varying degrees of improvement in mAP, mAP, mAP, and mAPm when applied to different detectors. However, mAPl experienced a slight decline. This is due to the significantly lower number of large targets compared to medium and small targets, resulting in an imbalance.
To better visualize the improvements of SCAResNet in feature extraction for transmission and distribution towers, the author used the RFLA-based detector as a baseline and detected remote sensing images primarily containing distribution towers and transmission towers with the baseline equipped with SCAResNet. As shown in Figure 6, it is evident that the SCAResNet-enhanced RFLA-based detector extracted richer features and performed better in detecting transmission and distribution towers.
IV Conclusion
The author proposed a backbone network named SCAResNet specifically designed for small target detection. The author eliminated the traditional scaling operation in data preprocessing, as losing valuable information from the very beginning is detrimental for small targets like transmission towers. The positional encoding multi-head CCA module designed by the author can learn more contextual features from images without losing information. The subsequent SPPRCSP module unifies feature maps of different sizes and scales to consistent sizes and scales, enabling propagation without sacrificing accuracy while reducing parameters. SCAResNet achieved impressive detection results on the ETDII dataset.
References
[1]. SCAResNet: A ResNet Variant Optimized for Tiny Object Detection in Transmission and Distribution Towers.
Download 1: OpenCV-Contrib Extension Module Chinese Version Tutorial
Reply "OpenCV Extension Module Chinese Tutorial" in the "Beginner's Visual Learning" public account backend to download the first OpenCV extension module tutorial in Chinese on the internet, covering installation of extension modules, SFM algorithms, stereo vision, object tracking, biological vision, super-resolution processing, and more than twenty chapters of content.
Download 2: Python Visual Practice Projects 52 Lectures
Reply "Python Visual Practice Projects" in the "Beginner's Visual Learning" public account backend to download 31 visual practice projects including image segmentation, mask detection, lane line detection, vehicle counting, eyeliner addition, license plate recognition, character recognition, emotion detection, text content extraction, facial recognition, etc., to help quickly learn computer vision.
Download 3: OpenCV Practical Projects 20 Lectures
Reply "OpenCV Practical Projects 20 Lectures" in the "Beginner's Visual Learning" public account backend to download 20 practical projects based on OpenCV, achieving advanced learning of OpenCV.
Group Chat
Welcome to join the public account reader group to communicate with peers. There are currently WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions, etc. (will gradually be subdivided in the future). Please scan the WeChat number below to join the group, with a note: "Nickname + School/Company + Research Direction", for example: "Zhang San + Shanghai Jiao Tong University + Visual SLAM". Please follow the format; otherwise, it will not be approved. After successful addition, you will be invited into relevant WeChat groups based on your research direction. Please do not send advertisements in the group; otherwise, you will be removed from the group. Thank you for your understanding~