Click the “Visual Learning for Beginners” above, and choose to add “Star” or “Pin“
Important content delivered first-hand
Introduction: The extraction of edges and contours is a very tricky task, as details may be obscured by overly strong image lines. Texture itself is a weak edge distribution pattern, and hierarchical representation is a commonly used method, often referred to as scale space. In the past, when working on mobile visual platforms, we sometimes had to disable certain image processing functions due to feature distortion. Now, the CNN model, which has a natural feature description mechanism, provides a good tool for image preprocessing, integrating image processing and visual preprocessing.
Edge Extraction
1. HED
Holistically-Nested Edge Detection (HED) is a deep learning algorithm for edge extraction with two main features: (1) holistic image training and prediction; (2) multi-scale, multi-layer feature learning. This deep model utilizes fully convolutional networks to automatically learn rich hierarchical representations (guided by deep supervision based on side responses).
Multi-scale deep learning can be divided into four categories: multi-stream learning, skip-net learning, multi-input single model, and independent network training, as shown in the figure: (a) multi-stream architecture; (b) skip-net architecture; (c) single model with multi-scale input; (d) independent training of different networks; (e) HED, which adds multiple side outputs.
In terms of representation and computational complexity, methods (a) to (d) have redundancies. (e) HED is a relatively simple variant that can produce predictions from multiple scales. This structure can be interpreted as the “holistic nesting” version of the “independent network” method (d), which includes a single-stream deep network with multiple side outputs. Hidden layer supervision can improve optimization and generalization for image classification tasks. If unified output is needed, multiple side outputs can flexibly add extra fusion layers.
As shown in the figure, the network architecture of HED inserts side output layers after convolutional layers, applying deep supervision to each side output to guide it to become edge predictions. HED outputs are multi-scale and multi-layer, with side output areas decreasing and receptive fields increasing. A weighted fusion layer automatically learns how to combine outputs from multiple scales. The entire network is trained through multiple error propagation paths (dashed lines).
The following figure shows part of the experimental results of HED: (a) test image; (b) manually annotated edges; (c) HED results; (d), (e), and (f) side responses from layers 2, 3, and 4 of the convolutional neural network; (g), (h), and (i) edge responses from the Canny edge detector at scales σ=2.0, σ=4.0, and σ=8.0. HED shows a clear advantage over Canny in terms of consistency.
2. CASENet
CASENet associates each edge pixel with more than one edge class. It is an end-to-end deep semantic edge learning architecture based on ResNet, and also a skip-layer architecture where the top convolutional layer’s class edge activations are shared and fused with the same set of bottom features. A multi-label loss function supervises the fusion of activations.
A comparison of three CNN architectures is shown in figures (a) to (c). The solid rectangles in the figure represent combined blocks of CNN layers. The reduced width indicates a 2x decrease in the spatial resolution of the output feature map. The numbers next to the arrows indicate the number of output feature channels. The blue solid rectangles are stacks of ResNet blocks. The purple solid rectangles are classification modules. The red dashed box indicates outputs monitored by the loss function. The gray solid rectangles are side feature extraction modules. The dark green solid rectangles are fusion classification modules that perform K groups of 1×1 convolutions. Figures (d) to (h) detail the modules in figures (a) to (c), where the rectangles represent input and output feature maps.
Figure (a) is the base network, using a fully convolutional network framework. The average pooling and fully connected layers are removed from ResNet-101, keeping the bottom convolutional blocks; the stride of the first and fifth convolutional blocks in ResNet-101 is changed from 2 to 1; and dilation factors are introduced into subsequent convolutional layers to maintain the same receptive field as the original ResNet; on top of the base network, a classification module (figure (d)) is added as a 1×1 convolutional layer, followed by bilinear upsampling (implemented by K-group deconvolution layers) to produce a set of K activation maps {A1, …, AK}, where each activation map has the same size as the image; then, the sigmoid unit defined by Yk(p)=σ(Ak(p)) models the probability of a pixel belonging to the k-th class edge.
The basic idea of the previous HED network is to consider not only the top network loss but also the bottom convolutional loss. Additionally, a fusion edge map is obtained through the linear combination of supervised side activations. However, HED only performs binary edge detection. The Deep Supervision Network (DSN) extends this architecture, handling K channels of side outputs and K channels of final output, as shown in figure (b). In this network, the aforementioned classification module is connected to the output of each residual block, producing five side classification activation maps {A(1), …, A(5)}, each with K channels. Then, these five activation maps are fused through a sliced concatenation layer (the color in the figure indicates the channel index in figure (g)) to produce a 5K channel activation map:
Af feeds into the fusion classification layer, which performs K-group 1×1 convolutions (figure (f)) to produce K-channel activation map A(6). Finally, six loss functions are computed for the activation maps {A(1), …, A(6)}, providing deep supervision for the network.
By supervising five side activations, it implicitly constrains each channel of those side activations to carry only the most relevant information for that class. Due to sliced concatenation and grouped convolutions, the fused activation for pixel p is given by:
CASENet adopts a nested architecture with the following improvements:
① Replace the bottom classification module with a feature extraction module.
② Place the classification module and apply supervision only at the top of the network.
③ Use shared concatenation (figure (h)) instead of sliced concatenation.
The difference between side feature extraction and side classification is that the former outputs only a single channel feature map F(j) instead of K-class activations. Shared concatenation copies bottom features F = {F(1), F(2), F(3)} from Side-1-3, connecting each with K top activations:
The connected activation maps are then fed into the fusion classification layer with K-group convolutions, producing K-channel activation map A(6).
Generally, CASENet is considered a network that jointly detects edges and classifies them, allowing lower-level features to participate and enhancing high-level semantic classification through a skip-layer structure.
The following figure compares the results of CASENet and DSN, from left to right: input, ground truth, DSN, and CASENet. CASENet shows better detection quality on challenging targets, while DSN has more false positives on non-edge pixels.
Contour Extraction
1. DeepEdge
Previously, low-level features such as texture or saliency were mostly used to detect contours, while DeepEdge utilizes object-related features as high-level cues for contour detection. It is designed as a multi-level deep network consisting of five convolutional layers and a bifurcated fully-connected subnetwork. From the input layer to the fifth convolutional layer is a pre-trained network directly used for image input at four different scales. These four parallel and identical data streams are connected to a bifurcated subnetwork composed of two independently trained branches, one branch learning to predict contour likelihood (targeting classification), while the other branch learns the presence of contours at given points (based on regression measures). The following figure shows the DeepEdge architecture, where the Canny edge detector extracts candidate contour points, and extracts patches of four different scales around each candidate point while passing through the pre-trained KNet’s five convolutional layers.
A single-scale schematic diagram explaining the model architecture is given below: First, input patches centered on candidate points pass through the five convolutional layers of KNet; to extract high-level features, feature maps around the center point are extracted from small sub-volumes at each convolutional layer, and maximum, average, and center pooling are performed on these sub-volumes. The pooling values are fed into the bifurcated subnetwork. During testing, the scalar outputs calculated from the bifurcated subnetwork branches are averaged to generate the final contour prediction.
The following figure presents part of the experimental results: from left to right, the input image, the candidate point set produced by the Canny edge detector, non-threshold predictions, threshold predictions, and the ground truth image. To generate threshold predictions, the probability threshold is set to 0.5.
2. CEDN
Another method utilizes a fully convolutional encoder-decoder network (CEDN), trained end-to-end on data from inaccurate polygon annotations corrected in PASCAL VOC, and its network architecture is shown in the figure.
Part of the experimental results are shown in the figure: from left to right are (a) input image, (b) GT contour, (c) contour detection with pre-trained CEDN, and (d) contour detection with fine-tuned CEDN.
Download 1: OpenCV-Contrib Extension Module Chinese Version Tutorial
Reply "Extension Module Chinese Tutorial" in the background of the "Visual Learning for Beginners" public account to download the first Chinese version of the OpenCV extension module tutorial available online, covering over twenty chapters including extension module installation, SFM algorithms, stereo vision, object tracking, biological vision, super-resolution processing, etc.
Download 2: Python Visual Practical Projects 52 Lectures
Reply "Python Visual Practical Projects" in the background of the "Visual Learning for Beginners" public account to download 31 visual practical projects including image segmentation, mask detection, lane line detection, vehicle counting, eyeliner addition, license plate recognition, character recognition, emotion detection, text content extraction, facial recognition, etc., to help quickly learn computer vision.
Download 3: OpenCV Practical Projects 20 Lectures
Reply "OpenCV Practical Projects 20 Lectures" in the background of the "Visual Learning for Beginners" public account to download 20 practical projects based on OpenCV to achieve advanced learning of OpenCV.
Group Chat
Welcome to join the reader group of the public account to communicate with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions, etc. (which will be gradually subdivided). Please scan the WeChat number below to join the group, and note: "Nickname + School/Company + Research Direction", for example: "Zhang San + Shanghai Jiao Tong University + Visual SLAM". Please follow the format for notes, otherwise, you will not be approved. After successful addition, you will be invited to the relevant WeChat groups based on research direction. Please do not send advertisements in the group, or you will be removed from the group. Thank you for your understanding~