Click the above“Beginner’s Visual Learning”, select to addstar or “pin”
Important content delivered promptly
Image source: Internet
Author: Huang Yu, Chief Scientist at Singularity Auto
Editor: Hoh Xil
Source:
https://zhuanlan.zhihu.com/p/78051407
Introduction: The extraction of edges and contours is a very tricky task, as details may be obscured by overly strong image lines. Texture itself is a weak edge distribution pattern, and hierarchical representation is a commonly used method, often referred to as scale space. Previously, when working on mobile visual platforms, it was sometimes necessary to disable certain image processing functions due to feature distortion. Now, the CNN model, with its inherent feature description mechanism, provides a good tool for image preprocessing, combining image processing and visual preprocessing into one.
—— Edge Extraction ——
1. HED
Holistically-Nested Edge Detection (HED) is a deep learning edge extraction algorithm with two features: (1) holistic image training and prediction; (2) multi-scale, multi-layer feature learning. This deep model utilizes a fully convolutional network to automatically learn rich hierarchical representations (guided by deep supervision based on side responses).
Multi-scale deep learning can be divided into four categories: multi-stream learning, skip-net learning, multi-input single model, and independent network training, as shown in the figure: (a) multi-stream architecture; (b) skip-net architecture; (c) single model with multi-scale input; (d) independent training of different networks; (e) HED, which adds multiple side outputs.
In terms of representation and computational complexity, methods (a) to (d) have redundancies. (e) HED is a relatively simple variant capable of producing predictions from multiple scales. This structure can be interpreted as a “holistically nested” version of the “independent network” method (d), which includes a single-stream deep network with multiple side outputs. Hidden layer supervision can improve the optimization and generalization of image classification tasks. If unified output is needed, multiple side outputs can flexibly add additional fusion layers.
The figure shows the network architecture of HED, where side output layers are inserted after the convolutional layers, applying deep supervision to guide the side outputs to become edge predictions. HED outputs are multi-scale and multi-layer, with side output areas becoming smaller and receptive fields enlarging. A weighted fusion layer automatically learns how to combine outputs from multiple scales. The entire network is trained through multiple error propagation paths (dashed lines).
The following figure provides some experimental results of HED. (a) Test image; (b) manually annotated edges; (c) HED results. (d), (e), and (f) are the side responses of layers 2, 3, and 4 of the convolutional neural network. (g), (h), and (i) show the edge responses of the Canny edge detector at scales σ= 2.0, σ= 4.0, and σ= 8.0. HED shows significant advantages over Canny in terms of consistency.
2. CASENet
CASENet associates each edge pixel with more than one edge class, and is an end-to-end deep semantic edge learning architecture based on ResNet, which is also a skip-layer architecture. In this architecture, the category edge activations of the top convolution layer are shared and fused with the same set of bottom features, where a multi-label loss function supervises the fusion of activations.
Comparisons of three CNN architectures are shown in figures (a) to (c). The solid rectangles in the figure represent the combined blocks of CNN layers. The decreasing width indicates a reduction in spatial resolution of the output feature maps by a factor of 2. The numbers next to the arrows indicate the number of output feature channels. The blue solid rectangles represent a stack of ResNet blocks. The purple solid rectangles are the classification modules. The red dashed boxes indicate outputs monitored by the loss function. The gray solid rectangles are the side feature extraction modules. The dark green solid rectangles are the fusion classification modules, executing K groups of 1×1 convolutions. Figures (d) to (h) detail the modules in figures (a) to (c), where rectangles represent the input and output feature maps.
Figure (a) is the base network, adopting a fully convolutional network framework. In ResNet-101, the average pooling and fully connected layers are removed, retaining the bottom convolution blocks; the strides of the first and fifth convolution blocks in ResNet-101 (“res1” and “res5”) are changed from 2 to 1; dilation factors are introduced into subsequent convolution layers to maintain the same receptive fields as the original ResNet. On top of the base network, a classification module (figure (d)) is added as a 1×1 convolution layer, followed by bilinear upsampling (implemented by K grouped deconvolution layers) to produce a set of K activation maps { A1,···,AK } where each activation map has the same size as the image; then, the sigmoid unit defined as Yk(p)=σ(Ak(p)) models the probability of the pixel belonging to the k-th class edge.
Unlike the basic idea of the previous HED network, which considers the bottom layer convolution loss in addition to the top network loss, the fusion edge map is obtained through the linear combination of supervised side activations. However, HED only performs binary edge detection. The Deep Supervised Network (DSN) extends this architecture, handling K channels of side outputs and K channels of final outputs, as shown in figure (b). In this network, the aforementioned classification module is connected to the output of each residual block, producing five side classification activation maps { A(1), … , A(5) }, where each activation map has K channels. Then, these five activation maps are fused through a sliced concatenation layer (the color in the figure indicates the channel index in figure (g)) to produce a 5K channel activation map:
The fused activation of pixel p is given by the following formula due to the supervision of five side activations, which implicitly constrains each channel of those side activations to carry only the most relevant information for that class. Due to the sliced concatenation and grouped convolutions, the fused activation of pixel p is given by:
CASENet adopts a nested architecture with the following improvements:
① Replace the bottom classification module with a feature extraction module.
② Place the classification module only at the top of the network for supervision.
③ Use shared concatenation (figure (h)) instead of sliced concatenation.
The distinction between side feature extraction and side classification is that the former only outputs single-channel feature maps F(j) instead of K class activations. Shared concatenation copies bottom features F = {F(1), F(2), F(3)} from Side-1-3, respectively connecting with each of the K top activations:
The connected activation maps are then fed into a fusion classification layer with K-group convolutions, producing K-channel activation map A(6). Finally, six loss functions are calculated for the activation maps {A(1), … , A(6)}, providing deep supervision for the network.
By supervising five side activations, it implicitly constrains each channel of those side activations to carry only the most relevant information for that class. Due to sliced concatenation and grouped convolutions, the fused activation of pixel p is given by:
CASENet is generally considered a network for joint edge detection and classification, allowing lower-level features to participate and enhancing high-level semantic classification through a skip-layer structure.
The following figure compares the results of CASENet and DSN, from left to right: input, ground truth, DSN, and CASENet. CASENet shows better detection quality on challenging targets, while DSN has more false positives on non-edge pixels.
—— Contour Extraction ——
1. DeepEdge
Previously, low-level features such as texture or saliency were mostly used to detect contours, while DeepEdge utilizes object-related features as high-level cues for contour detection. It is designed as a multi-level deep network consisting of five convolution layers and a bifurcated fully-connected subnetwork. From the input layer to the fifth convolution layer is a pre-trained network directly used for image input at four different scales. These four parallel and identical data flows connect to a bifurcated subnetwork composed of two independently trained branches, one learning to predict contour likelihood (aimed at classification) and the other learning to detect the presence of contours at given points (based on regression measures). The figure is the DeepEdge architecture diagram, where the Canny edge detector extracts candidate contour points, and then, around each candidate point, four different scale patches are extracted while passing through the pre-trained KNet’s five convolution layers.
A single-scale schematic diagram explaining the model architecture is given in the figure below: first, input patches centered around candidate points are processed through the five convolution layers of KNet; to extract high-level features, feature maps around the center points are extracted from small sub-volumes at each convolution layer, and max, average, and center pooling are performed on the sub-volumes. The pooled values are fed into the bifurcated subnetwork. During testing, the scalar outputs computed from the branches of the bifurcated subnetwork are averaged to generate the final contour prediction.
The figure presents some experimental results: from left to right are the input image, the candidate point set generated by the Canny edge detector, non-threshold predictions, threshold predictions, and the ground truth image. To generate threshold predictions, the probability threshold is set to 0.5.
2. CEDN
Another method utilizes a fully convolutional encoder-decoder network (CEDN), with end-to-end training data sourced from inaccurately corrected polygon annotations in PASCAL VOC, and its network architecture is shown in the figure.
Some experimental results are shown in the figure: from left to right are (a) input image, (b) GT contour, (c) contour detection with pre-trained CEDN, and (d) contour detection with fine-tuned CEDN.
—— References ——
1. G Bertasius, J Shi, L Torresani, “DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection”, CVPR, 2015
2. S Xie, Z Tu, “Holistically-Nested Edge Detection”, ICCV 2015
3. W Shen et al., “DeepContour: A Deep Convolutional Feature Learned by Positive-sharing Loss for Contour Detection”, CVPR 2015
4. Z Yu et al., “CASENet: Deep Category-Aware Semantic Edge Detection”, CVPR 2017
Good News!
Beginner’s Visual Learning Knowledge Planet
Is now open to the public👇👇👇
Download 1: OpenCV-Contrib Extension Module Chinese Version Tutorial
Reply: Extension Module Chinese Tutorial in the backend of the “Beginner's Visual Learning” public account to download the first OpenCV extension module tutorial in Chinese on the internet, covering installation of extension modules, SFM algorithms, stereo vision, object tracking, biological vision, super-resolution processing, etc., with more than twenty chapters.
Download 2: Python Visual Practical Project 52 Lectures
Reply: Python Visual Practical Project in the backend of the “Beginner's Visual Learning” public account to download 31 visual practical projects including image segmentation, mask detection, lane line detection, vehicle counting, eyeliner addition, license plate recognition, character recognition, emotion detection, text content extraction, facial recognition, etc., to help quickly learn computer vision.
Download 3: OpenCV Practical Project 20 Lectures
Reply: OpenCV Practical Project 20 Lectures in the backend of the “Beginner's Visual Learning” public account to download 20 practical projects based on OpenCV, achieving advanced OpenCV learning.
Group Chat
Welcome to join the reader group of the public account to exchange with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions, etc. (which will be gradually subdivided). Please scan the WeChat number below to join the group, and note: “Nickname + School/Company + Research Direction”, for example: “Zhang San + Shanghai Jiao Tong University + Visual SLAM”. Please follow the format, otherwise, it will not be approved. After successful addition, you will be invited into the relevant WeChat group based on your research direction. Please do not send advertisements in the group, or you will be removed from the group, thank you for your understanding~