Design and Implementation Challenges of Computer Vision in Automated Parking Systems

3.1 Overview of Automated Parking

Automated parking systems have been on the mass market for a while, starting with parallel parking and then evolving in recent years to include vertical parking. Parking systems have surpassed merely controlling steering in driver assistance systems, achieving partial automation of both lateral and longitudinal control. The challenge of parking assistance systems is to reliably and accurately detect parking spaces to allow parking maneuvers with minimal independent movement. The goal of automated parking systems is to provide drivers with robust, safe, comfortable, and most importantly, useful functions that save time, ensure accurate and collision-free parking. Currently, systems on the market rely solely on distance sensor data, usually ultrasonic, for empty space detection, re-measurement, and collision avoidance during automated parking. While such systems have proven to be very successful in the field and are being released in the low-end market, they have some inherent limitations that cannot be resolved without the help of other sensor technologies. The use cases described below focus on the benefits of camera-based solutions fused with ultrasonic sensors, particularly trying to address some limitations of current systems and push automated parking technology to the next level.

Before automated parking can begin, the parking system must first search for, identify, and accurately locate valid parking spaces around the vehicle. Current systems typically rely on the driver to initiate search mode, and parking spaces can take different forms as described in the following use cases. Once parking spaces are located, they are presented to the driver to allow for the selection of the desired parking space and the orientation of the car facing the final parking position. After the driver selects a parking space, the vehicle automatically traverses the computed trajectory to reach the desired endpoint position, usually at a limited speed below 10 km/h. To keep this functionality at automation level 2 and avoid the legal implications of conditional automation, in which the system monitors the driving environment and requires the driver to show their attention through an unattended switch in the vehicle.

Partial automation systems may also allow the driver to exit the vehicle after identifying a parking space and remotely start the parking operation via a remote key or smartphone. In this case, the driver must always be responsible for monitoring the environment around the vehicle and controlling the parking operation via switches on the remote key or smartphone. Remote parking is suitable for scenarios where parking spaces have been located and measured or in controlled environments (like garage parking), where the vehicle can safely explore the forward environment within limited distances and steering angles. During parking maneuvers, the system continues to re-measure the expected parking space and the position of the vehicle itself. Due to inaccuracies in slot measurements and self-odometry measurement errors, continuous re-measurement of positioning during the parking process is necessary to improve the accuracy of the endpoint position and avoid collisions with static or dynamic obstacles (such as pedestrians). The way the parking trajectory is calculated should ensure that the selected trajectory is the most suitable for the parking situation, i.e., the trajectory selection is completed in the middle of the parking space from the current position without any collisions and limited maneuvering/direction changes (i.e., driving in reverse, and vice versa). The enhancement of automated parking system functionalities is not only to park the car in the parking space but also to retrieve the car from the parking space.

3.2 Benefits of Camera-Based Parking

Current systems rely on distance sensor information, typically ultrasonic sensors, to identify and locate parking spaces. However, there are many inherent problems with distance sensors for automated parking that can be partially or fully resolved by using camera data. In this case, the best camera data comes from four surround-view fisheye cameras located at the front and rear and in both side mirrors to assist with parking space search, parking automation, and visualization of all parking use cases.

After distance sensors have already located parking spaces, a single rearview fisheye camera is also beneficial in a limited number of reverse parking use cases. The narrower field of view front camera is of little benefit for parking space search but can help automate forward parking scenarios, similar to the rearview camera.

The biggest limitation of distance sensors used for parking space detection is that they require the presence of other obstacles in the scene to identify the boundaries of parking spaces. Cameras can be used to detect parking spaces using road markings while leveraging the type of line endings to understand the intended use of the parking space. Using LiDAR technology can detect parking space markings; however, the cost of the sensors and limited field of view are the biggest drawbacks.

Figure 6: Accurate parking scene based on parking space rather than other vehicles.

Figure 6 illustrates that using a camera fusion system allows for more accurate vehicle parking. A parking system based solely on ultrasonic/radar will attempt to align with other parked vehicles, while a camera/fusion allows parking based on the position relative to the parking space. While the range of the camera detection (∼10 meters) is smaller than that of radar or LiDAR (∼100 meters), the camera does provide a greater range (∼4m) than ultrasonic sensors while also having overlapping fields of view. Ultrasonic can provide precise distance data, while cameras are better suited for providing high angular resolution. These characteristics make the sensing capabilities of cameras and ultrasonics complementary. The large vertical field of view (∼140◦) of fisheye cameras can target obstacles above vehicle height within near distances (<1m), which is beneficial for automated parking situations, such as entering a garage with a roll-up door where the door is not fully opened to allow the vehicle to enter. Most distance sensors have very limited vertical fields of view and thus cannot cover this scenario.

Since cameras have a significant measurement resolution advantage, they can generate point cloud data for certain types of objects that active sensors may not be able to detect, such as poles or chain-link fences. These “blind spots” of ultrasonic and other sensors can greatly impact the robustness and reliability of automated parking functionalities. Surround-view cameras can generate precise ground topology around the vehicle to help locate curbs, parking lots, and parking locks, as well as understand surface changes of free space. Cameras can also be used to provide reliable vehicle odometry information through popular visual simultaneous localization and mapping (SLAM) techniques in robotics. This visual odometry can overcome many accuracy issues inherent in mechanical odometer sensors and provide the resolution needed to minimize parking adjustment times after the initial parking space selection.

3.3 Classification of Parking Scenarios

Automated parking has various applications but can conceptually be divided into four main parking use cases:

1. Vertical parking (forward and backward): As the vehicle passes, the system detects the positions of objects and line markings in the near field and measures the size and orientation of the parking space to understand whether it can offer a parking space to the user, thereby detecting the lateral parking space of the vehicle. If the user chooses to park, the system will find a safe driving trajectory to reach the target parking position while orienting the vehicle according to the boundaries or markings of the parking space created relative to other objects. Figure 8(b) describes an example of backward parking maneuver completed through three steps, while (c) describes a forward parking maneuver. Computer vision methods support obstacle detection through classification and SFM techniques. This data enhances the detection rates and ranges of systems fused with traditional ultrasonic-based systems (Figure 7(a)), thus increasing the true positive and false positive rates of parking spaces provided by users while also improving parking space orientation and measurement, thereby reducing parking adjustments. Computer vision can also park based on parking space markings, providing more accurate parking results, which is not feasible in traditional ultrasonic-based parking systems.

Figure 7: Highlights the benefits of fusing computer vision with traditional ultrasonic-based parking systems. (a) Improved detection performance and range, (b) detecting non-parking environments with ultrasonic (only lane markings).

2. Parallel parking: Parallel parking (Figure 8(a)) is a well-defined parking situation, similar to vertical parking. However, the strategies and situations differ significantly. Typically, entering a parking space requires only one adjustment, while further adjustments can align more accurately with the parking space. Additionally, parking tolerances are usually low, as parking close to surrounding vehicles and curbs within the parking space is desired. Fusion with camera systems can reduce parking operation tolerances and achieve more reliable curb detection (ultrasonic and radar can detect curbs but are often unreliable).

Figure 8: Classification of parking scenarios – (a) Parallel reverse parking, (b) Vertical reverse parking, (c) Vertical forward parking, (d) Ambiguous parking, and (e) Fishbone parking with road markings.

3. Fishbone parking: Figure 8(e) shows an example of fishbone parking, where current ultrasonic-based parking systems are limited because the detection density is too low to identify the orientation of the parking space. In this case, using camera systems can increase the range of observation within the parking space to determine the orientation of the parking space from objects or line markings, which current ultrasonic-based systems cannot cover.

4. Ambiguous parking: The last broad category of use cases is ambiguous parking situations, where parking spaces are not well-defined unless there are other vehicles and objects (Figure 8(d)). By increasing the detection range and providing more complete sensor coverage around the vehicle (ultrasonic usually does not cover the sides of the vehicle), using cameras allows for better planning of parking maneuvers, thus enabling more appropriate adjustments in some ambiguous use cases. Additionally, using camera systems in parking lots can enable or improve the reliability of other functionalities compared to ultrasonic/radar parking systems, such as:

1. Emergency braking/comfort braking: Certainly, in any level of automated driving conditions, vehicles need to respond to the presence of vulnerable road users. Sometimes, the environment may change rapidly (for example, pedestrians quickly entering the automated parking area), so the vehicle must respond quickly and safely. By complementing existing parking systems, low-speed automatic emergency braking or comfort braking becomes more robust due to the additional redundancy provided by camera fusion.

2. Overlaying object distance information: A very common use of combining visual system data with traditional parking systems is to overlay object distance information in the video output stream, such as in surround view systems. This helps the driver accurately estimate the 360° video output stream when manually adjusting the vehicle for more precise navigation within the parking space. This is especially useful for parallel parking spaces with curbs, as the curb is not visible to the driver.

Vision-based ADAS applications began mass production in the early 2000s with systems like Lane Departure Warning (LDW). Since then, the field of vision-based ADAS has rapidly evolved. This is due to significant improvements in processing and imaging hardware, as well as the automotive industry’s drive to add more ADAS functions to enhance safety and improve brand visibility in the market. As cameras are rapidly accepted as standard equipment to improve driver visibility (surround view systems), it is logical that these sensors are used in parallel for ADAS applications.

In this section, we will discuss the use of four important ADAS functions and their relevance in automated parking systems. The focus is on algorithms feasible on current ADAS systems, where we need to consider detection, localization, and in some cases, classification of 1) non-moving obstacles, such as parked vehicles, 2) parking lines and other ground markings, 3) pedestrians and general moving obstacles, and 4) free space to support the removal of tracked obstacles from parking maps. The algorithms discussed in the following sections are based on feasibility deployed on embedded systems two years ago.

4.1. 3D Point Cloud

Depth estimation refers to a set of algorithms aimed at obtaining a representation of the spatial structure of the environment within the sensor’s field of view. In the context of automated parking, it is a primary scheme used by computer vision to build maps, which is crucial for all parking use cases: it enables better estimation of the depth of parking spaces on existing ultrasonic parking systems, thus better planning trajectories for vertical and fishbone parking; it improves curb detection reliability, enhancing maneuverability during parallel parking; additionally, it provides detection results for obstacles, significantly reducing false positives in automatic emergency braking. Depth estimation is a major focus of many active sensor systems, such as TOF (Time-of-Flight) cameras, LiDAR, and radar, which remains a complex topic for passive sensors like cameras. There are two main types of depth perception techniques for cameras: stereo cameras and monocular systems. The primary advantage of stereo cameras over monocular systems is improved depth perception capabilities. It works by resolving the correspondence problem for each pixel, allowing pixel positions from the left camera image to be mapped to the right camera image. The map showing these distances between pixels is called a disparity map, which is proportional to the physical distance of the corresponding world points from the camera. Using known camera calibration and baseline, rays forming pixel pairs between the two cameras can be projected and triangulated to solve for the 3D position of each pixel in the world coordinate system. Figure 9 shows an example of sparse 3D reconstruction.

Monocular systems can also perceive depth but require camera motion to create a baseline for scene reconstruction; this scene reconstruction method is called Structure from Motion (SFM), using sparse or dense techniques to track or match pixels in images from one frame to the next. The camera processes known motion between frames and uses camera calibration to project and triangulate the corresponding world positions of points. Bundle adjustment is a commonly used method to refine the estimated 3D positions in the scene and the relative motion of the camera based on optimal criteria, involving projections of all points in the images.

4.2. Parking Space Marking and Recognition

Of course, the detection of parking spaces is critical for any automated parking system; the system must know where it will park before completing the maneuver. To detect parking spaces without obstacles limiting the parking location, detecting road markings that define parking locations is crucial. Consider: how can an automated parking system select a valid parking space in an empty parking lot? This applies to all parking maneuvers (vertical, parallel, and fishbone), where operations need to be performed based on marked parking spaces. Technologies like LiDAR can be used for parking space marking recognition; LiDAR has spectral responses to road markings. However, LiDAR systems are often expensive, have limited detection areas, and do not have the wide field of view that cameras do (>140◦FOV). In vision, lane marking detection can be accomplished using image top-down correction, edge extraction, and Hough space analysis to detect markings and marking pairs.

Figure 10 shows an example result of a similar method

using a 190◦ horizontal field of view parking camera. The same authors also proposed a different method based on manually determined seed point inputs, followed by applying structural analysis techniques to extract parking spaces. Alternatively, a method based on HOG (Histogram of Oriented Gradients) and LBP (Local Binary Patterns) features with a pre-trained model was proposed in paper [18], applying linear SVM (Support Vector Machine) to build classification models. Regardless of the specific approach taken, it is evident that detecting marked parking spaces is critical for a complete automated parking system, and the only reasonable and effective technology to achieve this is cameras with a wide field of view.

4.3. Vehicle and Pedestrian Detection/Tracking

Vehicle detection and tracking are typically performed in the context of front camera detection for automatic emergency braking or traffic monitoring applications. However, the parking process usually occurs in the presence of other vehicles (whether parked or moving), so the detection and tracking of vehicles are very important for automating such maneuvers. More importantly, the system must reliably detect and classify pedestrians so that the vehicle can take appropriate actions, such as automatically emergency braking in the presence of potential risk from pedestrians (Figure 11).

Figure 11: Classification and tracking of pedestrians using cameras

Typically, both vehicle detection and pedestrian detection problems are solved using some type of classification. Compared to visual systems, no other sensors can classify detections based on object type as easily and reliably. Object classification typically falls into the supervised classification group of machine learning algorithms. This is based on human knowledge of selecting sample thumbnails across multiple images representing specific object categories. Using feature extraction methods such as HOG, LBP, and wavelets applied to human classification sample images, predictive models are built using machine learning to classify objects. Many vision-based ADAS features use machine learning methods for classification. As mentioned earlier, classification is widely used for detecting pedestrians and vehicles but also for face detection and traffic sign recognition (TSR). The quality of the final algorithm heavily depends on the amount and quality of sample data used to learn the classifier, as well as the overall quality of the classification techniques and the appropriateness of feature selection methods for the target application. Typical classifiers include SVM, random forests, and convolutional neural networks (CNN). Recently, this trend has changed with deep learning methods that automatically learn features.

4.4. Drivable Space

Drivable space is a feature used by most environmental sensing maps. It is the area around the vehicle that is not occupied by objects within the sensor’s field of view, typically classified as an occupancy grid map problem, usually detected by segmenting the ground from other objects. In occupancy grid map methods, free space information is integrated and stored over time. In the case of vector-based map representations, the presence probability of each object is updated based on free space measurements. Drivable space is used to clear dynamic and static obstacles that have not been actively measured or updated in environmental sensing maps. This means that a good free space model can quickly eliminate dynamic obstacles from previous positions without removing valid static information. Free space should also clear previously detected static objects that have moved since the last valid measurement, as well as detections that exist in the map due to odometry update errors and false positives.

Figure 12: (a) Shows the segmentation result using image-based drivable area, (b) Shows the radial unit drivable space definition based on (a) road segmentation

Figure 12 shows an example of image segmentation based on camera free space; additionally, free space supports collision-free trajectory search and planning, especially in cumulative free space grid maps. Unlike other sensor types, visual systems can provide different, independent estimates of vehicle free space. For example, another way to determine camera-based free space is to utilize 3D point clouds and their corresponding obstacle information. However, it also reconstructs features of the road surface around the vehicle; the reconstructed features related to the ground can provide valuable free space information. If there are reconstructed features related to the ground, a reasonable assumption is that the area between that point and the sensor (camera) is not obscured by objects and can be used to define the free space area around the main vehicle. Since these methods are independent and complementary, fusing these techniques (with themselves as well as free space provided by other sensors such as ultrasonic) may also be beneficial to improve the accuracy and robustness of free space measurements.

4.5. Other Visual Functions

Computer vision technologies can also support several other areas in automated parking scenarios. Visual odometry is a task closely related to depth estimation, through visual SLAM/bundle adjustment techniques, although there are other visual odometry methods. While vehicle odometry can be used on vehicle networks (CAN/FlexRay), relying solely on these signals is inaccurate due to network latency, signal constraints, and inaccuracies (e.g., relying on accelerometers) and limited degrees of freedom (usually limited to speed and heading). In automated parking, the quality of odometry is crucial for user comfort and parking accuracy – with improved odometry, parking can be completed with fewer adjustments, and the final position is closer to the target position. Intersection traffic alert algorithms aim to detect critical traffic situations at intersections that may pose a threat to the main vehicle, such as T-intersections (especially in limited visibility).

Figure 13: Example of traffic conditions at parking space exits, (b) Shows a screenshot of the intersection traffic detection algorithm

The second image in Figure 13 shows an example of an intersection vehicle detection algorithm based on optical flow and main odometry compensation. In addition to the parking space marking detection discussed earlier, it is also necessary to be able to detect other road markings, such as arrows and no-parking signs, which allow automated parking vehicles to comply with the rules stipulated in the parking area.

Considerations for automated parking systems are illustrated in Figure 14, where many factors influence the specifications of vision-based or partially vision (fusion) automated parking systems. Since most choices impact the entire system, very few parts of the system can be considered in isolation. A simple example is the choice of camera pixel resolution, which can affect the potential use cases implemented through hardware and software; camera resolution affects ISP selection, SerDes selection, memory bandwidth requirements, memory requirements, computational requirements for computer vision, accuracy and range performance of the system, low-light performance, and display requirements. Therefore, it is necessary to understand and define some limitations in the aspects of hardware, use cases, and computer vision algorithms.

Figure 14: Application stack for fully automated parking systems using cameras.

From a hardware perspective, the main variables are the selection of camera imagers and ECU processing SOCs, considering thermal, power, and cost within tolerances. Typical automotive imagers for surround view applications are transitioning from 1MP resolution to 2-4MP resolution. The challenge of increasing resolution is to do so while maintaining or preferably improving low-light sensitivity. This is crucial for the usability and availability of camera-based automated parking systems. Additional pixel resolution improves the accuracy and range of the system, allowing for more parking use cases, more robust parking performance, and higher speed availability. Once image formation occurs, it is critical to process as many computer vision functions in parallel at the highest possible frame rate and resolution. This is where SOC selection becomes crucial. Reducing system load always requires trade-offs, including deploying smart state machines to ensure only critical computer vision algorithms run, downscaling and skipping processed images to reduce load. As pixel-level processing, traditionally performed on hardware vector processing engines or DSPs (which typically account for 60-70% of any computer vision algorithm load), is replaced by specific computer vision hardware accelerators, these trade-offs become less stringent. These computer vision accelerators, used for processing tasks such as dense optical flow, stereo disparity, and convolution, can achieve higher pixel processing throughput at lower power consumption at the cost of flexibility. The use cases that the system needs to cover also play an important role in system specifications. The automated parking use cases to be covered define requirements in terms of detection capabilities, accuracy, coverage, operational range, operational speed, and system availability. This will influence the choice of sensors and SOCs, but most importantly, it defines the performance required from computer vision algorithms to enable functionalities. For instance, line-interleaved automatic vertical parking requires many computer vision algorithms to work in parallel with the required accuracy and robustness to achieve reliable and useful functionalities. First, line marking detection algorithms need to operate at speeds and detection ranges practical for automatic parking space searches. At the same time, an algorithm like motion structure is needed to ensure that no objects (parking locks, cones, trash cans, etc.) are in the slot while measuring the end position of the slot, which could be in the form of a curb. Pedestrian detection is also a good complement to reduce but not eliminate the user’s supervisory burden during parking maneuvers. These computer vision functionalities require support from online calibration algorithms and contamination detection capabilities to operate and understand when they are unavailable, thereby informing the system and the user. Camera information is often fused over time with other distance sensor information (e.g., ultrasonic or radar data) to improve system robustness, accuracy, and availability. However, some detections can only be achieved with cameras, such as classification. As surround-view cameras become a standard sensor, the more functionalities that cameras can achieve, the lower the system cost will be, thus requiring fewer supporting sensors.

Figure 15: Example use case of Park4U family, where the vision-based system uses landmarks to position the car to stored trajectories for autonomous navigation to the family parking space.

Computer vision has recently made significant advances in deep learning, particularly convolutional neural networks (CNN), which have greatly improved the accuracy of object detection, enhancing the perception capabilities of automated driving. It has also achieved dense pixel classification through semantic segmentation, which was previously not feasible. Furthermore, CNN has demonstrated strong trends in achieving recent breakthroughs in geometric visual algorithms, such as optical flow, structure from motion, and relocalization. Advances in CNN have also led hardware manufacturers to incorporate customized hardware IPs to provide high throughput of over 10 Tera operations (TOPS) per second. In addition, next-generation hardware will feature dense optical flow and stereo vision hardware accelerators to achieve general detection of moving and stationary objects. From the perspective of use cases, the next step for parking systems is to make them truly autonomous, allowing drivers to leave the car and autonomously locate and park in unmapped environments. In addition, vehicles should be able to leave the parking space and safely return to the driver. Cameras can play a very important role in future automated parking systems by providing important information about the vehicle’s surrounding environment. This includes information on objects and free space, parking space marking detection, pedestrian detection, and more, to be fused with other sensor technologies.

As described in this article, current automated parking systems control the vehicle after the user identifies and selects a parking space. During the slot search, the system’s state is essentially passive. Future trends and challenges are the automation of the slot search itself to achieve complete vehicle parking automation, including searching, selecting, and parking, all done in a robust, repeatable, and safe manner. These automated parking scenarios can be divided into the following categories:

1) Automated parking in known areas and 2) Automated parking in unknown areas.

Automated parking in known areas generally involves the driver “training” the automated parking system and parking trajectory (see Figure 15). During this training, the sensors locate landmarks in the scene and record the expected trajectory driven by the driver relative to these landmarks. When the automated parking system returns, it can recognize the scene and use the training information to automatically position the vehicle to allow for automated parking along the stored trajectory. Valeo launched its automated parking 4U at the Frankfurt Motor Show for unknown area automated parking. The challenge of achieving a leap to new levels of automation is to extend vision-based automated parking systems for self-localization (SLAM) and allow accurate identification of stored home areas. To achieve the highest level of automation in automated parking systems, a combination of sensor technologies (cameras, ultrasonics, radar, or LiDAR) is clearly needed to achieve maximum accuracy and reliability in self-localization, detection, and environmental prediction.

Automated driving is a rapidly developing technology field, and many high-end vehicles have begun to be equipped with automated parking functionalities. This has led to significant improvements in sensors and computational capabilities, enabling the production of more robust and accurate systems. Despite the challenges of liability, progressive legislation is being introduced by government regulatory agencies such as the European New Car Assessment Programme (EuroNCAP) and the National Highway Traffic Safety Administration (NHTSA) to enforce safety systems and begin legislation allowing automated vehicles to operate on public road networks. Camera sensors will continue to play a vital role due to their low cost and the rich semantics they capture compared to other sensors. In this article, we focused on the benefits of camera sensors and how they enable parking use cases. We discussed the system implementation of automated parking systems, which features four fisheye cameras surrounding the vehicle. We detailed the various aspects of the system, including embedded system components, the parking use cases that need to be addressed, and the visual algorithms required to solve these use cases. As a focus on computer vision, we omitted the details of sensor fusion, trajectory control, and motion planning.

For more detailed content, please join the knowledge circle to view the original text.