DIY License Plate Recognition System with Raspberry Pi

(This article is reprinted from: Machine Heart)

With some free time on our hands, we equipped our beloved car with a Raspberry Pi, paired it with a camera, designed a client, and successfully created a real-time license plate detection and recognition system.

How can we create an intelligent car system without changing the car? For some time, the author Robert Lucian Chiriac has been thinking about how to give a car the ability to detect and recognize objects. This idea is intriguing because we have already witnessed the capabilities of Tesla, and while we cannot buy a Tesla right away (it must be mentioned that the Model 3 is looking increasingly attractive), he had an idea to work hard to achieve this dream.

So, the author did it with a Raspberry Pi, which can detect license plates in real-time when placed in the car.

In the following content, we will introduce each step of the project and provide the GitHub project address, where the project address is just the client tool, and other datasets and pre-trained models can be found at the end of the original blog.

Project Address:

https://github.com/RobertLucian/cortex-license-plate-reader-client

Next, let’s see how the author Robert Lucian Chiriac built a useful onboard detection and recognition system step by step.

Here is a finished product image to begin with.

Step 1:Define the Scope of the Project

Before starting, the first question that comes to my mind is what this system should be able to do.If there is one thing I have learned in my life, it is that taking things step by step is always the best strategy.

So, besides basic visual tasks, what I need is to clearly identify license plates while driving.This recognition process involves two steps:

1. Detect the license plate.

2. Recognize the text within each license plate bounding box.

I think if I can accomplish these tasks, it will be much easier to do other similar tasks (such as determining collision risks, distances, etc.).I might even be able to create a vector space to represent the surrounding environment — sounds cool just thinking about it.

Before confirming these details, I knew I had to achieve:

• A machine learning model that detects license plates from unlabeled images;

• Some hardware.Simply put, I need a computer system connected to one or more cameras to run my model.

Let’s start with the first thing — building an object detection model.

Step 2:Select the Right Model

After careful research, I decided to use the following machine learning models:

1. YOLOv3 – This is one of the fastest models available, and its mAP is comparable to other SOTA models.We use this model to detect objects;

2. CRAFT Text Detector – We use it to detect text in images;

3. CRNN – Simply put, it is a recurrent convolutional neural network model.It must be sequential data to arrange detected characters into words in the correct order;

How do these three models work together?The following describes the operational flow:

1. First, the YOLOv3 model receives frames from the camera and finds the bounding boxes for the license plates in each frame.It is not recommended to use very precise predicted bounding boxes — it is better for the bounding box to be slightly larger than the actual detected object.If it is too tight, it may affect the performance of subsequent processes;

2. The text detector receives the cropped license plates from YOLOv3.If the bounding box is too small, it is likely that part of the license plate text is cut off, resulting in a poor prediction.However, when the bounding box is enlarged, we can allow the CRAFT model to detect the positions of the letters, making the position of each letter very accurate;

3. Finally, we can pass the bounding boxes of each word from CRAFT to the CRNN model to predict the actual words.

With my basic model architecture sketch, I can start working on the hardware.

Step 3:Design the Hardware

When I realized I needed a low-power hardware solution, I thought of my old love: Raspberry Pi.It has a dedicated camera, the Pi Camera, and sufficient computing power to preprocess each frame at a decent frame rate.The Pi Camera is the physical camera for the Raspberry Pi and has a mature and complete library.

To connect to the internet, I could use the 4G access of the EC25-E, which I also used in a previous project with a GPS module, details can be found:

Blog Address:

https://www.robertlucian.com/2018/08/29/mobile-network-access-rpi/

Then I need to start designing the casing — it should be fine to hang it on the rearview mirror of the car, so I ultimately designed a support structure divided into two parts:

1. On the rearview mirror side, the Raspberry Pi + GPS module + 4G module will be retained.For the GPS and 4G antennas I used, you can check my article on the EC25-E module;

2. On the other side, I used an arm that utilizes a ball joint to support the Pi Camera.

I will use my reliable Prusa i3 MK3S 3D printer to print these parts, and I also provide the 3D printing parameters at the end of the original text.

Figure 1: The appearance of the Raspberry Pi + 4G/GPS casing

Figure 2: Using a ball joint arm to support the Pi Camera

Figures 1 and 2 show what they look like when rendered.Note that the c-shaped bracket here is pluggable, so the Raspberry Pi’s accessories and the support for the Pi Camera are not printed together with the bracket.They share a socket, with the bracket plugged into it.If any reader wants to replicate this project, this is very useful.

They only need to adjust the bracket on the rearview mirror.Currently, this base works very well in my car (Land Rover Freelander).

Figure 3: Side view of the Pi Camera support structure

Figure 4: Front view of the Pi Camera support structure and RPi base

Figure 5: Expected camera field of view

Figure 6: Close-up of the embedded system with built-in 4G/GPS module and Pi Camera

Clearly, these things take some time to model, and I need to do several iterations to get a sturdy structure.The PETG material I used has a layer height of 200 microns.PETG works well at 80-90 degrees Celsius and has strong UV radiation resistance — while not as good as ASA, it is still strong.

This was designed in SolidWorks, so all my SLDPRT/SLDASM files, along with all STLs and gcode, can be found at the end of the original text.You can also use these files to print your own version.

Step 4:Train the Model

Now that the hardware is sorted, it’s time to start training the model.As everyone knows, it’s best to work by standing on the shoulders of giants.This is the essence of transfer learning — learning from a very large dataset initially and then leveraging the knowledge gained.

YOLOv3

I found many pre-trained license plate models online, but not as many as I initially expected, although I did find one trained on 3600 license plate images.This training set is not large, but it is better than nothing.

Additionally, it was trained based on the pre-trained models from Darknet, so I could use it directly.

Model Address:

https://github.com/ThorPham/License-plate-detection

Since I already had a recordable hardware system, I decided to drive around town for a few hours to collect new video frame data to fine-tune the previous model.

I used VOTT to label the frames containing license plates, ultimately creating a small dataset of 534 images, all of which had labeled bounding boxes for license plates.

Dataset Address:

https://github.com/RobertLucian/license-plate-dataset

Then I found code to implement YOLOv3 in Keras and used it to train my dataset, then submitted my model to this repo so others could use it.Ultimately, I achieved an mAP of 90% on the test set, which is excellent considering my dataset is very small.

• Keras Implementation:

  https://github.com/experiencor/keras-yolo3

• Submit Merge Request:

  https://github.com/experiencor/keras-yolo3/pull/244

CRAFT & CRNN

To find a suitable network for text recognition, I went through countless attempts.Eventually, I stumbled upon keras-ocr, which packages CRAFT and CRNN, is very flexible, and has pre-trained models, which is fantastic.I decided not to fine-tune the models and keep them as they are.

keras-ocr Address:

https://github.com/faustomorales/keras-ocr

Most importantly, using keras-ocr to predict text is very simple.Basically, it only takes a few lines of code.You can check the project homepage to see how it’s done.

Step 5:Deploy My License Plate Detection Model

There are primarily two methods for model deployment:

1. Perform all inference locally;

2. Perform inference in the cloud.

Both methods have their challenges.The first means having a central “brain” computer system, which is complex and expensive.The second faces challenges related to latency and infrastructure, especially when using GPUs for inference.

In my research, I stumbled upon an open-source project called Cortex.It is a newcomer in the AI field, but as the next evolution of AI development tools, it undoubtedly makes sense.

Cortex Project Address:

https://github.com/cortexlabs/cortex

Essentially, Cortex is a platform for deploying machine learning models as production web services.This means I can focus on my application while leaving the rest for Cortex to handle.

It does all the preparation work on AWS, and all I need to do is write a predictor using template models.Even better, I only need to write a few dozen lines of code for each model.

Below is the terminal for the Cortex runtime obtained from the GitHub repo.If this doesn’t qualify as elegant and concise, I don’t know what does:

Since this computer vision system is not designed for autonomous driving, latency is not as critical for me, and I can use Cortex to solve this issue.If it were part of an autonomous driving system, using a cloud provider’s service would not be a good idea, at least not now.

Deploying ML models with Cortex requires:

1. Define the cortex.yaml file, which is the configuration file for our API.Each API will handle one type of task.I assigned the task of detecting license plate bounding boxes on the given frame to the yolov3 API, while the crnn API predicts the license plate number with the help of the CRAFT text detector and CRNN;

2. Define the predictor for each API.Basically, what you need to do is define a predict method for a specific class in Cortex to receive a payload (all the servy part is already handled by the platform), which can predict the result and return the prediction.It’s that simple!

Here’s an example of a predictor for the classic iris dataset, but due to the length of the article, I won’t go into specifics here.You can find methods for using these two APIs in the project link — all other resources for this project are at the end of this article.

# predictor.pyimport boto3
import pickle
labels = ["setosa", "versicolor", "virginica"]
class PythonPredictor:
    def __init__(self, config):
        s3 = boto3.client("s3")
        s3.download_file(config["bucket"], config["key"], "model.pkl")
        self.model = pickle.load(open("model.pkl", "rb"))
    def predict(self, payload):
        measurements = [
            payload["sepal_length"],
            payload["sepal_width"],
            payload["petal_length"],
            payload["petal_width"],
        ]
        label_id = self.model.predict([measurements])[0]
        return labels[label_id]

To make predictions, you just need to use curl like this:

curl http://***.amazonaws.com/iris-classifier \
    -X POST -H "Content-Type: application/json" \
    -d '{"sepal_length": 5.2, "sepal_width": 3.6, "petal_length": 1.4, "petal_width": 0.3}'

Then you will receive feedback like setosa, it’s very simple!

Step 6:Develop the Client

With Cortex helping me with deployment, I can start designing the client — this is the more challenging part.

I thought of the following architecture:

1. Collect frames at an acceptable resolution (800×450 or 480×270) at a frame rate of 30 FPS from the Pi Camera, and push each frame into a common queue;

2. In a separate process, I will take frames out of the queue and distribute them to multiple workstations on different threads;

3. Each worker thread (or what I call inference threads) will make API requests to my Cortex API.First, a request to my yolov3 API, and if any license plates are detected, another request will be sent with a batch of cropped license plates to my crnn API.The predicted license plate numbers will be returned in text format;

4. Each detected license plate (with or without recognized text) will be pushed to another queue, which will ultimately broadcast it to the browser page.At the same time, the predicted license plate numbers will be pushed to another queue to be saved to disk in CSV format later;

5. The broadcast queue will receive a set of unordered frames.The consumer’s task is to place them in a very small buffer (the size of a few frames) and broadcast a new frame to the client for reordering.This consumer runs separately in another process, and it must also try to keep the queue size fixed to a specified value to display frames at a consistent frame rate.Clearly, if the queue size decreases, then the frame rate will decrease proportionally, and vice versa;

6. Meanwhile, another thread will run in the main process, retrieving predictions and GPS data from another queue.When the client receives a termination signal, the predictions, GPS data, and time will also be saved to a CSV file.

The diagram below shows the relationship flowchart between the client and the cloud API provided by Cortex.

Figure 7: Flowchart of the cloud API provided by Cortex and the client

In our case, the client is the Raspberry Pi, and the inference requests are sent to the cloud API provided by Cortex on AWS.

The client’s source code can also be found in its GitHub:

https://github.com/robertlucian/cortex-licens-plate-reader-client

One challenge I had to overcome was the bandwidth of 4G.It’s best to reduce the bandwidth required for this application to minimize possible hang-ups or excessive use of available data.

I decided to have the Pi Camera use a very low resolution:480×270 (we can use a small resolution here because the Pi Camera has a very narrow field of view, so we can still easily identify license plates).

However, even at this resolution, the JPEG size for each frame is about 100KB (0.8MBits).Multiplying by 30 frames per second gives 3000KB, which is 24mb/s, and this is still without HTTP overhead, which is a lot.

Therefore, I used some tricks:

• Reduce the width to 416 pixels, which is the size required by the YOLOv3 model, and the scale is clearly intact;

• Convert the image to grayscale;

• Remove the top 45% of the image.The idea here is that the license plate will not appear at the top of the frame because cars do not fly, right?As far as I know, removing 45% of the image does not affect the performance of the predictor;

• Convert the image to JPEG again, but at a much lower quality at this time.

The final frame size is about 7-10KB, which is excellent.This corresponds to 2.8Mb/s.However, considering all overheads like responses, it’s about 3.5Mb/s.

For the crnn API, cropped license plates do not require much space at all, even without compression, they are only about 2-3KB each.

In summary, to run at 30FPS, the bandwidth required for the inference API is about 6Mb/s, which is an acceptable number for me.

Results

Success!

Follow Replay Share Like CloseWatch MoreMoreExit FullscreenSwitch to Portrait FullscreenExit FullscreenJuly Online LabFollowedShare Video, Duration26:38

0/0

00:00/26:38

Switch to Landscape Mode Continue PlayingProgress Bar, 0%Play00:00/26:3826:38Fullscreen Playback speed 0.5x 0.75x 1.0x 1.5x 2.0x HD Smooth

Continue watching

You have a Tesla, I have a Raspberry Pi, DIY license plate recognition detection system, household car transforms into smart car

Watch MoreReprinted, You have a Tesla, I have a Raspberry Pi, DIY license plate recognition detection system, household car transforms into smart carJuly Online LabFollowedShareLikeWowAdded to Top StoriesEnter comment Video Details

The above is an example of real-time inference using Cortex.I need about 20 GPU-equipped instances to run it smoothly.Depending on the latency of this group of GPUs, you may need more GPUs or fewer instances.

The average latency from capturing a frame to broadcasting it to the browser window is about 0.9 seconds, considering that the inference occurs far away, it’s truly amazing — I’m still surprised.

The text recognition part may not be the best, but it at least proves one point — it can be made more precise by increasing the video resolution or by reducing the camera’s field of view or through fine-tuning.

As for the issue of too high GPU demand, this can be solved through optimization.For example, using mixed precision/full half precision (FP16/BFP16) in the model.In general, allowing the model to use mixed precision has a minimal impact on accuracy, so we don’t have to make too many trade-offs.

In summary, if all optimizations are in place, reducing the number of GPUs from 20 to one is actually feasible.If properly optimized, even the resources of one GPU may not be fully utilized.

Original Address:

https://towardsdatascience.com/i-built-a-diy-license-plate-reader-with-a-raspberry-pi-and-machine-learning-7e428d3c7401

Copyright Notice:This article content comes from the internet, copyright belongs to the original author, if there is any infringement, please contact for deletion.

[Today’s Recommendation: CV Employment Small Class Fourth Session]

Top Instructor Team

Six Major Practical Projects

Comprehensive Employment Services

1. Job Interview

Custom project for specific directions, final adjustments and improvements of the project and resume.Then arrange for major companies’ interviewers to simulate interviews and internal recommendations, iterating through interviews and feedback, ultimately helping students successfully obtain offers.

2. Employment Escort

For technical issues encountered in work after students are employed, a week of follow-up service will be provided to ensure stable employment for students.

Course Start Date

March 23

Scan to View Course Details

↓ Scan Me ↓

Click ↓↓“Read Original” to view course details!