Real-Time Face Detection with Raspberry Pi

This article introduces how to use OpenCV and Python to achieve a face detection project on Raspberry Pi. The project not only describes the specific steps required to recognize faces but also provides a lot of extended knowledge. Moreover, this project does not require readers to understand detailed theoretical knowledge of face recognition, so beginners can easily follow the steps to achieve it.

Required Equipment

Hardware:

• Raspberry Pi 3 Model B;

• Raspberry Pi Camera Module (PiCam).

Languages and Libraries:

• OpenCV

• Python 3

Steps

This article mainly discusses how to use PiCam to achieve real-time face recognition, as shown in the following picture:

This tutorial uses OpenCV, a magical “open-source computer vision library”, and mainly focuses on the Raspberry Pi (thus, the operating system is Raspberry Pi OS) and Python, but I also tested the code on a Mac computer, and it runs well. OpenCV has strong computational efficiency and is specifically designed for real-time applications. Therefore, it is very suitable for real-time face recognition using a camera. To create a complete face recognition project, we must complete three stages:

1) Face detection and data collection;

2) Training the recognizer;

3) Face recognition.

As shown in the following picture:

Step 1: Material ListMain components:

• Raspberry Pi V3: 283 RMB (Taobao)

• 5 megapixel 1080p sensor OV5647 mini camera module: 83 RMB (Taobao)

Step 2: Install OpenCV 3 PackageI am using Raspberry Pi V3 with the updated latest Raspberry Pi OS (Stretch), so the best way to install OpenCV is to follow the tutorial written by Adrian Rosebrock: “Raspbian Stretch: Install OpenCV 3 + Python on your Raspberry Pi”. After several attempts, I found Adrian’s tutorial to be the best, and I recommend following this tutorial step by step to install.

After completing the above tutorial, you should have installed the OpenCV virtual environment, which can be used to run this experiment on the Raspberry Pi device.

Let’s go to the virtual environment and confirm that OpenCV 3 has been correctly installed.

Adrian recommends running the command “source” every time you open a new terminal to ensure that the system variables are set correctly.

source ~/.profile

Then, we enter the virtual environment:

workon cv

If you see (cv) appear before the prompt, then you have entered the cv virtual environment:

(cv) pi@raspberry:~$

Adrian wants everyone to note that the cv Python virtual environment is completely independent and isolated from the default Python version included in Raspbian Stretch. Therefore, any Python package in the global site-packages directory is not available to the cv virtual environment. Similarly, any Python package in the cv site-packages is also not available for global Python package installations.

Now, enter the Python interpreter:

python

Confirm you are running version 3.5 (or above). Inside the interpreter (“>>>” will appear), import the OpenCV library:

import cv2

If there are no error messages, then OpenCV has been correctly installed in your Python virtual environment.

You can also check the installed OpenCV version:

cv2.__version__

This should show 3.3.0 (or a higher version that may be released in the future).

The terminal screenshot above shows the above steps.

Step 3: Test the Camera After installing OpenCV on the Raspberry Pi, we test it to confirm that the camera is working properly. Assume you have already installed PiCam on the Raspberry Pi.

In the IDE, enter the following Python code:

import numpy as np
import cv2
cap = cv2.VideoCapture(0)
cap.set(3,640) # set Width
cap.set(4,480) # set Height
while(True):
    ret, frame = cap.read()
    frame = cv2.flip(frame, -1) # Flip camera vertically
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    cv2.imshow('frame', frame)
    cv2.imshow('gray', gray)

    k = cv2.waitKey(30) & 0xff
    if k == 27: # press 'ESC' to quit
        break
cap.release()
cv2.destroyAllWindows()

The above code captures the video stream generated by PiCam and displays it in BGR color and grayscale mode.

Note: I vertically rotated the camera according to the assembly method. If your situation is not like this, then comment or delete the “flip” command line.

You can also download the code from my GitHub: https://github.com/Mjrovai/OpenCV-Object-Face-Tracking/blob/master/simpleCamTest.py

Enter the following command line to start execution:

python simpleCamTest.py

To complete the program, you must press the [ESC] key on the keyboard. Before pressing the [ESC] key, first click the mouse on the video window.

The above image shows the result. To learn more about OpenCV, please refer to this tutorial: https://pythonprogramming.net/loading-video-python-opencv-tutorial/

Step 4: Face Detection The most basic task of face recognition is “face detection”. You must first “capture” the face (Phase 1) in order to recognize it when comparing it with the captured new face (Phase 3).

The most common method for face detection is to use the “Haar Cascade Classifier”. The goal of object detection using Haar feature-based cascade classifiers is an efficient object detection method proposed by Paul Viola and Michael Jones in their 2001 paper “Rapid Object Detection using a Boosted Cascade of Simple Features”. This machine learning method trains a cascade function based on a large number of positive and negative images, and then uses it to detect objects in other images. Here, we will use it for face recognition. Initially, the algorithm requires a large number of positive images (face images) and negative images (images without faces) to train the classifier. Then we need to extract features from it. The good news is that OpenCV has trainers and detectors. If you want to train your own object classifier, such as cars, airplanes, etc., you can create one using OpenCV.

For details, see: https://docs.opencv.org/3.3.0/dc/d88/tutorial_traincascade.html.

If you do not want to create your own classifier, OpenCV also includes many pre-trained classifiers for detecting faces, eyes, smiles, etc. The relevant XML files can be downloaded from this directory: https://github.com/Itseez/opencv/tree/master/data/haarcascades.

Now, let’s start creating a face detector using OpenCV!

Download the file faceDetection.py from my GitHub: https://github.com/Mjrovai/OpenCV-Face-Recognition/blob/master/FaceDetection/faceDetection.py.

import numpy as np
import cv2
faceCascade = cv2.CascadeClassifier('Cascades/haarcascade_frontalface_default.xml')
cap = cv2.VideoCapture(0)
cap.set(3,640) # set Width
cap.set(4,480) # set Height
while True:
    ret, img = cap.read()
    img = cv2.flip(img, -1)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = faceCascade.detectMultiScale(
        gray,     
        scaleFactor=1.2,
        minNeighbors=5,     
        minSize=(20, 20)
    )
    for (x,y,w,h) in faces:
        cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
        roi_gray = gray[y:y+h, x:x+w]
        roi_color = img[y:y+h, x:x+w]  
    cv2.imshow('video',img)
    k = cv2.waitKey(30) & 0xff
    if k == 27: # press 'ESC' to quit
        break
cap.release()
cv2.destroyAllWindows()

Using Python and OpenCV to perform face detection, the above few lines of code are sufficient. Note the following code:

faceCascade = cv2.CascadeClassifier('Cascades/haarcascade_frontalface_default.xml')

This line of code loads the “classifier” (which must be in the Cascades/ directory under the project folder). Then, we call the camera in the loop and load our input video in grayscale mode. Now, we must call the classifier function and input some very important parameters, such as scale factor, number of neighbors, and minimum size for face detection.

faces = faceCascade.detectMultiScale(
        gray,     
        scaleFactor=1.2,
        minNeighbors=5,     
        minSize=(20, 20)
    )

Where:

• gray represents the input grayscale image.

• scaleFactor represents the size of each image reduced by a scale.

• minNeighbors represents the number of neighbors for each candidate rectangle. The larger the number, the fewer false positives.

• minSize represents the minimum rectangle size for face recognition.

This function will detect faces in the image. Next, we must “mark” the faces in the image, for example, with a blue rectangle. Use the following code to complete this step:

for (x,y,w,h) in faces:
    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w] 

If the face has been marked, the function will return the position of the detected face as a rectangle, with the upper left corner (x,y), w representing width, h representing height ==> (x,y,w,h). See the diagram below.

After obtaining this positional information, we can create a “region of interest” (draw a rectangle) for the face and present the result using the imshow() function. Run the above Python script in your Python environment on the Raspberry Pi terminal:

python faceDetection.py

The result is as follows:

You can also add detectors such as “eye detection” or even “smile detection”. In those cases, you need to add the classifier function and rectangle within the original face recognition area, as recognition outside that area is meaningless.

Note that on the Raspberry Pi, the classification method (HaarCascades) consumes a lot of computing power, so using multiple classifiers in the same code will significantly slow down processing speed. Running these algorithms on a desktop is much easier.

You can see additional examples on my GitHub:

• faceEyeDetection.py

• faceSmileDetection.py

• faceSmileEyeDetection.py

In the following image, you can see our results:

To gain a deeper understanding of face recognition, please refer to this tutorial: https://pythonprogramming.net/haar-cascade-face-eye-detection-python-opencv-tutorial/

Step 5: Data Collection I recommend that readers check out the following two tutorials on face recognition:

• Implementing Face Recognition from Scratch using OpenCV and Python: https://www.superdatascience.com/opencv-face-recognition/

• Understanding Face Recognition: https://thecodacus.com/category/opencv/face-recognition/

Now, the first step of our project is to create a simple dataset that will store the ID of each face and a set of grayscale images used for face detection.

Therefore, the following command line will create a directory for our project, which can be named FacialRecognitionProject or something else:

mkdir FacialRecognitionProject

In this directory, in addition to the three Python scripts we created for the project, we also need to store the face classifier. We can download it from GitHub: haarcascade_frontalface_default.xml.

The next step is to create a subdirectory “dataset” to store face samples:

mkdir dataset

Then download the code 01_face_dataset.py from my GitHub.

import cv2
import os
cam = cv2.VideoCapture(0)
cam.set(3, 640) # set video width
cam.set(4, 480) # set video height
face_detector = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
# For each person, enter one numeric face id
face_id = input('
 enter user id end press <return> ==>  ')
print("\n [INFO] Initializing face capture. Look the camera and wait ...")
# Initialize individual sampling face count
count = 0
while(True):
    ret, img = cam.read()
    img = cv2.flip(img, -1) # flip video image vertically
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_detector.detectMultiScale(gray, 1.3, 5)
    for (x,y,w,h) in faces:
        cv2.rectangle(img, (x,y), (x+w,y+h), (255,0,0), 2)     
        count += 1
        # Save the captured image into the datasets folder
        cv2.imwrite("dataset/User." + str(face_id) + '.' + str(count) + ".jpg", gray[y:y+h,x:x+w])
        cv2.imshow('image', img)
    k = cv2.waitKey(100) & 0xff # Press 'ESC' for exiting video
    if k == 27:
        break
    elif count >= 30: # Take 30 face sample and stop video
         break
# Do a bit of cleanup
print("\n [INFO] Exiting Program and cleanup stuff")
cam.release()
cv2.destroyAllWindows()

The above code is very similar to the face recognition code; we just added an “input command” to capture the user ID (an integer).

face_id = input('
 enter user id end press  ==>  ')

For each captured frame, we should save it in the “dataset” directory:

cv2.imwrite("dataset/User." + str(face_id) + '.' + str(count) + ".jpg", gray[y:y+h,x:x+w])

To save the above files, we need to import the “os” library, and each file name follows the structure below:

User.face_id.count.jpg

For example, for user with face_id = 1, the fourth sample file in the dataset/ directory may be:

User.1.4.jpg

In my Raspberry Pi, this image can be opened as:

In my code, I capture 30 samples from each ID, and we can modify the number of samples taken in the last condition statement. If we want to recognize a new user or modify the existing user’s photo, we must use the above script.

Step 6: Training In the second phase, we need to extract all user data from the dataset and train the OpenCV recognizer, which can be done directly by specific OpenCV functions. This step will be saved as a .yml file in the “trainer/” directory.

So, let’s start creating a subdirectory to store the training data:

mkdir trainer

Download the second Python script 02_face_training.py from my GitHub.

import cv2
import numpy as np
from PIL import Image
import os
# Path for face image database
path = 'dataset'
recognizer = cv2.face.LBPHFaceRecognizer_create()
detector = cv2.CascadeClassifier("haarcascade_frontalface_default.xml");
# function to get the images and label data
def getImagesAndLabels(path):
    imagePaths = [os.path.join(path,f) for f in os.listdir(path)]     
    faceSamples=[]
    ids = []
    for imagePath in imagePaths:
        PIL_img = Image.open(imagePath).convert('L') # convert it to grayscale
        img_numpy = np.array(PIL_img,'uint8')
        id = int(os.path.split(imagePath)[-1].split(".")[1])
        faces = detector.detectMultiScale(img_numpy)
        for (x,y,w,h) in faces:
            faceSamples.append(img_numpy[y:y+h,x:x+w])
            ids.append(id)
    return faceSamples,ids
print ("\n [INFO] Training faces. It will take a few seconds. Wait ...")
faces,ids = getImagesAndLabels(path)
recognizer.train(faces, np.array(ids))
# Save the model into trainer/trainer.yml
recognizer.write('trainer/trainer.yml') # recognizer.save() worked on Mac, but not on Pi
# Print the number of faces trained and end program
print("\n [INFO] {0} faces trained. Exiting Program".format(len(np.unique(ids))))

Make sure the PIL library is installed in Rpi; if not, run the following command in the terminal:

pip install pillow

We will use the LBPH (LOCAL BINARY PATTERNS HISTOGRAMS) face recognizer, which is provided by OpenCV:

recognizer = cv2.face.LBPHFaceRecognizer_create()

The function “getImagesAndLabels(path)” will extract all the photos in the “dataset/” directory and return two arrays: “Ids” and “faces”. By using these arrays as input, we can train the recognizer.

recognizer.train(faces, ids)

After training, the file “trainer.yml” will be saved in the trainer directory we defined earlier. Additionally, we also used the print function at the end to confirm the number of trained user faces.

Step 7: Recognizer This is the final phase of the project. Here, we will capture a new face through the camera, and if this person’s face was previously captured and trained, our recognizer will return its predicted ID and index, displaying how confident the recognizer is about this judgment.

Let’s download the third phase Python script from GitHub 03_face_recognition.py.

import cv2
import numpy as np
import os 
recognizer = cv2.face.LBPHFaceRecognizer_create()
recognizer.read('trainer/trainer.yml')
cascadePath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath);
font = cv2.FONT_HERSHEY_SIMPLEX
# initiate id counter
id = 0
# names related to ids: example ==> Marcelo: id=1,  etc
names = ['None', 'Marcelo', 'Paula', 'Ilza', 'Z', 'W'] 
# Initialize and start realtime video capture
cam = cv2.VideoCapture(0)
cam.set(3, 640) # set video width
cam.set(4, 480) # set video height
# Define min window size to be recognized as a face
minW = 0.1*cam.get(3)
minH = 0.1*cam.get(4)
while True:
    ret, img =cam.read()
    img = cv2.flip(img, -1) # Flip vertically
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    faces = faceCascade.detectMultiScale( 
        gray,
        scaleFactor = 1.2,
        minNeighbors = 5,
        minSize = (int(minW), int(minH)),
       )
    for(x,y,w,h) in faces:
        cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0), 2)
        id, confidence = recognizer.predict(gray[y:y+h,x:x+w])
        # Check if confidence is less than 100 ==> "0" is perfect match 
        if (confidence < 100):
            id = names[id]
            confidence = "  {0}%".format(round(100 - confidence))
        else:
            id = "unknown"
            confidence = "  {0}%".format(round(100 - confidence))

        cv2.putText(img, str(id), (x+5,y-5), font, 1, (255,255,255), 2)
        cv2.putText(img, str(confidence), (x+5,y+h-5), font, 1, (255,255,0), 1)  

    cv2.imshow('camera',img) 
    k = cv2.waitKey(10) & 0xff # Press 'ESC' for exiting video
    if k == 27:
        break
# Do a bit of cleanup
print("\n [INFO] Exiting Program and cleanup stuff")
cam.release()
cv2.destroyAllWindows()

Here we include a new array, so we will display the “names” instead of the numbered IDs:

names = ['None', 'Marcelo', 'Paula', 'Ilza', 'Z', 'W']

So, as shown in the list above, Marcelo’s ID or index is 1, Paula’s ID is 2.

The next step is to detect a face, just as we did in the previous HaarCascade classifier.

id, confidence = recognizer.predict(gray portion of the face)

recognizer.predict() will take the captured portion of the face to be analyzed as a parameter and return its possible owner, indicating its ID and the confidence level associated with this match.

Note that if the match is perfect, the confidence index will return “zero”.

Finally, if the recognizer can predict the face, we will place a text on the image with the possible ID and the probability of whether the match is correct (probability = 100 – confidence index). If not, we will label the face as “unknown”.

Below is an image of this result:

In this image, I show some tests completed by this project, where I also used images to verify whether the recognizer is effective.

Step 8: Conclusion

I hope this project helps others discover more fun projects and also helps readers implement their own face recognition applications.

For more detailed code, please check the GitHub address: https://github.com/Mjrovai/OpenCV-Face-Recognition

References:

Machine Heart Practice | Amazon explains how to build a real-time object recognition system on Raspberry Pi using MXNet

Tutorial | BerryNet: How to implement a deep learning intelligent gateway on Raspberry Pi

Original link: https://www.hackster.io/mjrobot/real-time-face-recognition-an-end-to-end-project-a10826

This article is compiled by Machine Heart, please contact this public account for authorization to reprint.

✄————————————————

Join Machine Heart (full-time reporter/intern): [email protected]

Submissions or requests for coverage: [email protected]

Advertising & business cooperation: [email protected]