This article introduces how to complete a face detection project on Raspberry Pi using OpenCV and Python. The project not only describes the specific steps required to identify faces but also provides a lot of extended knowledge. Furthermore, this project does not require readers to have a detailed understanding of face recognition theory, making it easy for beginners to follow the steps.

Required Equipment

Hardware: Raspberry Pi 3, Raspberry Pi Camera Module (PiCam)

Language and Libraries: OpenCV, Python 3

This article mainly discusses how to achieve real-time face recognition using PiCam.

The tutorial uses OpenCV, a magical “open-source computer vision library,” and mainly focuses on Raspberry Pi (hence, the operating system is Raspberry Pi OS) and Python, but I also tested the code on a Mac, and it runs well too. OpenCV has strong computational efficiency and is specifically designed for real-time applications. Therefore, it is very suitable for real-time face recognition using a camera. To create a complete face recognition project, we must complete three stages:

1) Face detection and data collection; 2) Training the recognizer; 3) Face recognition.

As shown in the figure below:

Step 1: Material List

Main component: Raspberry Pi 3, 5 MP 1080p sensor OV5647 mini camera module

Step 2: Install OpenCV 3 Package

I am using Raspberry Pi V3 with the latest updated Raspberry Pi OS (Stretch), so the best way to install OpenCV is to follow the tutorial written by Adrian Rosebrock: “Raspbian Stretch: Install OpenCV 3 + Python on your Raspberry Pi.” After several attempts, I found Adrian’s tutorial to be the best and recommend following it step by step for installation.

After completing the above tutorial, you should have installed the OpenCV virtual environment, which can be used to run this experiment on the Raspberry Pi device. We enter the virtual environment and confirm that OpenCV 3 has been correctly installed. Adrian recommends running the command line “source” every time you open a new terminal to ensure that the system variables are set correctly.

source ~/.profile

Then, we enter the virtual environment:

workon cv

If you see (cv) appear before the prompt, then you have entered the cv virtual environment:

(cv) pi@raspberry:~$

Adrian wants everyone to note that the cv Python virtual environment is completely isolated and separate from the default Python version included in Raspbian Stretch. Therefore, any Python packages in the global site-packages directory are not available for the cv virtual environment. Similarly, any Python packages in the cv site-packages are also not available for global Python package installations.

Now, enter the Python interpreter:

python

Confirm that you are running version 3.5 (or above).

Inside the interpreter (where >>> appears), import the OpenCV library:

import cv2

If there are no error messages, then OpenCV has been correctly installed in your Python virtual environment.

You can also check the installed version of OpenCV:

cv2.__version__

It should show 3.3.0 (or a higher version may be released in the future). The terminal screenshot above shows the above steps.

Step 3: Test the Camera

After installing OpenCV on Raspberry Pi, we will test it to confirm that the camera is functioning properly. Assuming you have installed PiCam on the Raspberry Pi.

Enter the following Python code in the IDE:

import numpy as np
import cv2

cap = cv2.VideoCapture(0)
cap.set(3,640) # set Width
cap.set(4,480) # set Height
 
while(True):
    ret, frame = cap.read()
    frame = cv2.flip(frame, -1) # Flip camera vertically
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    
    cv2.imshow('frame', frame)
    cv2.imshow('gray', gray)
    
    k = cv2.waitKey(30) & 0xff
    if k == 27: # press 'ESC' to quit
        break

cap.release()
cv2.destroyAllWindows()

The above code captures the video stream generated by PiCam and displays it in BGR color and grayscale mode. Note: I vertically rotated the camera according to the assembly method. If your situation is different, comment out or delete the “flip” command line. You can also download the code from my GitHub: https://github.com/Mjrovai/OpenCV-Object-Face-Tracking/blob/master/simpleCamTest.py

Enter the following command line to start execution:

python simpleCamTest.py

To complete the program, you must press the [ESC] key on the keyboard. Before pressing the [ESC] key, click the video window with the mouse.

The above image shows the result.

To learn more about OpenCV, please refer to this tutorial: https://pythonprogramming.net/loading-video-python-opencv-tutorial/

Step 4: Face Detection

The most basic task of face recognition is “face detection.” You must first “capture” the face (Step 1) before you can recognize it when comparing with captured new faces (Step 3).

The most common method for face detection is using the “Haar Cascade Classifier.” The goal detection using Haar feature-based cascade classifiers is an efficient object detection method proposed by Paul Viola and Michael Jones in their 2001 paper “Rapid Object Detection using a Boosted Cascade of Simple Features.” This machine learning method trains cascade functions based on a large number of positive and negative images, and then uses them to detect objects in other images. Here, we will use it for face recognition. Initially, the algorithm requires a large number of positive images (face images) and negative images (images without faces) to train the classifier. Then we need to extract features from them. The good news is that OpenCV has trainers and detectors. If you want to train your own object classifier, such as cars, airplanes, etc., you can create one using OpenCV.

For details, see: Cascade Classifier Training

If you don’t want to create your own classifier, OpenCV also includes many pre-trained classifiers for detecting faces, eyes, smiles, etc. The related XML files can be downloaded from this directory: haarcascades

Now, let’s start creating a face detector using OpenCV!

Download the file faceDetection.py from my GitHub: faceDetection.py

import numpy as np
import cv2

faceCascade = cv2.CascadeClassifier('Cascades/haarcascade_frontalface_default.xml')

cap = cv2.VideoCapture(0)
cap.set(3,640) # set Width
cap.set(4,480) # set Height

while True:
    ret, img = cap.read()
    img = cv2.flip(img, -1)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = faceCascade.detectMultiScale(
        gray,     
        scaleFactor=1.2,
        minNeighbors=5,     
        minSize=(20, 20)
    )

    for (x,y,w,h) in faces:
        cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
        roi_gray = gray[y:y+h, x:x+w]
        roi_color = img[y:y+h, x:x+w]  

    cv2.imshow('video',img)

    k = cv2.waitKey(30) & 0xff
    if k == 27: # press 'ESC' to quit
        break

cap.release()
cv2.destroyAllWindows()

Using Python and OpenCV to perform face detection, the above few lines of code are sufficient. Note the following code:

faceCascade = cv2.CascadeClassifier('Cascades/haarcascade_frontalface_default.xml')

This line of code loads the “classifier” (which must be in the Cascades/ directory under the project folder). Then, we call the camera in the loop and load our input video in grayscale mode. Now, we must call the classifier function and provide some very important parameters, such as scale factor, number of neighbors, and minimum size for face detection.

faces = faceCascade.detectMultiScale(
        gray,     
        scaleFactor=1.2,
        minNeighbors=5,     
        minSize=(20, 20)
    )

Where: gray represents the input grayscale image. scaleFactor indicates the size reduction ratio for each image. minNeighbors indicates the number of neighboring rectangles for each candidate rectangle. The larger the number, the fewer false positives. minSize indicates the minimum rectangle size for face recognition.

This function will detect faces in the image. Next, we need to “mark” the faces in the image, for example, by drawing a blue rectangle. Use the following code to complete this step:

for (x,y,w,h) in faces:
    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w] 

If the faces have been marked, the function will return the detected face positions as a rectangle, with the top left corner (x,y), w representing the width, and h representing the height ==> (x,y,w,h). See the figure below.

After obtaining this position information, we can create a “region of interest” for the face (draw a rectangle) and present the result using the imshow() function. Run the above Python script in your Python environment on the Raspberry Pi terminal:

python faceDetection.py

You can also add detectors such as “eye detection” or even “smile detection.” In those cases, you need to include the classifier functions and rectangle boxes within the original face recognition area because it makes no sense to recognize outside the region. Note that on Raspberry Pi, the classification method (HaarCascades) consumes a lot of computing power, so using multiple classifiers in the same code will significantly slow down processing speed. Running these algorithms on a desktop is much easier.

You can see other examples on my GitHub: faceEyeDetection.py, faceSmileDetection.py, faceSmileEyeDetection.py

In the figure below, you can see our results:

To gain a deeper understanding of face recognition, you can refer to this tutorial: Haar Cascade Object Detection Face & Eye OpenCV Python Tutorial

Step 5: Collect Data

I recommend that readers check out the following two tutorials on face recognition: Implementing Face Recognition from Scratch with OpenCV and Python: superdatascience.com Understanding Face Recognition: thecodacus.com

Now, the first step of our project is to create a simple dataset that will store each face’s ID and a set of grayscale images for face detection.

Therefore, the following command line will create a directory for our project, which can be named FacialRecognitionProject or something else:

mkdir FacialRecognitionProject

In this directory, in addition to the three Python scripts we create for the project, we also need to store the face classifier. We can download it from GitHub: haarcascade_frontalface_default.xml

The next step is to create a subdirectory “dataset” to store face samples:

mkdir dataset

Then download the code 01_face_dataset.py from my GitHub.

import cv2
import os

cam = cv2.VideoCapture(0)
cam.set(3, 640) # set video width
cam.set(4, 480) # set video height

face_detector = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# For each person, enter one numeric face id
face_id = input('\n enter user id end press==>  ')

print("\n [INFO] Initializing face capture. Look the camera and wait ...")
# Initialize individual sampling face count
count = 0

while(True):
    ret, img = cam.read()
    img = cv2.flip(img, -1) # flip video image vertically
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_detector.detectMultiScale(gray, 1.3, 5)

    for (x,y,w,h) in faces:
        cv2.rectangle(img, (x,y), (x+w,y+h), (255,0,0), 2)     
        count += 1

        # Save the captured image into the datasets folder
        cv2.imwrite("dataset/User." + str(face_id) + '.' + str(count) + ".jpg", gray[y:y+h,x:x+w])

        cv2.imshow('image', img)

    k = cv2.waitKey(100) & 0xff # Press 'ESC' for exiting video
    if k == 27:
        break
    elif count >= 30: # Take 30 face sample and stop video
         break

# Do a bit of cleanup
print("\n [INFO] Exiting Program and cleanup stuff")
cam.release()
cv2.destroyAllWindows()

The above code is very similar to the face recognition code; we just added an “input command” to capture the user ID (an integer).

face_id = input('\n enter user id end press  ==>  ')

For each captured frame, we should save it in the “dataset” directory:

cv2.imwrite("dataset/User." + str(face_id) + '.' + str(count) + ".jpg", gray[y:y+h,x:x+w])

For saving the above files, we need to import the “os” library, and each file’s name follows the structure:

User.face_id.count.jpg

For example, for user with face_id = 1, the fourth sample file name in the dataset/ directory may be:

User.1.4.jpg

In my Raspberry Pi, this image can be opened as:

In my code, I capture 30 samples from each ID, and we can modify the number of samples extracted in the last conditional statement. If we want to recognize new users or modify existing users’ photos, we must use the above script.

Step 6: Training

In the second stage, we need to extract all user data from the dataset and train the OpenCV recognizer, which can be done directly by specific OpenCV functions. This step will be saved as a .yml file in the “trainer/” directory.

So, let’s start creating a subdirectory to store the training data:

mkdir trainer

Download the second Python script 02_face_training.py from my GitHub.

import numpy as np
from PIL import Image
import os

# Path for face image database
path = 'dataset'

recognizer = cv2.face.LBPHFaceRecognizer_create()
detector = cv2.CascadeClassifier("haarcascade_frontalface_default.xml");

# function to get the images and label data
def getImagesAndLabels(path):
    imagePaths = [os.path.join(path,f) for f in os.listdir(path)]     
    faceSamples=[]
    ids = []
    for imagePath in imagePaths:
        PIL_img = Image.open(imagePath).convert('L') # convert it to grayscale
        img_numpy = np.array(PIL_img,'uint8')
        id = int(os.path.split(imagePath)[-1].split(".")[1])
        faces = detector.detectMultiScale(img_numpy)
        for (x,y,w,h) in faces:
            faceSamples.append(img_numpy[y:y+h,x:x+w])
            ids.append(id)
    return faceSamples,ids

print ("\n [INFO] Training faces. It will take a few seconds. Wait ...")
faces,ids = getImagesAndLabels(path)
recognizer.train(faces, np.array(ids))

# Save the model into trainer/trainer.yml
recognizer.write('trainer/trainer.yml') # recognizer.save() worked on Mac, but not on Pi

# Print the number of faces trained and end program
print("\n [INFO] {0} faces trained. Exiting Program".format(len(np.unique(ids))))

Make sure the PIL library is installed in Rpi; if not, run the following command in the terminal:

pip install pillow

We will use the LBPH (LOCAL BINARY PATTERNS HISTOGRAMS) face recognizer provided by OpenCV:

recognizer = cv2.face.LBPHFaceRecognizer_create()

The function “getImagesAndLabels(path)” will extract all photos in the “dataset/” directory and return two arrays: “Ids” and “faces”. By using these arrays as inputs, we can train the recognizer.

recognizer.train(faces, ids)

After training, the file “trainer.yml” will be saved in the trainer directory we defined earlier. Additionally, we also used the print function at the end to confirm the number of trained user faces.

Step 7: Recognizer

This is the final stage of the project. Here, we will capture a new face through the camera, and if this person’s face has been captured and trained before, our recognizer will return its predicted ID and index, showing how confident the recognizer is in that judgment.

Let’s download the third-stage python script from GitHub 03_face_recognition.py.

import cv2
import numpy as np
import os 

recognizer = cv2.face.LBPHFaceRecognizer_create()
recognizer.read('trainer/trainer.yml')
cascadePath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath);

font = cv2.FONT_HERSHEY_SIMPLEX

# initiate id counter
id = 0

# names related to ids: example ==> Marcelo: id=1,  etc
names = ['None', 'Marcelo', 'Paula', 'Ilza', 'Z', 'W'] 

# Initialize and start realtime video capture
cam = cv2.VideoCapture(0)
cam.set(3, 640) # set video width
cam.set(4, 480) # set video height

# Define min window size to be recognized as a face
minW = 0.1*cam.get(3)
minH = 0.1*cam.get(4)

while True:
    ret, img =cam.read()
    img = cv2.flip(img, -1) # Flip vertically
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    
    faces = faceCascade.detectMultiScale( 
        gray,
        scaleFactor = 1.2,
        minNeighbors = 5,
        minSize = (int(minW), int(minH)),
       )

    for(x,y,w,h) in faces:
        cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0), 2)
        id, confidence = recognizer.predict(gray[y:y+h,x:x+w])

        # Check if confidence is less than 100 ==> "0" is perfect match 
        if (confidence < 100):
            id = names[id]
            confidence = "  {0}%".format(round(100 - confidence))
        else:
            id = "unknown"
            confidence = "  {0}%".format(round(100 - confidence))
        
        cv2.putText(img, str(id), (x+5,y-5), font, 1, (255,255,255), 2)
        cv2.putText(img, str(confidence), (x+5,y+h-5), font, 1, (255,255,0), 1)  
    
    cv2.imshow('camera',img) 

    k = cv2.waitKey(10) & 0xff # Press 'ESC' for exiting video
    if k == 27:
        break

# Do a bit of cleanup
print("\n [INFO] Exiting Program and cleanup stuff")
cam.release()
cv2.destroyAllWindows()

Here we include a new array, so we will display “names” instead of numbered IDs:

names = ['None', 'Marcelo', 'Paula', 'Ilza', 'Z', 'W']

So, as shown in the list, Marcelo’s ID or index is 1, and Paula’s ID is 2.

Next, we will detect a face, just as we did in the previous haasCascade classifier.

id, confidence = recognizer.predict(gray portion of the face)

recognizer.predict() will take the captured portion of the face to be analyzed as a parameter and return its possible owner, indicating its ID and the confidence level related to that match. Note that if the match is perfect, the confidence index will return “zero.”

Finally, if the recognizer can predict the face, we will place a text on the image, showing the possible ID and the probability of the match being correct (Probability = 100 – Confidence Index). If not, we will place the label “unknown” on the face.

In the image above, I show some tests completed by this project, where I also used images to verify the effectiveness of the recognizer.

Step 8: Conclusion

I hope this project helps others discover more fun projects and assists readers in achieving their own face recognition applications.

For more detailed code, please refer to the GitHub address: https://github.com/Mjrovai/OpenCV-Face-Recognition (Original text from hackster, compiled by Machine Heart)

Click 【Read Original】 to see the text link in the article

Submission: [email protected]