Merry Christmas – Keras + Raspberry Pi: Using Deep Learning to Recognize Santa Claus

Selected from PyImageSearch

Author:Adrian Rosebrock

Translated by Machine Heart

Contributors: Panda

As Christmas approaches, you may have already spotted Santa Claus in shopping malls, parks, or your own chimney. With the advancement of machine recognition technology, using artificial intelligence to identify Santa Claus passing by or delivering gifts seems like a great idea. Recently, Adrian Rosebrock published a tutorial on PyImageSearch, introducing the process of implementing a deep learning Santa Claus recognizer on Raspberry Pi using Keras. You can access the original text via the link at the end of this article—also, you can leave your email address at the end of the original article to request the complete code for this project from the original author. Additionally, this tutorial mentions many other related tutorials and projects, and for easier reading, Machine Heart has shortened these links.

This is the most interesting tutorial I’ve written on PyImageSearch! The topics covered include:

• Deep Learning

• Raspberry Pi Development Board

• 3D Christmas Tree

• Reference to the ‘Not Hotdog’ detector from the HBO TV series Silicon Valley

• I dressed up as Santa Claus

To make the most of this Christmas holiday, I will introduce how to deploy a deep learning model trained with Keras on Raspberry Pi.

But this is not just any machine learning model…

This image classifier is specifically designed to detect whether Santa Claus is present in our video stream.

If we do detect Santa Claus…

Well, I don’t want to spoil it (but it does relate to the 3D Christmas tree and a cheerful tune).

Please enjoy this tutorial, download the code, and try it out yourself!

The most important thing: have fun!

Using Keras for Deep Learning on Raspberry Pi

This article will comprehensively introduce the process of running deep neural networks on Raspberry Pi using Keras. The goal of this project is to create a Not Santa detector, so we can demonstrate this specifically and also have fun.

In the first part of the main text, we will talk about what the Not Santa detector is (because you may not be familiar with the Not Hotdog detector from HBO’s Silicon Valley, which has become quite popular).

Then we will configure our Raspberry Pi to perform deep learning tasks—we will install TensorFlow, Keras, and some other essential packages and libraries.

Once our Raspberry Pi is ready for deep learning, we will create a Python script that can:

1. Load our Keras model from disk

2. Access our Raspberry Pi camera module/USB webcam

3. Apply deep learning to detect whether Santa Claus is present in the video frames

4. If Santa Claus is present, access our GPIO pins and play music

I love writing this kind of article on PyImageSearch because tutorials like this can bring together various technologies, and this article includes:

• Deploying deep learning on Raspberry Pi; see: https://goo.gl/3MPJUp

• Accessing Raspberry Pi camera module/USB webcam; see: https://goo.gl/NEZrGK

• Controlling GPIO and computers on Raspberry Pi; see: https://goo.gl/nwoS38

So let’s get started!

What is the Not Santa Detector?

Figure 1: The Not Hotdog detector application from the TV series Silicon Valley

The Not Santa detector is inspired by the HBO series Silicon Valley, where a character creates a smartphone app that can detect whether an input image is a ‘hotdog’ or ‘not hotdog’.

This show evidently makes fun of the startup culture in Silicon Valley, including:

1. The hype around machine learning/deep learning concepts

2. The satire of their redoing a lot of minimally useful smart applications (but their creators believe their apps will ‘change the world’).

I decided to have some fun myself.

Today we are going to create a Not Santa detector that can detect whether Santa Claus is present in images or video frames.

If you don’t know what Santa Claus is, here’s a brief introduction. He is a fictional character in Western culture, characterized as a jolly, plump man with a white beard, who delivers gifts to children while they sleep on Christmas Eve.

However, our application is not just for fun or satire!

We can learn many practical skills in the process, including:

1. How to configure your Raspberry Pi for deep learning tasks

2. Installing Keras and TensorFlow on your Raspberry Pi

3. Deploying a previously trained convolutional neural network (using Keras) on your Raspberry Pi

4. Executing a given action when a positive case is detected

But before diving into the code, let’s take a look at the hardware we need.

What Hardware is Required?

Figure 2: The hardware for the Not Santa detector includes Raspberry Pi 3, speakers, a 3D Christmas tree, and a webcam (not shown in the image). This Raspberry Pi contains a Python script using Keras to detect Santa Claus.

If you want to follow this tutorial (without modifications), you will need:

• A Raspberry Pi 3 development board (or a Raspberry Pi 3 starter kit, highly recommended)

• A Raspberry Pi camera module or a USB webcam. In this tutorial, I used the Logitech C920, as it provides good value (and has a USB cable that gives you a bit more operational space compared to the short cable of the Raspberry Pi camera).

• A 3D Christmas tree available for Raspberry Pi (designed by Rachel Rayns): https://goo.gl/GSqxjs

• A set of speakers—I recommend this one from Pi Hut: https://goo.gl/A7MmR4; or this one from Adafruit: https://www.adafruit.com/product/1363. Or if you prefer small and powerful, you can choose this one: http://amzn.to/2AF920r

Of course, you don’t need all these components.

In fact, just having a Raspberry Pi and a camera module/USB webcam is enough (but then you will need to modify the code so that it doesn’t try to access the GPIO pins or play music through the speakers).

Your setup should be similar to mine, as shown in the above Figure 2, which has the speakers, 3D Christmas tree, and webcam connected (but the webcam is off camera).

I also recommend connecting an HDMI display and keyboard on top for testing and debugging your script.

Figure 3: My deep learning setup includes Raspberry Pi and its components, along with a keyboard, mouse, and a small HDMI display. With this setup, we should definitely catch Santa Claus delivering gifts in front of my Christmas tree.

You can see my Raspberry Pi, HDMI display, keyboard, and a Christmas animal friend in the above image, who accompanied me while I completed this tutorial.

How to Install TensorFlow and Keras on Raspberry Pi?

Figure 4: We will use Keras with TensorFlow backend to implement the deep learning Not Santa detector on Raspberry Pi

We previously discussed how to train a convolutional neural network using Keras that can recognize whether there is Santa Claus in the input image; see: https://goo.gl/imxkrY

Here we will deploy the previously trained model on Raspberry Pi.

I have also mentioned that Raspberry Pi is not suitable for training neural networks (except for simple play cases). But once the neural network is trained, we can use Raspberry Pi to deploy it (of course, the model must be small enough to fit into the Raspberry Pi’s memory).

I assume you have installed OpenCV on your Raspberry Pi. If you haven’t installed OpenCV on your Raspberry Pi, you can refer to this tutorial: https://goo.gl/ARPdYa. I demonstrate how to optimize the Raspberry Pi + OpenCV installation for speed (which can achieve a 30% performance boost).

Note: This tutorial is not suitable for Python 3—you should use Python 2.7. I will explain why later. Now go ahead and configure your Raspberry Pi with Python 2.7 and OpenCV. In step 4 of the Raspberry Pi + OpenCV installation guide, make sure to replace -p python2 to create a virtual environment.

Now, I recommend increasing the swap space on the Raspberry Pi. Increasing the swap allows you to use the Raspberry Pi SD card to increase memory (this step is crucial when you want to compile and install large libraries on a memory-limited Raspberry Pi).

To increase your swap space, first open /etc/dphys-swapfile, then edit the CONF_SWAPSIZE variable:

# set size to absolute value, leaving empty (default) then uses computed value
#   you most likely don't want this, unless you have a special disk situation
# CONF_SWAPSIZE=100
CONF_SWAPSIZE=1024

Note here that I increased the swap from 100MB to 1024MB.

Now, restart the swap service:

$ sudo /etc/init.d/dphys-swapfile stop
$ sudo /etc/init.d/dphys-swapfile start

Note: Increasing the size of swap can damage your memory card, so be sure to revert this change and restart the swap service after you’re done. For information on how large swap can damage your memory card, see: https://goo.gl/ZpIuaI

Now that your swap space has been increased, it’s time to configure the development environment.

First, create a virtual environment named not_santa using Python 2.7 (I will explain the reason for using Python 2.7 when installing TensorFlow):

$ mkvirtualenv not_santa -p python2

Note that the -p points to python2, indicating that this virtual environment will use Python 2.7.

If you are unfamiliar with Python virtual environments and how they work and why we use them, please refer to these two links: https://goo.gl/nwoS38 and https://goo.gl/1vPU6b

You also need to ensure that you have symlinked your cv2.so package to your not_santa virtual environment (if you haven’t done this yet):

$ cd ~/.virtualenvs/not_santa/lib/python2.7/site-packages
$ ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so

Also, make sure you have compiled OpenCV with the Python 2.7 package. You also need to carefully check the path of your cv2.so file in case your installation path differs from mine.

If you have compiled Python 3 + OpenCV and created the symlink, then try importing cv2 into your Python shell, and you will receive a confusing traceback saying that the import failed.

Important note: For the next few pip commands, ensure you are in the not_santa environment (or your chosen Python environment), otherwise you will be installing these packages into your Raspberry Pi’s system Python.

To enter that environment, simply use the workon command in the bash prompt:

$ workon not_santa

Then you will see ‘(not_santa)’ at the start of your bash prompt.

Make sure you install NumPy in the not_santa environment using the following command:

$ pip install numpy

Since this project needs access to GPIO pins, we need to install RPi.GPIO and gpiozero:

$ sudo pip install RPi.GPIO gpiozero

Now install TensorFlow on your Raspberry Pi

Here’s the issue: there is no suitable official (Google-released) version of TensorFlow.

We can compile TensorFlow from scratch for Raspberry Pi, but this process is very lengthy, troublesome, and painful; see: https://goo.gl/a2hnzK

Or we can use precompiled binaries created by Sam Abrahams; GitHub address: https://goo.gl/WJzbL6

But the problem is that there are only two precompiled TensorFlow binaries:

1. For Python 2.7

2. For Python 3.4

And the Raspbian Stretch release (the latest version of Raspbian at the time of writing this article) uses Python 3.5—so there is a version mismatch issue here.

To avoid the hassle between Python 3.4 and Python 3.5, I decided to stick with Python 2.7.

Even though I would prefer to use Python 3 in this tutorial, doing so would complicate the installation process (and since the focus of this tutorial is not on installation, I decided to make this task a bit simpler).

Let’s use the following command to install TensorFlow using Python 2.7:

$ wget https://github.com/samjabrahams/tensorflow-on-raspberry-pi/releases/download/v1.1.0/tensorflow-1.1.0-cp27-none-linux_armv7l.whl
$ pip install tensorflow-1.1.0-cp27-none-linux_armv7l.whl

Note: The URL in the above code is quite long, so be sure to recognize it.

After compiling and installing TensorFlow (which took about an hour on my Raspberry Pi), you need to install HDF5 and h5py. These libraries allow us to load our previously trained model from disk:

$ sudo apt-get install libhdf5-serial-dev
$ pip install h5py

I didn’t run the time command when I installed HDF5 and h5py, so I don’t remember how long it took; I estimate it was about 30-45 minutes.

Finally, let’s install Keras and other packages needed for this project:

$ pip install pillow imutils
$ pip install scipy --no-cache-dir
$ pip install keras

Installing SciPy takes a few hours, so you can install it while working or overnight.

To test your setup, you can open the Python shell in the not_santa environment and execute the following commands:

$ workon not_santa
$ python
>>> import h5py
>>> from gpiozero import LEDBoard
>>> from gpiozero.tools import random_values
>>> import cv2
>>> import imutils
>>> import keras
Using TensorFlow backend.
>>> print("OpenCV version: {}".format(cv2.__version__))
'3.3.1'
>>> print("Keras version: {}".format(keras.__version__))
'2.0.8'
>>> exit()

If everything goes smoothly, you should see Keras being imported with the TensorFlow backend.

As demonstrated in the output above, you should check again if you can import your OpenCV package (cv2).

Finally, don’t forget to change your swap size back from 1024MB to 100MB, the steps are:

1. Open /etc/dphys-swapfile

2. Reset CONF_SWAPSIZE to 100MB

3. Restart the swap service (just like we mentioned earlier).

As mentioned earlier, changing the swap size back to 100MB is beneficial for the lifespan of the memory card. If you skip this step, you may encounter memory damage issues, and the lifespan of the card will shorten.

Running Keras Deep Learning Models on Raspberry Pi

Figure 5: Running deep learning models on Raspberry Pi using Keras and Python

Now we can start coding the Not Santa detector using Keras, TensorFlow, and Raspberry Pi.

Again, I will assume that your hardware setup is similar to mine (i.e., 3D Christmas tree and speakers), so if your setup is different, you will need to write or modify the code below yourself.

First, create a new file named not_santa_detector.py, and paste in the following code:

Lines 2-12 of the code handle our imports. It’s worth noting that:

• Keras is used for preprocessing the input frames for classification and for loading the trained model from disk.

• gpiozero is used to access the 3D Christmas tree.

• imutils is used to access the video stream (whether it’s the Raspberry Pi camera module or USB).

• threading is used for non-blocking operations, especially when we need to light up the Christmas tree or play music without blocking the execution of the main thread.

To do this, we will define a function to light up the 3D Christmas tree:

Our light_tree function takes a tree parameter (which should be an LEDBoard object).

First, we loop through all the LEDs in the tree and randomly light each LED to create a blinking effect (lines 17-19).

We keep the lights on for a while (line 23), then loop through these LEDs again, this time turning them off (lines 26-28).

The effect of lighting up the 3D Christmas tree looks like this:

Figure 6: A 3D Christmas tree controlled by Raspberry Pi

Our next function will play music when Santa Claus is detected:

In the play_christmas_music function, we make a system call to the aplay command, allowing us to play an audio file using the command line.

Using os.system calls requires some work, but playing audio files through pure Python (using a library like Pygame) would be overkill for this project.

So let’s hardcode the configuration we will use:

Lines 38 and 39 hardcode the paths to our trained Keras model and audio file.

We also initialize parameters for detection, including TOTAL_CONSEC and TOTAL_THRESH. These two values indicate the number of frames containing Santa Claus and the threshold at which we play music and turn on the Christmas tree lights (lines 43 and 44).

The final initialization is SANTA = False, which is a boolean value (line 47). We will later use the SANTA variable as a status flag in our logic.

Next, we load our previously trained Keras model and initialize our Christmas tree:

Keras allows us to save models for later use. In the tutorial at https://goo.gl/imxkrY, we saved the Not Santa model to disk, and now we will load it onto our Raspberry Pi. We use Keras’s load_model function on line 51 to load the model.

Our tree object is instantiated on line 54. You can see that tree is an LEDBoard object from the gpiozero package.

Now we initialize our video stream:

To access the camera, we will use the VideoStream from the imutils package (line 58 or 59); refer to the documentation: https://goo.gl/NEZrGK

Important note: If you are using the PiCamera module (instead of a USB webcam), then just comment out line 58 and uncomment line 59.

We sleep for 2 seconds so that we can warm up our camera between frame loops (line 60):

On line 63, we start looping through video frames until the stopping condition is met (provided later in the script).

First, we grab a frame by calling vs.read (line 66).

Then we resize the frame to width=400 while maintaining the original aspect ratio (line 67). We will send the frame to our neural network model after preprocessing it. After that, we will display the frame with a text label on the screen.

So let’s preprocess the image and pass it to our Keras deep learning model for prediction:

Lines 70-73 preprocess the image to prepare it for classification. For more information on deep learning preprocessing, refer to my latest book, “Deep Learning for Computer Vision with Python”: https://goo.gl/jTYng4

Then we use the image as a parameter to query model.predict. This sends the image to the neural network and returns a tuple containing class probabilities (line 77).

We initialize the label as “Not Santa” (we will access the label later) and initialize its probability proba as the value for notSanta (lines 78 and 79).

Let’s check if there is Santa Claus in the image:

Line 83 checks if the probability of Santa is greater than that of notSanta. If it is indeed greater, we continue and update the label and proba, then increment TOTAL_CONSEC (lines 85-90).

When enough consecutive frames with Santa Claus pass through, we need to trigger the Santa Claus alarm:

If SANTA is False and if TOTAL_CONSEC reaches TOTAL_THRESH, two actions will be performed:

1. Create and start a treeThread to blink the Christmas tree lights (lines 98-100);

2. Create and start a musicThread to play background music (lines 103-106).

These threads will run independently and will not block the forward execution of the script (i.e., non-blocking operations).

You can also see that on line 95, we set our SANTA status flag to True, indicating that we found Santa Claus in that input frame. In the next iteration of the loop, we will check this value, as shown in line 93.

Otherwise (if SANTA is True or TOTAL_CONSEC has not yet reached TOTAL_THRESH), we will reset TOTAL_CONSEC to zero and reset SANTA to False:

Finally, we display the frame with the generated text label on our screen:

Probability values are also attached to the label with “Santa” or “Not Santa” (line 115).

Then we can use OpenCV’s cv2.putText to write the label on the frame (in a Christmas-themed green), and then we will display the frame on the screen (lines 116-120).

The exit condition for our infinite while loop is pressing the “q” key on the keyboard (lines 121-125).

If the exit condition for the loop is met, we break and perform some cleanup before exiting the script itself (lines 129-130).

That’s all there is to it. You should review these 130 lines of code as a whole—this framework/template can also be easily used for other deep learning projects on Raspberry Pi.

Now let’s go find that jolly, plump Santa Claus!

Results from Deep Learning + Keras + Raspberry Pi

When developing the Not Santa deep learning model earlier, we used images collected from the internet.

But that was boring—and didn’t align with the pursuit of this article.

I have always wanted to dress up as Santa Claus, so I ordered a cheap Santa suit last week:

Figure 7: My Santa Claus disguise, Adrian Rosebrock. I will personally test the Not Santa detector we created using deep learning, Keras, Python, and OpenCV. My appearance is quite different from the real Santa Claus, but this costume should suffice.

Then I mounted the camera connected to the Raspberry Pi on my Christmas tree in my apartment:

Figure 8: My own Christmas tree will serve as the testing background for our Raspberry Pi Not Santa detector deep learning model.

If Santa Claus comes to this Christmas tree to deliver gifts to good children, we will welcome him by blinking the 3D Christmas tree lights and playing Christmas tunes.

Then I started the Not Santa deep learning + Keras detector with the following command:

$ python not_santa_detector.py

If you want to follow this tutorial, you can download the source code, pre-trained model, and audio files in the download section at the end of the original text.

Once Not Santa is up and running, I got into action:

Figure 9: Successfully detecting Santa Claus in the video stream using deep learning, Python, Keras, and Raspberry Pi

When Santa Claus is detected, the 3D Christmas tree lights illuminate, and music begins to play.

This is the link to the demonstration video with sound: https://www.youtube.com/watch?v=RdK-8pSQIP0

Every time Santa Claus comes into view, you can see the 3D Christmas tree lights turn on, accompanied by the cheerful laughter from the Raspberry Pi speakers.

Our deep learning model is a small network architecture, but it performs surprisingly well in terms of accuracy and robustness.

I’ve been good all year long, so I believe Santa Claus will come to my apartment.

I am also more confident than ever that my Not Santa detector will see Santa Claus delivering gifts to me.

Before Christmas, I might modify this script (calling cv2.imwrite or better yet saving video) to ensure I capture evidence of Santa Claus on disk. If someone else leaves gifts under my Christmas tree, I will definitely know.

Dear Santa Claus: If you read this article, you will know that I caught you with my Raspberry Pi!

Conclusion

In this article, you learned how to run Keras deep learning models on Raspberry Pi.

To achieve this, we first trained a Keras deep learning model on a laptop or desktop computer that can detect whether there is ‘Santa’ or ‘Not Santa’ in the images.

Then we installed TensorFlow and Keras on our Raspberry Pi, allowing us to deploy the previously trained deep learning image classifier onto this Raspberry Pi. Although Raspberry Pi is not suitable for training deep neural networks, it can be used to deploy these networks—as long as the network architecture is simple enough, we can even run our models in real-time.

To demonstrate this, we created a Not Santa detector on Raspberry Pi that can classify each input frame in the video stream.

If Santa Claus is detected, we access the GPIO pins to light up the 3D Christmas tree and play festive music.

Original link: https://www.pyimagesearch.com/2017/12/18/keras-deep-learning-raspberry-pi/

This article was translated by Machine Heart, please contact this public account for authorization to reprint.

✄————————————————

Join Machine Heart (full-time reporter/intern): [email protected]

Submissions or requests for reports: [email protected]

Advertising & business cooperation: [email protected]