Detecting Santa Claus: A Deep Learning AI Based on Raspberry Pi and Keras

Happy Christmas to all MAKERS!

The popular TV show “Silicon Valley” features a character who created a “Not Hotdog” app that can determine whether an input photo is a “hotdog” or “not hotdog”. Today, we will introduce how to run a deep neural network that recognizes Santa Claus on Raspberry Pi using Keras, with just 130 lines of code, successfully lighting up the Christmas tree and playing Christmas songs when Santa comes to your house! Let’s give it a try~ This article is a complete guide to running deep neural networks on Raspberry Pi using Keras. A video demonstration is included at the end!

I consider this project as a “Not Santa” detector, teaching you how to implement it practically (and it’s a lot of fun in the process).

In the first part, we will discuss what a “Santa Detector” is (you may not be familiar with the “Not Hotdog” recognition app from the hit show “Silicon Valley”, but it has already been implemented).

Then, we will configure the Raspberry Pi for deep learning by installing TensorFlow, Keras, and some other dependencies.

Once the Raspberry Pi is set up for deep learning, we will continue to build a Python script that can:

• Load the Keras model from disk

• Access the Raspberry Pi camera module/USB webcam

• Apply deep learning to detect whether Santa Claus is in the frame

• If Santa is detected, access the GPIO pin and play music

So let’s get started!

What is the Not Santa detector?

Figure 1: The “Not Hotdog” recognition app from HBO’s “Silicon Valley”

The “Not Santa” detector is inspired by HBO’s “Silicon Valley”. The characters in the show created an app that can determine whether an input photo is a “hotdog” or “not hotdog”. The show clearly pokes fun at the startup culture in Silicon Valley, USA:

• Hype around machine learning and deep learning

• Satire on the numerous mobile apps that are utterly useless (but their inventors believe their apps will “change the world”).

Today, we decided to make a “Not Santa” detector that can detect whether Santa Claus is in an image/video frame.

For those unfamiliar with Santa Claus, he is a jolly, plump, bearded fictional character from Western culture who delivers gifts to children on Christmas Eve.

However, this app is not just for fun and satire. We will learn some practical skills, including:

• Configuring Raspberry Pi for deep learning

• Installing Keras and TensorFlow on Raspberry Pi

• Deploying a pre-trained convolutional neural network on your Raspberry Pi

• Executing a given action once detection occurs

Hardware Configuration

But before we write any code, let’s talk about the hardware we need.

Figure 2: The setup for the “Not Santa” detector includes Raspberry Pi 3, speakers, a 3D Christmas tree, and a webcam. To detect Santa Claus, the Raspberry Pi implements LeNet in Python script using Keras.

To follow this tutorial, you will need:

• Raspberry Pi 3 (I highly recommend the Raspberry Pi 3 starter kit)

• A Raspberry Pi camera module or a USB camera.

• A 3D Christmas tree suitable for Raspberry Pi

• A set of speakers

Of course, these are not all mandatory. As long as you have a Raspberry Pi + camera module/USB webcam, you can set everything up (but you will need to modify the code so it won’t try to access GPIO pins or play music through the speakers).

Your setup should be similar to the one in Figure 2 above, connecting the speakers, 3D Christmas tree, and webcam together. I also recommend using an HDMI monitor + keyboard to test and debug the script:

Figure 3: My deep learning setup includes Raspberry Pi and components, along with a keyboard, mouse, and a small HDMI display. Once set up, I can catch Santa when he comes in front of my Christmas tree to deliver gifts.

How to install TensorFlow and Keras on Raspberry Pi?

For how to use Keras to train a convolutional neural network to determine whether Santa Claus is present in the input image, you can refer to [1]:

We will use a pre-trained model and deploy it on the Raspberry Pi. As I mentioned earlier, the Raspberry Pi is not suitable for training neural networks. However, the Raspberry Pi can deploy trained neural networks (of course, the model needs to fit into a very small memory footprint).

I assume you have already installed OpenCV on the Raspberry Pi. If not, you can check the tutorial in [2].

It is advisable to increase the swap space on the Raspberry Pi, which will allow you to use the Raspberry Pi SD card to increase memory (this is a key step when trying to compile and install large libraries on the memory-limited Raspberry Pi).

To increase the swap space, open /etc/dphys-swapfile and edit the CONF_SWAPSIZE variable:

# set size to absolute value, leaving empty (default) then uses computed value
#   you most likely don't want this, unless you have a special disk situation
# CONF_SWAPSIZE=100
CONF_SWAPSIZE=1024

I increased the swap space from 100MB to 1024MB. Then, restart the swap server:

$ sudo /etc/init.d/dphys-swapfile stop
$ sudo /etc/init.d/dphys-swapfile start

Note: Increasing swap space can easily burn out the storage card, so make sure to revert this change and restart the swap server after you are done.

Then, we start configuring the development environment.

First, create a Python virtual environment named not_santa using Python 2.7:

$ mkvirtualenv not_santa -p python2

Note that the -p switch points to python2, indicating that Python 2.7 will be used for the virtual environment.

Also, ensure that you have bound cv2.so to the not_santa virtual environment:

$ cd ~/.virtualenvs/not_santa/lib/python2.7/site-packages
$ ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so

Similarly, confirm that OpenCV has been bound to Python 2.7 and carefully check the path of the cv2.so file in case the installation path differs from my demonstration.

If you compiled Python 3 + OpenCV bindings, created a sym-link, and tried to import cv2 into your Python shell, you would get a confusing traceback saying the import failed.

Important Note: For the next few pip commands, make sure you are in the not_santa environment, otherwise, you will install these packages into the Raspberry Pi’s system Python.

To enter the environment, just use the workon command at the bash prompt:

$ workon not_santa

Then, you will see “(not_santa)” at the beginning of the bash prompt.

Make sure to install NumPy in the not_santa environment using the following command:

$ pip install numpy

Since we will access the GPIO pins for this project, we also need to install RPi.GPIO and gpiozero:

$ sudo pip install RPi.GPIO gpiozero

Now, install TensorFlow on the Raspberry Pi. The problem is that there is no official (Google-released) TensorFlow distribution, so we will have to compile TensorFlow from scratch on the Raspberry Pi, refer to [3].

Or we can use pre-compiled binaries from Sam Abrahams (available on GitHub [4]).

The problem is there are only two types of pre-compiled TensorFlow binaries, one for Python 2.7 and another for Python 3.4.

The Raspbian Stretch distribution comes with Python 3.5, so our versions do not match. To avoid issues between Python 3.4 and Python 3.5, I decided to stick with Python 2.7 installation.

Let’s continue by installing TensorFlow for Python 2.7 using the following commands:

$ wget https://github.com/samjabrahams/tensorflow-on-raspberry-pi/releases/download/v1.1.0/tensorflow-1.1.0-cp27-none-linux_armv7l.whl
$ pip install tensorflow-1.1.0-cp27-none-linux_armv7l.whl

Once TensorFlow is compiled and installed (it took me about an hour), we need to install HDF5 and H5py. These libraries will allow us to load the pre-trained model from disk:

$ sudo apt-get install libhdf5-serial-dev
$ pip install h5py

Finally, let’s install Keras and other dependencies required for this project:

$ pip install pillow imutils
$ pip install scipy --no-cache-dir
$ pip install keras

To test your configuration, open a Python shell (in the not_santa environment) and execute the following commands:

$ workon not_santa
$ python
>>> import h5py
>>> from gpiozero import LEDBoard
>>> from gpiozero.tools import random_values
>>> import cv2
>>> import imutils
>>> import keras
Using TensorFlow backend.
>>> print("OpenCV version: {}".format(cv2.__version__))
'3.3.1'
>>> print("Keras version: {}".format(keras.__version__))
'2.0.8'
>>> exit()

If everything goes according to plan, you should see Keras imported with the TensorFlow backend.

As indicated in the output above, you should also carefully check if the OpenCV binding (cv2) can be imported.

Finally, don’t forget to reduce the swap space from 1024MB back to 100MB by:

• Opening /etc/dphys-swapfile

• Resetting CONF_SWAPSIZE to 100MB.

• Restarting the swap server

Running Keras + Deep Learning Model on Raspberry Pi

Now we are ready to write a Not Santa detector using Keras, TensorFlow, and Raspberry Pi. Again, I will assume your hardware setup is the same as mine; if not, you will need to modify the code below.

First, open a new file, name it not_santa_detector.py, and insert the following code:

# import the necessary packages
from keras.preprocessing.image import img_to_array
from keras.models import load_model
from gpiozero import LEDBoard
from gpiozero.tools import random_values
from imutils.video import VideoStream
from threading import Thread
import numpy as np
import imutils
import time
import cv2
import os

Lines 2-12 handle the input, specifically:

• keras: Used for preprocessing input frames for classification and loading pre-trained models from disk.

• gpiozero: Used to access the 3D Christmas tree.

• imutils: Used to access the video stream (whether it’s the Raspberry Pi camera module or USB).

• threading: Used for non-blocking operations, especially when we want to light up the Christmas tree or play music without blocking the execution of the main thread.

Next, define a function to light up the 3D Christmas tree:

def light_tree(tree, sleep=5):
	# loop over all LEDs in the tree and randomly blink them with
	# varying intensities
	for led in tree:
		led.source_delay = 0.1
		led.source = random_values()

	# sleep for a bit to let the tree show its Christmas spirit for
	# santa clause
	time.sleep(sleep)

	# loop over the LEDs again, this time turning them off
	for led in tree:
		led.source = None
		led.value = 0

The light_tree function takes a tree parameter (set as a LEDBoard object).

First, we loop through all the LEDs in the tree, randomly lighting each LED to create a “blinking” effect (lines 17-19).

We let the lights shine for a while (line 23), then loop through the LEDs again and turn them off (lines 26-28).

Below is an example of turning on the 3D Christmas tree lights:

Figure 6: 3D Christmas tree for Raspberry Pi

When Santa is detected, the next step is to play music

def play_christmas_music(p):
	# construct the command to play the music, then execute the
	# command
	command = "aplay -q {}".format(p)
	os.system(command)

In the play_christmas_music function, we perform a system call to the aplay command, allowing us to play music files just like from the command line.

Then, let’s hard-code the configuration we will use:

# define the paths to the Not Santa Keras deep learning model and
# audio file
MODEL_PATH = "santa_not_santa.model"
AUDIO_PATH = "jolly_laugh.wav"

# initialize the total number of frames that *consecutively* contain
# santa along with threshold required to trigger the santa alarm
TOTAL_CONSEC = 0
TOTAL_THRESH = 20

# initialize is the santa alarm has been triggered
SANTA = False

Lines 38 and 39 hard-code the paths to the pre-trained Keras model and audio file. You can download the audio file at the end of this article.

Initialize the parameters for detection, including TOTAL_CONSEC and TOTAL_THRESH. These two values represent the number of frames containing Santa and the threshold at which we will play music and light up the tree (lines 43 and 44).

The final initialization is SANTA = False, a boolean (line 47). We will later use the SANTA variable as a status flag in the script.

Next, load the pre-trained Keras model and initialize the Christmas tree:

# load the model
print("[INFO] loading model...")
model = load_model(MODEL_PATH)

# initialize the christmas tree
tree = LEDBoard(*range(2, 28), pwm=True)

Keras allows us to save the model to disk for future use. The Not Santa model has already been saved to disk [1], so we load it onto the Raspberry Pi. Line 51 loads the model using Keras’s load_model function.

Line 54 instantiates the tree object. As shown, tree is a LEDBoard object from the gpiozero package.

Then initialize the video stream:

# initialize the video stream and allow the camera sensor to warm up
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
# vs = VideoStream(usePiCamera=True).start()
time.sleep(2.0)

To access the camera, use VideoStream from the imutils package

Important Note: If you want to use the PiCamera module in this project (instead of the USB camera), just comment out line 58 and uncomment line 59.

Sleep for two seconds to allow the camera to warm up (line 60), then start looping through the frames:

# loop over the frames from the video stream
while True:
	# grab the frame from the threaded video stream and resize it
	# to have a maximum width of 400 pixels
	frame = vs.read()
	frame = imutils.resize(frame, width=400)

Line 63 starts looping through the video frames until a stop condition is met (which will be displayed later in the script).

First, get a frame by calling vs.read (line 66).

Then resize the frame to width = 400 while maintaining the aspect ratio (line 67). Preprocess this frame before feeding it into the neural network model. Later, we will display the frame along with the text label.

Next, preprocess the image and make predictions using Keras + deep learning model:

	# prepare the image to be classified by our deep learning network
	image = cv2.resize(frame, (28, 28))
	image = image.astype("float") / 255.0
	image = img_to_array(image)
	image = np.expand_dims(image, axis=0)

	# classify the input image and initialize the label and
	# probability of the prediction
	(notSanta, santa) = model.predict(image)[0]
	label = "Not Santa"
	proba = notSanta

Lines 70-73 preprocess the image and prepare it for classification. Then, we query model.predict with image as a parameter. This sends the image to the neural network and returns a tuple containing the class probabilities (line 77).

We initialize the label as “Not Santa” and the probability proba as the value of notSanta in lines 78 and 79.

Let’s check if Santa is in the image:

	# check to see if santa was detected using our convolutional
	# neural network
	if santa > notSanta:
		# update the label and prediction probability
		label = "Santa"
		proba = santa

		# increment the total number of consecutive frames that
		# contain santa
		TOTAL_CONSEC += 1

In line 83, we check if the probability of Santa is greater than notSanta. If it is, we continue to update the label and proba, then increment TOTAL_CONSEC (lines 85-90).

If we have provided enough consecutive “Santa” frames, we need to trigger the Santa alarm:

		# check to see if we should raise the santa alarm
		if not SANTA and TOTAL_CONSEC >= TOTAL_THRESH:
			# indicate that santa has been found
			SANTA = True

			# light up the christmas tree
			treeThread = Thread(target=light_tree, args=(tree,))
			treeThread.daemon = True
			treeThread.start()

			# play some christmas tunes
			musicThread = Thread(target=play_christmas_music,
				args=(AUDIO_PATH,))
			musicThread.daemon = False
			musicThread.start()

If SANTA is False and TOTAL_CONSEC reaches the TOTAL_THRESH threshold, we will execute two operations:

• Create and start a treeThread to blink the Christmas tree lights (lines 98-100).

• Create and start a musicThread to play music in the background (lines 103-106).

These threads will run independently without interrupting the forward execution of the script (i.e., non-blocking operations).

In line 95, we set our SANTA status flag to True, meaning we have found Santa in the input frame. In the next loop of the iteration, we will check this value as in line 93.

Otherwise (if SANTA is True or TOTAL_THRESH has not been met), we reset TOTAL_CONSEC to zero and SANTA to False:

	# otherwise, reset the total number of consecutive frames and the
	# santa alarm
	else:
		TOTAL_CONSEC = 0
		SANTA = False

Finally, we display the frame on the screen with the generated text label:

	# build the label and draw it on the frame
	label = "{}: {:.2f}%".format(label, proba * 100)
	frame = cv2.putText(frame, label, (10, 25),
		cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF
 
	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
print("[INFO] cleaning up...")
cv2.destroyAllWindows()
vs.stop()

The probability value is appended to the label containing either “Santa” or “Not Santa” (line 115).

Then, using OpenCV’s cv2.putText, we can display the label on top of the frame (in Christmas-themed green), and then display the frame on the screen (lines 116-120).

The exit condition for the infinite while loop is pressing the “q” key on the keyboard (lines 121-125). If the exit condition for the loop is met, we break and perform some cleanup as in lines 129 and 130 before exiting the script.

Done!

Looking back at this 130 lines of code, this framework/template can also be easily adapted for other deep learning projects on Raspberry Pi.

Now, let’s catch that jolly, fat, bearded Santa Claus!

Deep Learning + Keras + Raspberry Pi Results

Figure 7: Me, Adrian Rosebrock, dressed as Santa Claus. I will personally test the “Not Santa” detector built with deep learning, Keras, Python, and OpenCV.

Then, I pointed the camera towards the Raspberry Pi on the Christmas tree in the living room:

Figure 8: The Christmas tree will serve as the background for testing the Not Santa deep learning model already deployed on the Raspberry Pi.

If Santa comes to deliver gifts to my good children, I want to ensure he is welcomed with the blinking 3D Christmas tree lights and Christmas songs playing.

Then, I launched the Not Santa deep learning + Keras detector with the following command:

$ python not_santa_detector.py

Once the Not Santa detector was up and running, I got ready:

Dear Santa: If you read this, know that I will find you with Raspberry Pi!

References: [1] https://www.pyimagesearch.com/2017/12/11/image-classification-with-keras-and-deep-learning [2] https://www.pyimagesearch.com/2017/10/09/optimizing-opencv-on-the-raspberry-pi [3] https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md [4] https://github.com/samjabrahams/tensorflow-on-raspberry-pi

Original article: https://www.pyimagesearch.com/2017/12/18/keras-deep-learning-raspberry-pi/ Author: Adrian Rosebrock Translation: Rudolf