DIY Raspberry Pi AR Glasses with Custom Gesture Control

Source: Quantum Bit

ID: QbitAI

With stylish gestures, control the electronic imaging projected in front of you, isn’t this the basic configuration from sci-fi movies?

Now, someone has brought it from science fiction movies into reality. Move your fingers to gain control over the world in front of you.

Teemu Laurila, a blogger enthusiastic about making smart gadgets, has DIYed a pair of AR glasses that can recognize custom gestures using a Raspberry Pi.

By recording the gestures you want to set, you can achieve cool operations.

I have a bold idea!

The World of DIY AR Glasses

Let’s start performing!

Pinch your fingers up and down to complete the brightness adjustment command. (This is your first-person perspective)

Gesture recognition is overlaid and displayed in the lens imaging.

Let’s take a more direct perspective and see the effect through the glasses.

DIY Process

The AR glasses themselves are full of a sense of technology, making the real world full of a cyberpunk flavor.

Why not make it even cooler? With a snap of your fingers, commands can be executed, isn’t that appealing?

Let’s get started, first we need to design what parts the device will contain.

In addition to the main glasses frame, the hardware part also includes a lens group, 0.6 mm PETG projection lens, and the accessories are made from polylactic acid material through 3D printing.

After all, it’s a DIY smart device, how could we not invite the versatile mini-computer Raspberry Pi to the scene?

As for the software part, the gesture recognition program relies on the python open-source project MediaPipe.

In addition, Teemu Laurila also wrote two program scripts.

One is an application example that controls brightness through finger pinching, and the other captures gestures in real-time video and transmits them to the computer for processing, overlaying the display through smart glasses.

All conditions are ready, so let’s assemble and try it out.

After several adjustments, the various parts were finally combined into the following device.

To make the program usable on the device, first, a Raspberry Pi is needed for program support.

Then set up memory, drivers, runtime environment, multimedia interfaces, networks, and other conditions to allow the entire device to run at high speed.

Once the hardware and software environment are ready, debug the application program.

The core functionality of the application program—the gesture recognition model consists of three frameworks, including the hand palm recognition model BlazePalm (for recognizing the overall framework and direction of the hand), the Landmark model (recognizing the three-dimensional hand node), and the gesture recognition model (classifying the recognized nodes into a series of gestures).

During the training process of the recognition algorithm, the BlazePalm model recognizes the initial position of the palm, optimizing real-time recognition for mobile devices.

Within the range of the palm recognized by BlazePalm, the Landmark model identifies the coordinates of 21 three-dimensional nodes.

The gesture recognition model then identifies the state of each finger based on the joint angles and maps the state to predefined gestures, predicting basic static gestures.

Through the Raspberry Pi Zero W, gesture information is captured. The image information is transmitted to the computer, processed by the gesture recognition AI, and then conveyed to the device, issuing corresponding gesture commands and synchronizing them in the projected image.

Its Past and Present

Wait a minute, there’s a camera, a mini projector, a computer processor, and a side projection display. It seems like I’ve seen such AR glasses somewhere.

That’s right, even the gesture recognition code used is open-sourced by Google.

Although it does not have the smartphone-like functions of Google Glass, compared to its voice control and touch functions, Teemu Laurila’s smart glasses chose to use custom gestures to trigger commands, adding a touch of black technology.

Additionally, while the camera of Google Glass is only used for taking photos and videos, Teemu Laurila’s camera also undertakes the function of receiving gesture commands and conveying instructions. At the same time, the projection also chose larger square lenses for easier observation of the field of view.

This device is already the second version of the smart glasses completed by Teemu Laurila, with optimizations in appearance and performance.

In terms of material selection, a 0.6mm thickness projection lens replaced the 1mm thickness; polylactic acid material replaced acrylic; and a bolt-fixed bracket was added, abandoning glue.

The most important optimization is that the camera uses a square lens for clearer images.

Teemu Laurila shared the two additional pieces of code he created on the GitHub platform for interested viewers to replicate.

Reference links:

https://www.tomshardware.com/news/raspberry-pi-smart-glasses-recognize-hand-gestures

https://www.youtube.com/watch?v=60Os5Iqdbsw

https://www.youtube.com/watch?v=Gu4oOYo38rQ

GitHub links:

https://github.com/Teneppa/CameraStream

https://github.com/Teneppa/HandTrackingBrightnessControl