Yesterday, I shared the design plan for the smart assistant ‘JARVIS’. I wonder what everyone thinks after reading it? If this plan were actually implemented, what would the effect be?

Today, I am excited to present the project showcase, let’s see how to create our own smart voice assistant!

Project Introduction

This project uses a PC as the main computing platform (the program is universal and can run directly on Raspberry Pi and other Linux boards, achieving wearability), using ESP32-S2 to create a network camera module as the hardware for the entire system’s visual input. The goal is to design an ESP32-S2 visual development board with a built-in camera and display, and certain expandability.

Solution Block Diagram and Schematic Introduction

Project Schematic

PCB Drawing and Board Production

As a portable development module, to minimize the size, the circuit board is designed to be very compact, which brings considerable difficulty to wiring, especially for the onboard camera part, which has many lines, and to maintain signal quality, the routing cannot repeatedly cross holes to change layers. The final product is shown below, all wiring has only a single layer change, and the USB differential signal lines have been length-matched.

The camera position design supports dual-direction use of the camera. By default, the camera is folded over and attached to the board, so the camera and LCD are on the same side, equivalent to a front-facing camera; if you want to use the rear camera mode, just insert the camera directly as shown in the figure above.

Board Physical Diagram

The prototype circuit board is shown in the figure:

Since there are components on both sides of the circuit board, this presents some challenges for soldering. The ESP32-S2-MINI module used requires hot soldering for assembly. Therefore, the board is divided into two; the ESP32-S2-MINI module is soldered on the front with a hot soldering station, while the back is soldered with a soldering iron; other areas are soldered with a hot soldering station on the back and a soldering iron on the front.

The physical diagram of the board after component soldering

Back of the Circuit Board

Front of the Circuit Board

Code Design

For the ESP32-S2 part, I used CircuitPython to perform a simple test to quickly check whether each part of the hardware works normally.

The code uses the Arduino framework. According to the official tutorial, I downloaded the ESP32 development board package in Arduino, and after placing the order, I opened the official CameraWebServer example and modified the example code.

PC-side code. Image acquisition is completed using OpenCV, and object recognition is done using ImageAI. ImageAI can use the three models: ResNet50, YOLOv3, and Tiny-YOLOv3. Here, I encapsulated the image acquisition and recognition into an independent class that can be directly imported as an external library.

For the voice input part, the voice input uses the speech_recognition library, and recognition uses Whisper. At the same time, once the voice recognition starts, it indicates that we have asked a question, so image recognition begins simultaneously.

The project has been open-sourced on the Electronic Forest project website, and you can click on “Read Original” to view the complete project report for specific code details.

Function Demonstration

Flashlight supplement demonstration

Object recognition

The complete demonstration process can be referenced in the project video.

Reflections

This project is one of the more complex ones I have done. The hardware part is more complex compared to conventional development board designs. Due to the presence of a camera, achieving a compact design makes wiring quite difficult; while the software part integrates voice recognition, ChatGPT, OpenCV, CNN, and TTS to work collaboratively, which presents certain challenges. After completing this project, I feel that my abilities have greatly improved, and I have gained a better understanding of the implementation process for many features.

To learn more about the project details, you really have to look at the project report to appreciate the intricacies involved. This project has been open-sourced and displayed on the Electronic Forest, what are you waiting for? Hurry up and click on the “Read Original” at the end to explore!

Completing the task can also make you an event ambassador. Hurry up and invite like-minded friends to participate in the event and earn promotional rewards!

Image source: Internet, if there is any infringement, please contact for deletion

Hardhe Academy

Let’s explore and advance together in the field of electronics

Follow the Hardhe public account for direct access to the classroom anytime

Click to read the original text to view the complete project