MAKER:David Packman／Translated by: Qu Wujin

This is the MBO robot based on Raspberry Pi created by maker David Packman. The design of this robot is inspired by the character MBO from the animated series Adventure Time.

It features powerful interactive capabilities, including offline wake word detection, chatting with ChatGPT 3.5, and machine vision for image analysis and description.

Please see the demo below.

Component List

Hardware

Raspberry Pi 3 Model B × 1, Raspberry Pi Camera Model 3 × 1, Mono 2.5W Audio Amplifier × 1, 8Ω 1W Speaker × 1, Mini USB Microphone × 1, 16mm Touch Module × 1, 18650 Battery × 1, Toggle Switch × 1, 5V Fan × 1, 9g Servo × 1, 5V Voltage Regulator Module × 1, USB Wireless Keyboard × 1, 3D Printer × 1, 3D Printing Pen × 1, Hot Glue Gun × 1, Soldering Iron × 1, Electric Screwdriver × 1, Acrylic Paint × 1.Software Raspberry Pi Official System, Microsoft Azure Cognitive Speech Service API, Microsoft Azure Computer Vision, OpenAI Chat GPT 3.5 Turbo, OpenAI DALL-E 2.

Project Introduction

BMO-AI is a 3D printed mobile desktop companion robot, prototyped from the robot character BMO in the Adventure Time series. This robot employs several different AI functions to engage in social interactions in a more engaging and useful manner. BMO-AI utilizes a Raspberry Pi 3B+ running Raspbian Bullseye and an Adafruit CRICKIT Hat for servo motor control and signal input from its button array.

BMO has four joints and a 5-inch display for showing facial expressions and images using the Pygame library. BMO also uses several other libraries to implement the following AI functions: speech-to-text and text-to-speech, offline wake word detection using Azure Speech Service, single-turn Q&A text using OpenAI ChatGPT 3.5 Turbo, multi-turn and context-aware chatting completed with OpenAI ChatGPT 3.5 Turbo, image analysis and description using Azure Computer Vision Service. The Raspberry Pi Camera Module v3 has image capture and sharing capabilities. New drawing mode uses Stable Diffusion to draw images based on your verbal description. New DALL-E images from ChatGPT 3.5 description mode will create images based on descriptions generated by ChatGPT.

3D Printing Section

Print part files can be downloaded from the project file library: https://make.quwj.com/project/442 1. It is recommended to use PETG or other more flexible and temperature-resistant filaments than PLA in this version, especially if you are not installing a fan to cool the SBC and other components. 2. Except for the buttons, the models shown in these build instructions are printed with Polymaker Polylite teal PETG on Prusa MK2 and MK3 printers, with a layer height of .02. 3. The buttons are marked with the colors they should be printed in, PLA is suitable. 4. Regarding the arms, there are two separate STL versions for the arms, one version (ArmRtFull and ArmLtFull) is for single-part printing, and the other versions (ArmRT and ArmLt) are split at 7mm to reduce support blemishes and need to be glued together after printing.5. For the upper leg and lower leg, you need to print without support or use support actuators so that your printer does not print support inside the tunnel through the leg.

Otherwise, you will not be able to push the string through the leg for kicking actions.

Preparing Servos for Assembly

1. After testing your servos to ensure they work properly, you need to label each motor according to its joint point.

For example, use a permanent marker to label one as LA (Left Arm) and another as LL (Left Leg). (Left and right are from BMO’s perspective). 2. After labeling the servos, you should pre-position the servos so that the left side servo is at the farthest counterclockwise position (usually 0 degrees), and the right side servo is at the farthest clockwise position (usually 180 degrees).

This way, all servo systems start from a lowered position and have the range to lift arms or kick legs.3. The servo horns for the arm servos need to be modified so that you can install the shoulder bracket of the arm. Basically, you need to cut the horn depicted in the image below to fit the printed shoulder mount as shown in the image below.4. Once the servo horns for the arm are cut to allow them to slide into the shoulder bracket, you will need to hot glue or plastic weld them in place to secure them to the shoulder bracket as shown in the image above. 5. Now you can connect the shoulder mount to the arm servo, ensuring the arm’s mounting holes point towards the back of BMO as shown in the image.

At this point, you can screw it into the servo arm while holding the shoulder bracket, so the servo arm does not rotate from that pre-positioned angle.

Preparing the Button Panel

1. First, cut the through-hole prototype board as shown in the image below and create mounting holes using a 5/64 or 3/32 drill bit as shown.2. Loosely mount the board in position inside the body, then rotate the body and place a dot with a permanent marker at the center of each button position as shown. Then remove the board.3. Note that each pole of the switch has two pins, so try to align the switch so you can track which pins need to be soldered to connect the switch when pressed.

Then position each switch so that the button itself is centered on each point you created in the previous step.4. Now, you can solder the buttons and wires to the board to create a circuit where one pole on each button connects to a jumper wire.

This jumper will go to the CRICKIT HAT on the Raspberry Pi, while the other will connect to a common ground that will plug into the ground pin on the CRICKIT hat signal connector.

Preparing the Speaker

1. After soldering and testing the button board, you need to cut two smaller through-hole prototype boards, about 5 holes x 5 holes. 2. Then, solder the speaker leads and some 2-pin JST or Dupont connectors so that the leads connect to the connectors as shown in the image below.

These components will be hot glued to the side of the BMO body so that the sound from the speaker can be heard through the speaker hole in the body in the later assembly steps.

Body Installation

1. First, you may want to paint letters on the side before you start assembly, doing this now will be easier. 2. Insert the heat-set insert nuts into the servo motor mounting holes as shown in the image below:3. When using it, push some M2 hex nuts into the slots of the servo motor mounting bracket. You only need to connect the servo motor to the top mounting bracket, but you can also do a bottom mount if you really want to.

Bottom Plate Installation

1. Insert three M2 heat-set inserts into the back cover as shown in the image below.2. Insert square M3 nuts into the battery cover screw holes as shown in the image below, then hot glue them in place after securing with M3 screws.3. Use hot glue or a 3D pen to secure the 18650 battery holder, voltmeter, and toggle switch in place as shown in the image below.4. Install the 5V voltage regulator on the back cover. Solder the red lead from the battery holder to one terminal of the switch, then solder another jumper wire with a female Dupont connector that connects the regulator to another terminal.

Also, crimp the end of the voltmeter’s lead and the battery holder’s ground lead with a female Dupont connector.

5. Connect the voltmeter and battery leads, and switch to the VIN pin of the regulator. If needed, use hot glue to secure all leads to the back cover.

Preparing the Raspberry Pi and CRICKIT Hat

1. If you have already installed the 40-pin connector adapter for the CRICKIT Hat, please remove it.

If we want the button leads to fit without being crushed by the back panel, we need the hat to be installed on top of the Raspberry Pi at a very low profile.

Note: If you plan to install a heatsink on the Raspberry Pi, please use a very low-profile heatsink to avoid shorting the CRICKIT hat.

2. Connect the long CSI and DSI cables to the Raspberry Pi, then slide them over the CRICKIT hat, and loosely connect the CRICKIT hat to the Raspberry Pi 40-pin connector. Do not push the pins all the way in. 3. Grab one of the printed Raspberry Pi spacers and a 14mm M2.5 screw, then slide the spacer into the lower left position and push the screw in to secure the spacer in place. Use an M2.5 nut to secure that screw.

4. Perform the same operation for the other mounting holes, but do not secure them with nuts, as these holes will be screwed into the brackets in the display mount during assembly.

You may also want to crimp a 2-pin female Dupont connector onto one of the right angle barrel jack leads, and install the barrel jack socket onto the CRICKIT power socket at this time.

5. Important! Now is a good time to hold the components and look inside from various sides to ensure that none of the CRICKIT hat pins on the bottom of the circuit board are shorting against any components on top of the Raspberry Pi!

If you do find any pins touching anything, especially the heatsink or audio jack, please disassemble and try to trim those contact points with side cutters. 6. Set this component aside for later use, ensuring the screws and spacers stay in place.

Assembly Steps

1. First, place the body upside down and then insert all buttons into their respective slots as shown in the image.

2. Install the button board on top of the buttons as shown in the image.

3. Flip the body over and check to ensure each button activates its corresponding switch without playing too much. If you find that a button is activating any switch without being pressed.

You may need to re-solder that button to be flush with the circuit board. 4. Check the length of the button cables; they should reach the center of the display hole with some extra slack but not too much.

Installing Display, Speaker, RasPi Components and Camera

1. Install 3 M2.5 hex nuts at the bottom of the display bracket in the upper right and lower right corners as shown in the image.

2. Use 5mm M2.5 screws to secure the nuts in place.

3. Connect the small fan to the display bracket in the lower left corner using M2 screws as shown.

4. Place the display above the display hole inside the body, with the DSI cable connector on the left side. (Yes, this means the display will be upside down.) 5. Use M2.5 screws to install the bracket onto the display and use M3 and M2 screws to secure the bracket to the body, fixing the display with the upper right and lower right display brackets.

6. Without installing the RasPi components, connect the DSI/CSI cables to the display and camera module. 7. Install the camera module as shown in the image. Note how the M2 screw in the upper right corner is inserted from the front into the lower right bracket and secured with an M2 nut.

8. Install the speaker on either side of the body, ensuring the speaker covers the speaker hole, with the connector board trailing at an angle behind the speaker hole.

9. Install the RasPi components at the mounting points on the display bracket, ensuring the RasPi spacers are secured in place.

Note: It may be easier to connect any USB connectors before doing this, though I found it easier to insert them later to keep the cables from getting in the way.

10. Use hot glue to install the audio amplifier above the RasPi components on the inside top wall of the body as shown (wiring will be done later).

Assembling the Legs

1. Slide the thighs into the bottom of the body and try to angle them slightly outward or at least parallel to each other. Then plastic weld them in place at the appropriate positions under the body.

Note: It is best to angle them slightly so that the front of the legs rests against the front edge of the body while the back of them is further away, causing BMO to lean slightly forward when sitting.

2. Also, weld the legs inside the body to firmly secure them in place. Be careful to ensure the string tunnel opening remains clear while doing this.

3. Cut two 40cm long strings, preferably braided fishing line, and tie one end of each string to the horn of the leg servo motor with a double or triple knot.

You may need to enlarge one of the horn holes to fit the string, I suggest using the second hole closest to the end of the horn.

4. Slide the leg servo motor into place and secure it with M2 screws. Note that you may need to slightly drill the mounting holes of the servo to make the M2 fit. 5. Thread the string through the thigh tunnel and then through the calf tunnel until the string emerges from the bottom of the foot. Use M2 screws to connect the calf to the thigh, ensuring not to pinch the string. 6. Bend the leg to 90 degrees, positioning the servo motor so the horn is near the bottom, tie a knot on the string near the foot to ensure the leg stays bent, playing slightly on the string. 7. Plastic weld the hole in the foot to make the hole and string invisible.

Installing the Arms

1. After installing the shoulder bracket, slide the arm servo into place and secure it with M2 screws. Ensure the arm mounting holes point towards the back of BMO.

2. Insert the arms into the square shoulder socket and secure with M2 screws. Note: You may need to file or sand the square pegs of the arms a bit to ensure they slide smoothly into the shoulder but have almost no gap after being screwed in.

Wiring

1. Ensure the servo leads are connected to the CRICKIT hat servo pins in the following order (otherwise you will need to update the code): a port – left arm, b port – right arm, c port – left leg, d port – right leg 2. Connect all button signal wires to the CRICKIT signal ports and connect two common grounds to the ground pins. 3. Use JST connectors to connect two pairs of jumper wires for the speaker component leads and connect them to the audio amplifier output screw terminal. 4. Use jumper wires to connect the amplifier’s VIN and ground to the 5v and ground terminals on CRICKIT’s NEO-PIXEL. Additionally, when using the fan, connect the power and ground leads from the fan to the same terminal. 5. Create another pair of jumper wires, connecting them to the screw-in audio jack and the amplifier’s audio input, then plug the audio jack into the RasPi audio output. 6. Plug the USB keyboard jumper wire and right angle USB extension cable into the RasPi USB port. 7. Insert the USB microphone into the USB cable extension, then hot glue it down so that the microphone hole is visible through the small port below the buttons on BMO-AI as shown:

8. Plug the USB power line into the RasPi USB power port and plug the other end into the 5v output of the voltage regulator on the back cover. At the same time, plug the CRICKIT power line into the other 5v output of the regulator. At this point, everything should be plugged in and look something like this:

Note: Ensure the wire direction entering the 5v regulator has the ground on the inside and the power wire on the outer pin. 9. Attempt to connect the back cover to the body while ensuring the larger cables do not obstruct the path of the leg servo horns, and that no wires are pinched between the body and back cover. 10. Continue to secure the back cover to the body with M3 screws. 11. Insert two 18650 batteries into the battery holder, then place the battery cover on the battery holder and secure it with M3 screws.

Software Setup

1. I highly recommend using the 2022-09-22 version of Raspbian Bullseye as the operating system for the SBC in this version.

Bullseye comes pre-installed with several necessary libraries, making things much easier here. If you are not using Bullseye, you may need to look at the libraries imported in the code and take steps to ensure you manually install all libraries. Note: I suggest using the username “bmo” as this will reduce the number of changes you need to make to the code. 2. You will need to flip the display because it is mounted upside down. Simply open “Preferences”, select “Screen Configuration”, right-click on the display, select “Orientation”, and then select “Flip” to easily do this.

If this does not work, you may need to revert to the 2022-09-22 Raspbian version as some newer versions have bugs that prevent the screen from rotating. 3. In the configuration, you need to enable i2c but may also need to enable other features like SSH and VNC. 4. Open a terminal and run the following commands:

sudo apt-get update          sudo apt-get upgrade

5. In the terminal, use the following command to set up the CRICKIT CAT:

pip install Adafruit-blinka          i2cdetect -y 1

6. At this point, you should see a matrix with the number 49. If you cannot, ensure the CRICKIT hat is properly connected and not shorted. 7. In the terminal, run the following command to install the CRICKIT circuit Python library:

pip install Adafruit-circuitpython-crickit

Setting Up Azure Speech Service

1. Register for a free Azure account at azure.microsoft.com (we will use the free tier of the speech service, so unless you use BMO-AI frequently, you should not expect to incur any charges!). 2. Follow these steps to create a speech resource in the Azure portal: https://portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices 3. Get the key and region for the resource: https://learn.microsoft.com/azure/cognitive-services/cognitive-services-apis-create-account#get-the-keys-for-your-resource. 4. In the file I shared, there will be a file containing the offline keyword model table for the wake word “Hey BMO” that can be used without additional steps or code changes. However, if you want a different wake word, you will need to follow the steps in https://learn.microsoft.com/azure/cognitive-services/speech-service/custom-keyword-basics?pivots=programming-language-python to create a wake word. 5. In the terminal, type the following command:

sudo apt-get install build-essential libssl-dev libasound2 wget          pip3 install azure-cognitiveservices-speech

Setting Up Azure Computer Vision

1. Create a computer vision resource in the Azure portal. Make sure to use one of the following regions: East US, Central France, Central Korea, North Europe, Southeast Asia, West Europe, or West US. 2. Use the F0 free pricing tier, which has some usage limits but should be fine for a single robot like this. 3. Copy the API key and endpoint, which will be needed when starting the coding part. 4. Open a terminal on BMO-AI and run the following command:

pip install azure-cognitiveservices-vision-computervision          pip install pillow

Setting Up Open AI

1. Register for an account on OpenAI. 2. Copy your API key. 3. Open a terminal on BMO-AI and run the following command:

pip install openai          pip install tiktoken

Installing the Stable Diffusion Library1. Register for an account at beta.dreamstudio.ai and get the API key. 2. Copy the API key. 3. Install the library by typing the following in the console.

pip install stability-sdk

File Download and Create Directory1. I recommend creating the following directory: /home/bmo/Photos as a place to store the photos taken by BMO-AI. 2. Download the main code and keyword table files to the same directory. 3. Download and extract the facial images to the image directory under home. 4. Edit the Python file to add the API keys, settings, and other information as described in the “code” section.

Operating BMO-AI

1. Start the Python file by typing python filename.py in the terminal window. If you did everything right, there should be no errors. 2. After saying the wake phrase “Hey BMO” (pronounced “be moe”), use any of the following verbal commands. Take a photo Including the word “photo” in your reply (i.e., “Please take a photo”) will prompt BMO-AI to take a photo of whatever it is facing and store it in the photo directory.

It will then ask you if you want the picture, and if you say no, it will email it to you. What do you see? BMO-AI will use Azure cognitive computer vision to describe what it sees. What are you thinking? BMO-AI will take a photo and show you how it imagines what it is looking at using DALL-E 2 image variants. Draw Including the word “draw” in your reply (i.e., “Please draw a picture of a dog talking to a penguin”) will prompt Stable Diffusion to create an image based on what you tell BMO-AI to draw. Describe Including the word “describe” in the prompt (i.e., let’s chat – saying “Let’s chat” will initiate OpenAI ChatGPT 3.5 conversation mode, which will remember your previous questions for context throughout the conversation. Saying “I’m done” will end the chat mode. Ask any question If you say anything other than the above key phrases, BMO-AI will use OpenAI ChatGPT 3.5 to answer a single question but will not remember any follow-up content you ask. 3. Note: The AI functions require an internet connection. If you plan to take the companion robot outside the range of home WiFi, you should set it up to connect with a mobile phone to maintain normal use and have fun in other scenarios. 4. Note: Use the voltmeter on the back to determine when to turn off BMO-AI and replace the batteries. A general rule of thumb is to turn off between 6.8v and 6.6v to prevent shortening battery life due to over-discharge.

via https://www.hackster.io/david-packman/bmo-ai-the-companion-robot-with-artificial-intelligence-b96e8f

Links in the text can be clicked to read the original article at the end