Can the ESP32 Be Used as a Web Crawler Server?

ESP32 as a Web Crawler Server

I have seen people online suggesting using the ESP32 as a web crawler server, so I decided to give it a try. Since I am not an expert in microcontrollers, I am not sure if the issue was with my board or my settings. However, my conclusion is that using the ESP32 as a web crawler server is not very practical for two reasons:

• Because this microcontroller has limited computing resources, it cannot crawl HTTPS pages.

• The circuit board overheats significantly.

Of course, these two issues may be specific to the board I used. After all, there are many ESP32 boards on the market, and my testing with one board cannot generalize the entire experience.

However, it is still possible to play around with it, so below I will teach you how to set up a web crawler server using the ESP32.

Installation Tools

Typically, programs for microcontrollers are developed in C language, but using C for small crawlers can be quite cumbersome. Fortunately, microcontrollers like the ESP32 can also use Python (technically a derivative of Python) called MicroPython.

First, install the Python tools for flashing firmware and uploading Python script files on your computer. Use the following commands to install the two Python-based tools, esptool and ampy.

pip install esptool # Used for flashing firmware
pip install adafruit-ampy # Used for uploading files to ESP8266

After installation on macOS, the directory is: /Library/Frameworks/Python.framework/Versions/(your actual Python version)/bin. To successfully run the esptool and ampy commands in the terminal, you need to add this directory to your environment variables or directly switch to this path using the cd command.

Flashing Firmware

Downloading Firmware

The environment on the computer is ready, and now we need to prepare the ESP32 environment. Actually, the PCB board we are using is the ESP8266 because it has a Wi-Fi module, making it easy to connect to the internet. To run Python on the ESP32, we need to install a Python interpreter on it, just like installing Python on your computer, but the microcontroller’s CPU is different, requiring a special version of Python: MicroPython.

You can download it from the official MicroPython website. I downloaded the ESP8266_GENERIC-20250415-v1.25.0.bin file.

Flashing the Firmware

Connect the ESP8266 development board to your computer via USB and check the COM port (Windows: check in Device Manager; Linux/Mac: run the command ls /dev/tty* in the terminal). The terminal output will show a COM port like /dev/tty.usbserial-xxxx.

ls /dev/tty* # /dev/tty.usbserial-1400

Erasing Flash

To avoid environmental interference, we need to erase the Flash of the ESP8266. With the microcontroller properly connected to the computer, execute the following command in the terminal:

./esptool --port your_board_serial_port erase_flash

For example:

./esptool --port /dev/tty.usbserial-xxxx erase_flash

Writing Firmware

The firmware here acts as a Python interpreter. Execute the following command in the terminal to write the firmware to the ESP8266, ensuring that you use the correct COM port and firmware file path.

./esptool --port your_board_serial_port --baud 460800 write_flash --flash-size=detect 0x0 your_firmware_directory/ESP8266_GENERIC-20250415-v1.25.0.bin

Verification

After writing the firmware, press the RST button on the board or reconnect the USB cable to restart the ESP8266. Then execute the following command in the terminal to verify if the firmware was written successfully. If successful, the command will display the filename boot.py, which is the startup file for MicroPython.

./ampy --port your_board_serial_port ls # boot.py

If all the above steps are successful, congratulations! You have completed the necessary conditions for running Python code on the ESP8266. Next, you just need to upload the Python code files to the ESP8266.

Uploading Scripts

You can use the ampy command to upload Python code files to the ESP8266. Note that if you want the code to execute when the board is powered on, the filename must be main.py.

You can also install a tool called Thonny on your computer, which allows you to directly select the MicroPython interpreter for the ESP8266, avoiding the hassle of frequently executing ampy commands during code debugging.

./ampy --port your_board_serial_port put main.py

Running Python Web Crawler on ESP8266

You can first test the running environment of the ESP8266 by saving the following code as main.py and uploading it to the ESP8266. Then press the RST button on the board, and the LED on the board should start blinking.

import machine
import time
# The built-in LED on NodeMCU is usually on GPIO2
led = machine.Pin(2, machine.Pin.OUT)
while True:
    led.on()  # Turn on LED
    time.sleep(1)
    led.off()  # Turn off LED
    time.sleep(1)

Then you can use the same method to save the following crawler code as main.py and upload it to the ESP8266 to run.

import network
import urequests
import ure
import time
import os
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.connect("Your WIFI SSID", "Your WIFI Password")
while not sta_if.isconnected():
    time.sleep(1)
print("Wi-Fi connected, IP address:", sta_if.ifconfig())
# HTTP GET request
response = urequests.get("http://example.com")
html = response.text
print(html)
response.close()