Building a Portable PyPI Server with Raspberry Pi

In this article, I will discuss how to clone the PyPI repository onto a Raspberry Pi and provide content so that connected devices can install dependencies using the pip install command without an internet connection.

Objective

There are many ways to create your own PyPI mirror, and my method may be suitable for everyone now. The goals of this project are:

The hardware for this project must be affordable (<= $100).
There must be little to no setup requirements on the client computer.

I decided to use a Raspberry Pi 4 with a 200GB SD card for storage. I will use minirepo to clone PyPI, pypiserver to serve the packages, and nginx to create a reverse proxy.

Creating a local PyPI server involves four steps:

1. Download and install the operating system and system tools
2. Configure the Raspberry Pi as a WiFi hotspot, DHCP, and DNS server
3. Clone/download the PyPI packages
4. Configure a web server to serve the downloaded packages to connected clients.

Below, I will explain the above steps in detail.

1. Download and Install the Operating System and System Utilities

The Raspberry Pi is a small computer about the size of a credit card, starting at $35. In this project, I used the Raspberry Pi 4, but other models should also work. The Raspberry Pi 3 and 4 models have built-in WiFi adapters, making it easier to set up the Raspberry Pi as a WiFi hotspot or access point than using an external wireless adapter.

To get started, download and install Raspbian. Raspbian is a lightweight operating system based on Debian, optimized for the Raspberry Pi. To work as an access point, the Raspberry Pi will need to have access point software installed, as well as DHCP server software to provide network addresses to connected devices.

Next, before configuring the Raspberry Pi, download all the utilities and packages you will need.

You will need the following packages:

dnsmasq – DNS and DHCP server software
hostapd – Access point software
minirepo – For cloning PyPI for offline use.
pypiserver – To create an index from the cloned PyPI packages
nginx – A web server

To install these packages, run the following two commands:

$ sudo apt install dnsmasq hostapd nginx $ pip install minirepo pypiserver

2. Configure the Raspberry Pi as a WiFi Hotspot, DHCP, and DNS

The goal of this step is to configure a standalone network as a server, so the Raspberry Pi needs to have a static IP address assigned to the wireless interface. To configure a static IP, edit the dhcpcd configuration file.

$ sudo nano/etc/dhcpcd.conf

Add the following content:

Configure DHCP

Many default settings in the dnsmasq configuration are unnecessary. Create a new configuration file:

$ sudo mv/etc/dnsmasq.conf/etc/dnsmasq.conf.orig $ sudo nano/etc/dnsmasq.conf

Add the following configuration:

This sets up DHCP for clients connecting via the wireless interface wlan0. The second line tells the DHCP server (dnsmasq) to listen for connections from the static IP you set up in the previous step. The next line instructs the DHCP server to provide IP addresses from 192.168.4.2 to 192.168.4.30, with a lease time of 24 hours.

Create an Access Point

Next, configure the access point software (hostapd):

$ sudo nano/etc/hostapd/hostapd.conf

Add the following content:

Where it says ssid and wpa_passphrase, add your own network name and network password.

Tell the system where to find this file by opening the hostapd configuration file.

sudo nano/etc/default/hostapd

Find the line with #DAEMON_CONF and replace it with:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Add Routing and Masquerading

Edit /etc/sysctl.conf and uncomment this line.

net.ipv4.ip_forward=1

Add a masquerade for outbound traffic on eth0: sudo iptables-t nat-A POSTROUTING-o eth0-j MASQUERADE

Save the new rules: sudo sh-c"iptables-save > /etc/iptables.ipv4.nat"

Edit /etc/rc.local and add the following content above "exit 0 " to load these rules at startup.

iptables-restore</etc/iptables.ipv4.nat

If you decide to share an internet connection later or set up a bridge on the Raspberry Pi, this is important.

The Raspberry Pi should now be ready to work as an access point. If you connect directly to it, now is a good time to enable SSH. Restart the Raspberry Pi and test if everything is working correctly.

Using a different WiFi-capable device, such as a phone or laptop, scan for the new wireless network. If everything goes smoothly, you should see the WiFi network you created above. Try connecting to it.

3. Clone PyPI

In this section, you will see how to clone PyPI and configure the following packages.

minirepo
pypiserver
nginx

Minirepo

Minirepo is a command-line program that downloads packages from PyPI.org so you can use pip without internet access. The easiest way to install it is with pip.

$ pip install minirepo

The first time you run it, minirepo will ask for the path to the local repository (where it should save the downloaded packages), which defaults to ~/minirepo on Linux. A JSON configuration file is created and saved as ~/.minirepo, which you can edit according to your preferences.

There are many alternatives to cloning PyPI, but I used minirepo because it allows you to selectively download a mirror, for example, downloading all the source code for Python 3 only. At the time of writing this article, the entire PyPI repository is about 1TB, but by using selective downloads, I was able to reduce it to around 120GB. Below is the configuration I used for this project.

The configuration above downloads the source code for Python 3 and restricts the types of packages to sdist, bdist_wheel, and bdist_egg packages. The downside of using this method is that some packages that do not meet the filtering criteria will not be downloaded.

Cloning PyPI takes a long time, so you want to let it run in the background.

Pypiserver

At this point, you should have mirrored PyPI onto your computer. My local PyPI mirror has over 200,000 packages. Before we proceed to the next step, it is necessary to understand what pip is and how it works.

Pip is the most popular tool for installing Python packages and is included in modern versions of Python. It provides the basic core functionality for finding, downloading, and installing packages from PyPI and other Python package indexes and can be integrated into a wide range of development workflows through its command-line interface (CLI).

Pip supports installing packages from:

Installing packages from PyPI (and other indexes) using requirement specifiers.
The URL of VCS projects.
Local project directories.
Local or remote source files.

Since you have cloned the PyPI packages to the local repository, pip can install these packages directly from your newly downloaded local PyPI mirror. However, that is not the purpose of this article. The goal here is to allow remote clients to connect to the Raspberry Pi and download packages over the network. That is the role of pypiserver.

pypiserver will provide a local package index, allowing pip to find the packages in your repository over the network.

First, let’s test if it works.

$ pypi-server-p8080~/minirepo&# Will listen to all IPs.

Note that the command to run it is pypi-server and not pypyserver.

Here, you want to start pypiserver and run it on port 8080. It will find the packages in the minirepo folder. This process will run in the background until you kill it or shut down the Raspberry Pi. I will tell you later how to start it.

If you access the static IP you set for the Raspberry Pi on port 8080 in your browser, you should see a message similar to the one below.

You can now install from the local repository.

pip install--index-url http://localhost:8080/simple/

Or, install from the client computer.

pip install--index-url http://192.168.4.1:8080/

If you install pypiserver on a remote URL without HTTPS, you will receive a “untrusted” warning from pip urging you to add the –trusted-host option.

pip--trusted-host192.168.4.1install--index-url http://192.168.4.1:8080/

There is also a shorter way. pip--trusted-host192.168.4.1install-i http://192.168.4.1:8080/

Always specifying the local PyPI URL and trusted host flag on the command line can be cumbersome. If you want to always install packages from your own mirror, create this pip configuration file in your home directory or virtual environment.

Home Directory:

On Unix and macOS, the home directory file is: $HOME/.pip/pip.conf
On Windows, the file is: %HOME%\pip\pip.ini

In a virtual environment:

On Unix and macOS, the file is $VIRTUAL_ENV/pip.conf
On Windows, the file is: %VIRTUAL_ENV%\pip\pip.ini

I recommend placing this configuration file in the virtual environment.

4. Set Up a Web Server to Serve Packages

By default, pypiserver scans the entire package directory for every incoming HTTP request. When providing a large number of packages, as we are in this example, this can lead to significant slowdowns.

A faster way to serve files is to place pypiserver behind a reverse proxy and enable your web server’s built-in caching capabilities. In this article, I will use nginx, but feel free to use any web server you prefer.

Set Up a New Virtual Host in Nginx

Create a file /etc/nginx/sites-available/cheeshop.com. For the purposes of this article, I will call the new virtual host cheeshop.com.

Run $ sudo nano/etc/nginx/sites-available/cheeshop.com and add the following content.

The first part of the configuration instructs nginx to create a 10GB cache that will remain active for 2 hours. The upstream pypi section is responsible for serving content from pypiserver running on port 8080. CheeseShop is the codename for the Python Package Index, which is why I named the server that. You can use any name or IP address you like.

The server section specifies that port 80 will be used for incoming HTTP connections and will forward those requests to the pypi server.

I do not own the cheeseshop.com domain, but I can use it because we are creating a standalone network that cannot access the Internet. To enable client computers to connect to cheeseshop.com, you will need to tell the DNS server how to resolve it. This will be detailed later.

To enable this new virtual host, you need to create a symbolic link to the configuration file you just created in the /etc/nginx/sites-enabled/ folder:

$ sudo ln-s/etc/nginx/sites-available/cheeseshop.com/etc/nginx-sites-enabled/

Doing this will enable the new virtual host. Check everything is working by running sudo nginx-t. If everything is okay, great! Next, make a small change to DNS to map cheeseshop.com to the IP address.

Open /etc/hosts and add an entry for the newly created cheeseshop.com domain:

The hosts file contains mappings of domains to IP addresses, helping the computer serve you the correct content. Dnsmasq will check this file at startup, so it’s best to restart it:

sudo service dnsmasq restart

Also restart nginx for good measure:

sudo service nginx restart

Assuming everything goes smoothly, you should be able to install Python packages from the client computer using the hostname instead of the IP now.

Using the Server

To test this, connect to the WiFi network of the Raspberry Pi and create a new virtual environment on the client computer and run the following command in the virtual environment:

pip--trusted-host cheeseshop.cominstall-i http://cheeseshop.com django

Running that command will produce the following output:

Start pypiserver at Boot (Optional)

To ensure the pypiserver software starts automatically on boot, create a new Linux service and manage it with systemd.

1. Create a startup script for the service to manage, name it start-pypi-server.sh. Add the following content to it:

2. Copy the script to /usr/bin and make it executable: sudo cp start-pypi-server.sh/usr/bin/start-pypi-server.sh sudo chmod+x/usr/bin/start-pypi-server.sh

3. Create a unit file to define a systemd service. Name it pypiserver.service:

This defines a basic service. The ExecStart directive specifies the command that will run to start the service.

4. Copy the unit file to /etc/systemd/system and set permissions: sudo cp pypiserver.service/etc/systemd/system/pypiserver.service

sudo chmod644/etc/systemd/system/pypiserver.service

Start and Enable the Service

1. Once the unit file is created, you can test the service:

sudo systemctl start pypiserver

2. Check the status of the pypiserver service:

sudo systemctl status pypiserver

This will produce output similar to this:

3. Stop or restart the service: sudo systemctl stop pypiserver sudo systemctl restart pypiserver

4. Finally, use the enable command to ensure the service starts on system boot:

sudo systemctl enable pypiserver

Conclusion

We have learned how to create our own local PyPI clone on the Raspberry Pi. You have learned how to

Set up the Raspberry Pi as an access point
Set the Raspberry Pi as a DHCP and DNS server
Clone PyPI
Serve the cloned packages using a web server.

I did this to demonstrate that a PyPI-like concept can run offline. I am sure there are better or more efficient ways to do this. If you have any suggestions or criticisms, please leave a comment below. Thank you for reading.

– Click below to read the original text to joinCommunity member –

Related posts

Leave a Comment Cancel reply