Writing Makefiles for Python Projects

As a fan of Makefiles, I use them in almost every hobby project. I also advocate for their use in work projects.

For open source projects, Makefiles inform code contributors on how to build, test, and deploy the project. Moreover, if you use Makefiles correctly, they can greatly simplify your CI/CD process scripts, as you only need to call the corresponding make command. Most importantly, Makefiles can streamline your development work.

For Python projects, I always use a virtual environment, so I use two different Makefiles strategies:

1. Assume the make command is executed within the virtual environment
2. Encapsulate virtual environment commands via the make command

Assuming the make command is executed in the virtual environment

Let’s look at a very simple Makefile that allows you to build, test, and release a Python project:

all: lint test

.PHONY: test
test:
    pytest

.PHONY: lint
lint:
    flake8

.PHONY: release
release:
    python3 setup.py sdist bdist_wheel upload

clean:
    find . -type f -name *.pyc -delete
    find . -type d -name __pycache__ -delete

The code snippets are straightforward, and all potential contributors immediately know where the entry point of your project is.

Assuming a virtual environment already exists, you need to activate it first and then run the make command:

$ . venv/bin/activate
$ make test

Of course, the inconvenient part is that you have to manually activate the virtual environment in each of your shell windows. So it becomes troublesome when you use tmux to activate a new terminal window or background vim to run something.

Activating the virtual environment within the make command seems difficult to achieve since each piece of code, or even each command, runs in its own shell. However, we will later see a way to bypass this limitation, such as using the .ONESHELL flag, but this does not solve the problem of new code snippets running in a new shell.

Encapsulating virtual environment commands in the make command

The second method basically solves the issue of activating the virtual environment within the make command. This method is learned from makefile.venv^[2], and I simplified it:

# system python interpreter. used only to create virtual environment
PY = python3
VENV = venv
BIN=$(VENV)/bin

# make it work on windows too
ifeq ($(OS), Windows_NT)
    BIN=$(VENV)/Scripts
    PY=python
endif

all: lint test

$(VENV): requirements.txt requirements-dev.txt setup.py
    $(PY) -m venv $(VENV)
    $(BIN)/pip install --upgrade -r requirements.txt
    $(BIN)/pip install --upgrade -r requirements-dev.txt
    $(BIN)/pip install -e .
    touch $(VENV)

.PHONY: test
test: $(VENV)
    $(BIN)/pytest

.PHONY: lint
lint: $(VENV)
    $(BIN)/flake8

.PHONY: release
release: $(VENV)
    $(BIN)/python setup.py sdist bdist_wheel upload

clean:
    rm -rf $(VENV)
    find . -type f -name *.pyc -delete
    find . -type d -name __pycache__ -delete

Functionally, this Makefile is similar to the previous one, but the code looks more complex. Let’s take a look at it line by line to see how it works.

If the virtual environment is already activated, or if packages like pytest and flake8 are already installed in the system Python environment, we can call them directly. However, in the new Makefile, we explicitly use the absolute paths in the virtual environment to invoke them. To ensure the virtual environment exists, each piece of code depends on $(VENV). This ensures that there is a current and updated virtual environment available.

This solution is effective because when we execute . venv/bin/activate, the virtual environment puts its own absolute path into the environment variable. Therefore, each time we call Python or other packages, we are using the versions installed in the virtual environment.

Although the Makefile has become a bit more complex, testing the code is still as simple as executing the command:

$ make test

We do not need to worry about whether the virtual environment is installed or not. If you do not need to support Windows, you can even remove the Windows-related parts from the Makefile. This way, even those who are not very familiar with it can understand this Makefile.

Which one is better?

I find the second solution more convenient. Although I have happily used the first method for years, I only recently learned about the second method. I hadn’t noticed this method before. However, I have noticed that almost all Python projects using Makefile employ the first method, and I am curious why.

Kingname’s Commentary

I frequently use Makefiles in both Python and Golang projects, where I mainly use them to delete __pycache__ in Python projects. In Golang projects, since I use VSCode for development, which has some issues with linting, I run a gofmt command using Makefile to format all .go files after writing the code.

However, there is a very frustrating aspect of Makefiles—they must use tabs for indentation, not spaces. So when writing a Makefile, I have to use vim. My PyCharm is already set to replace all tabs with spaces. If spaces get mixed into the indentation of a Makefile, it will throw an error.

If anyone is interested in Makefiles, I can write a comprehensive article from beginner to advanced. Interested readers please leave a comment~

References