4 Python Project Management and Build Tools Worth Collecting

Unlike Java, which has evolved from manual builds to semi-automated Ant, and then to Maven, which has become the de facto standard. During this time, Maven has faced challenges from others like Gradle (primarily for Android projects), SBT (mainly for Scala projects), Ant+Ivy, Buildr, etc., but none have shaken Maven’s status, and others have largely followed Maven’s directory layout.

Returning to Python, tools like pip, pipenv, and conda have emerged as package management tools, but there has been no agreement on project directory layout.

Many build processes still follow the traditional Makefile method, supplemented by setup.py and build.py for installation and building via code. There are templates for project layouts, which can then be used to create projects.

Below is a brief overview of the usage of four tools.

1. CookieCutter

2. PyScaffold

3. PyBuilder

4. Poetry

CookieCutter: A Classic Python Project Directory Structure

$ pip install cookiecutter
$ cookiecutter gh:audreyr/cookiecutter-pypackage   
# Use the template from audreyr/cookiecutter-pypackage on GitHub, then answer a series of questions to generate a Python project
......
project_name [Python Boilerplate]: sample
......

The project template generated by cookiecutter looks like this:

$ tree sample
sample
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── docs
│   ├── Makefile
│   ├── authors.rst
│   ├── conf.py
│   ├── contributing.rst
│   ├── history.rst
│   ├── index.rst
│   ├── installation.rst
│   ├── make.bat
│   ├── readme.rst
│   └── usage.rst
├── requirements_dev.txt
├── sample
│   ├── __init__.py
│   ├── cli.py
│   └── sample.py
├── setup.cfg
├── setup.py
├── tests
│   ├── __init__.py
│   └── test_sample.py
└── tox.ini

3 directories, 26 files

This is roughly the main framework of a popular directory structure today, the main elements are:

$ tree sample
sample
├── Makefile
├── README.rst
├── docs
│   └── index.rst
├── requirements.txt
├── sample
│   ├── __init__.py
│   └── sample.py
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

The sample directory contains Python source files in a repeated sample directory, the tests directory contains test files, and an additional docs directory for documentation, README.rst, and other files used for building such as setup, setup.cfg, and Makefile.

This is actually a classic Python project structure, and the subsequent build will use the make command. Entering make will show the instructions defined in the Makefile.

$ make
clean                remove all build, test, coverage and Python artifacts
clean-build          remove build artifacts
clean-pyc            remove Python file artifacts
clean-test           remove test and coverage artifacts
lint                 check style
test                 run tests quickly with the default Python
test-all             run tests on every Python version with tox
coverage             check code coverage quickly with the default Python
docs                 generate Sphinx HTML documentation, including API docs
servedocs            compile the docs watching for changes
release              package and upload a release
dist                 builds source and wheel package
install              install the package to the active Python's site-packages

To use the above build process, the necessary packages such as tox, wheel, coverage, sphinx, flake8 need to be installed via pip. Then commands like make test, make coverage, make docs, make dist, etc., can be executed. Among them, make docs can generate a very nice web documentation.

PyScaffold: Creating a Project

As the name suggests, PyScaffold is a tool for creating Python project scaffolding. Installation and usage:

$ pip install pyscaffold
$ putup sample

This creates a Python project with a directory structure similar to the template selected with cookiecutter, except that it places the source files in the src directory instead of the sample directory.

$ tree sample
sample
├── AUTHORS.rst
├── CHANGELOG.rst
├── CONTRIBUTING.rst
├── LICENSE.txt
├── README.rst
├── docs
│   ├── Makefile
│   ├── _static
│   ├── authors.rst
│   ├── changelog.rst
│   ├── conf.py
│   ├── contributing.rst
│   ├── index.rst
│   ├── license.rst
│   ├── readme.rst
│   └── requirements.txt
├── pyproject.toml
├── setup.cfg
├── setup.py
├── src
│   └── sample
│       ├── __init__.py
│       └── skeleton.py
├── tests
│   ├── conftest.py
│   └── test_skeleton.py
└── tox.ini

The entire project build will use the tox tool. tox is an automation testing and building tool that can create Python virtual environments during the build process, allowing for a clean environment for testing and building.

tox -av can show all tasks defined in tox.ini:

$ tox -av
default environments:
default   -> Invoke pytest to run automated tests

additional environments:
build     -> Build the package in isolation according to PEP517, see https://github.com/pypa/build
clean     -> Remove old distribution files and temporary build artifacts (./build and ./dist)
docs      -> Invoke sphinx-build to build the docs
doctests  -> Invoke sphinx-build to run doctests
linkcheck -> Check for broken links in the documentation
publish   -> Publish the package you have been developing to a package index server. By default, it uses testpypi. If you really want to publish your package to be publicly accessible in PyPI, use the `-- --repository pypi` option.

To execute a command, use tox -e build, tox -e docs, etc.

During my experience with the tox command, each step seemed to be relatively slow, likely due to the time taken to create the virtual machine.

PyBuilder

It’s worth looking at another build tool, PyBuilder, whose directory structure is very similar to Maven. Let’s take a look:

$ pip install pybuilder
$ mkdir sample && cd sample    # Project directory must be created manually
$ pyb --start-project          # Answer some questions to create the required directories and files

Afterward, check its directory structure:

$ tree sample
.
├── build.py
├── docs
├── pyproject.toml
├── setup.py
└── src
    ├── main
    │   ├── python
    │   └── scripts
    └── unittest
        └── python

The build process is still done using the pyb command. You can use pyb -h to view help, and pyb -t to list all tasks. PyBuilder’s tasks are added via plugins, and the plugin configuration is in the build.py file.

$ pyb -t sample
Tasks found for project "sample":
                  analyze -  Execute analysis plugins.
                            depends on tasks: prepare run_unit_tests
                    clean - Cleans the generated output.
          compile_sources - Compiles source files that need compilation.
                            depends on tasks: prepare
                 coverage - <no description available>
                            depends on tasks: verify
                  install - Installs the published project.
                            depends on tasks: package publish(optional)
                  package - Packages the application. Package a python application.
                            depends on tasks: compile_sources run_unit_tests(optional)
                  prepare - Prepares the project for building. Creates target VEnvs
        print_module_path - Print the module path.
       print_scripts_path - Print the script path.
                  publish - Publishes the project.
                            depends on tasks: package verify(optional) coverage(optional)
    run_integration_tests - Runs integration tests on the packaged application.
                            depends on tasks: package
           run_unit_tests - Runs all unit tests. Runs unit tests based on Python's unittest module
                            depends on tasks: compile_sources
                   upload - Upload a project to PyPi.
                   verify - Verifies the project and possibly integration tests.
                            depends on tasks: run_integration_tests(optional)
$ pyb run_unit_tests sample

PyBuilder also creates a virtual environment before building or testing. From version 0.12.9, you can skip the virtual environment creation step using the --no-venvs option. If --no-venvs is used, the Python code will run in the current Python environment where pyb is executed, and required dependencies must be installed manually.

Project dependencies must also be defined in the build.py file:

@init
def set_properties(project):
    project.depends_on('boto3', '>=1.18.52')
    project.build_depends_on('mock')

Then when executing pyb to create a virtual environment, the above dependencies will be installed and run tests and builds within it.

Poetry

The last one is Poetry, which feels like a more mature and actively maintained Python build tool. It has stronger dependency management features; you can add dependencies with poetry add boto3 and view the dependency tree with poetry show --tree. Let’s see how to install and create a project:

$ pip install poetry
$ poetry new sample

The project it creates is simpler than the above:

$ tree sample
sample
├── README.rst
├── pyproject.toml
├── sample
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_sample.py

If you add the --src argument to poetry new, the source file directory sample will be placed in the src directory, i.e., sample/src/sample.

poetry init will generate a pyproject.toml file in the current directory, while the generation of directories must be done manually.

It does not focus on documentation generation, code style checking, or code coverage. Its project configuration is more centralized, all in the pyproject.toml file. What is toml? It is a configuration file format called Tom’s Obvious, Minimal Language (https://github.com/toml-lang/toml).

pyproject.toml is somewhat similar to NodeJS’s package.json file, such as the commands poetry add and poetry install:

# Add dependency for boto3 to pyproject.toml and install (add can also install dependencies from local or git)
poetry add boto3    

# Install dependencies defined in pyproject.toml into the current Python virtual environment
# For example, in <test-venv>/lib/python3.9/site-packages directory, after installing the module, tests can also use it
poetry install       

Other main commands are:

1.  poetry build    # Build installable *.whl and tar.gz files
2.  poetry shell    # Create and use a virtual environment based on the dependencies defined in pyproject.toml
3.  poetry run pytest    # Run tests using pytest, such as tests/test_sample.py
4.  poetry run python -m unittest tests/sample_tests.py  # Run unittest test cases
5.  poetry export --without-hashes --output requirements.txt  # Export requirements.txt file, --dev exports dependencies including dev, or use poetry export --without-hashes > requirements.txt

poetry run can execute any system command, but it will run in the required virtual environment. Therefore, it can be imagined that to generate documentation or coverage for a poetry project, commands must be supported by poetry run ... for sphinx, coverage, or flake8.

In the sample directory (at the same level as the pyproject.toml file), create a file my_module.py with the content:

def main():
    print('hello poetry')

Then write in pyproject.toml:

[tool.poetry.scripts]
my-script="sample.my_module:main"

Finally, execute:

$ poetry run my-script

It will output “hello poetry”.

Through understanding these four tools, the complexity of project structures decreases in the order of cookiecutter-pyproject -> PyScaffold -> PyBuilder -> Poetry, and the difficulty of use follows a similar pattern.

Download 1: OpenCV-Contrib Extension Module Chinese Tutorial

Reply: Extension Module Chinese Tutorial in the backend of "Beginner's Guide to Vision" public account to download the first Chinese version of the OpenCV extension module tutorial online, covering over twenty chapters including extension module installation, SFM algorithms, stereo vision, target tracking, biological vision, super-resolution processing, etc.

Download 2: Python Vision Practical Project 52 Lectures

Reply: Python Vision Practical Project in the backend of "Beginner's Guide to Vision" public account to download 31 visual practical projects including image segmentation, mask detection, lane line detection, vehicle counting, eyeliner addition, license plate recognition, character recognition, emotion detection, text content extraction, face recognition, etc., to help quickly learn computer vision.

Download 3: OpenCV Practical Project 20 Lectures

Reply: OpenCV Practical Project 20 Lectures in the backend of "Beginner's Guide to Vision" public account to download 20 practical projects based on OpenCV, achieving advanced learning of OpenCV.

Community Group

Welcome to join the public account reader group to exchange with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions, etc. (which will gradually be subdivided). Please scan the WeChat number below to join the group, and note: "Nickname + School/Company + Research Direction", for example: "Zhang San + Shanghai Jiaotong University + Visual SLAM". Please follow the format, otherwise, you will not be approved. After successful addition, you will be invited to join relevant WeChat groups based on research direction. Please do not send advertisements in the group, or you will be removed from the group. Thank you for your understanding~