Background/Cause

The internal product involves many Docker images, and distributed installation involves many hosts, so Ansible was used to write deployment scripts. However, there are various issues due to differences in the operating systems of the deployment machines or the installed versions of Ansible.

The initial solution to this problem was to build an Alpine image with Ansible installed, allowing users to import the image and execute specific commands for installation.

To ensure that installation issues caused by inconsistent Docker versions are avoided, during the execution of the Ansible script, we clear the Docker on the target host and reinstall it using our solution. This means that the machine executing the installation commands cannot be used as a target host for deployment, requiring an additional machine (since we rely on the Ansible playbook inside the Docker container to execute, clearing the Docker service will cause the Ansible playbook to stop executing).

Thus, it naturally led to the thought: can we compile a standalone executable file for the Ansible playbook like Golang does? This would solve the problem.

Of course, there are also solutions like Pulumi, which can use Golang code to define the execution process. However, I have not yet researched whether it can be easily compiled into a binary. Since the deployment scripts are already using Ansible, it is best to continue using Ansible to minimize costs.

Solution Research

Ansible is a tool written in Python.

There are several options for compiling Python tools into executable files:

• pyinstaller

• Nuitka

• cx_Freeze

• py2exe

• PyOxidizer

At the time I was dealing with this issue, PyOxidizer was not yet mature, so I looked at Nuitka, which is a more modern solution that came after PyInstaller, and decided to use it.

Regarding the compilation environment, I planned to use Docker to solve this, and I happened to have used Earthly, which can compile and export files in a Docker environment, so I used that.

Initial Attempt

I first tried to package using the official Python image.

I found someone who had done related work [Ansika/Dockerfile at main · HexmosTech/Ansika].

Combining this Dockerfile with my own research and testing, I came up with the following Earthfile

VERSION 0.8
FROM python:3.9-slim-bullseye
WORKDIR /workdir

deps:
RUN sed -i "s|http://deb.debian.org/debian|http://mirror.sjtu.edu.cn/debian|g" /etc/apt/sources.list \
        && apt-get update -y
# Install ansible 7
RUN python3.9 -m pip install ansible==7.7.0 -i https://mirror.sjtu.edu.cn/pypi/web/simple
# Install nuitka and its dependencies
RUN apt-get install --no-install-recommends -y \
            python3.9-dev \
            build-essential \
            patchelf \
            ccache \
            clang \
            libfuse-dev \
            upx \
        && python3.9 -m pip install nuitka==2.1.3 -i https://mirror.sjtu.edu.cn/pypi/web/simple

build:
FROM +deps
RUN python3.9 -m nuitka \
        --onefile \
        --clang \
        --include-package=pty \
        --include-package=xml \
        --include-package-data=ansible:'*.py' \
        --include-package-data=ansible:'*.yml' \
        /usr/local/bin/ansible-playbook
    SAVE ARTIFACT ansible-playbook.bin AS LOCAL ansible-playbook.bin

Execute earthly +build to compile and export.

However, this solution depends on GLIBC, and because the base image is Debian Bullseye, the runtime environment requires GLIBC_2.29, which has been tested to require Ubuntu 22 or higher.

This environment requirement is too strict, and many internal corporate environments cannot meet it.

Static Compilation?

When compiling with Golang, we can use Alpine + musl to build a dependency-free static executable file. Can this be done?

After my research, the answer is that it is basically not feasible because some libraries in Python use ctypes to call dynamic link libraries. If compiled with musl, it simply cannot run.

I will provide my attempts at static compilation with Nuitka + musl later.

Lowering GLIBC Requirements?

Nuitka’s official response to static compilation issues has provided a solution to use a low-version GLIBC dependency image available in their paid plan for compilation.

This inspired me; I can also use a low-version image for compilation.

Getting on Track

At this point, I had a more feasible solution.

After some attempts, I chose debian:jessie-slim as the base image.

Preparing the Python Environment

However, we need to install a Python environment for compilation, and the official sources for Jessie are too outdated.

At this point, I thought of a tool I had previously researched called Rye, which can install Python versions at any time. In the source code, I found that this tool uses [Releases · indygreg/python-build-standalone], a pre-packaged Python environment.

I attempted to install and compile, but encountered gcc-related errors during Nuitka compilation.

After some research, it seems that the base compilation system GLIBC version used by python-build-standalone is too high, and the highest version of gcc that can be installed on debian:jessie cannot create a symbolic link with it.

So I decided to build Python from source.

I looked at the official documentation about compiling Python from source, which is quite complex, so I searched for any existing solutions.

First, I tried this repository [python-cmake-buildsystem/python-cmake-buildsystem: A cmake buildsystem for compiling Python].

However, the version of cmake required by this repository is too high, and debian:jessie does not meet it. Upgrading cmake would involve upgrading glibc, creating a deadlock.

Is there no simpler solution?

I remembered another well-known tool called pyenv. I looked at the tool’s source code, and it turns out that this tool compiles from Python source code, so I gave it a try.

Python installation was successful.

However, when compiling a helloworld example with Nuitka, an issue occurred.

Pyenv defaults to using gcc to compile Python, while Nuitka detects that the gcc version is too low and switches to g++, causing symbol mismatches, and Nuitka cannot compile (this is my guess).

So let’s try using clang; clang does not have this issue and can be used with any version, all supporting c11.

After testing, Nuitka compilation tests passed.

Thus, I settled on the pyenv solution.

Final Solution

Since the product currently uses Python 3.9 extensively, to maintain consistency, I used pyenv to install 3.9.

As for why I used these dependency packages, I deduced them bit by bit based on the error messages.

Based on the conclusions above, we can provide an Earthfile

VERSION 0.8
FROM debian:jessie-slim
WORKDIR /workdir

deps:
ENV PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM=1
ENV PYTHON_BUILD_MIRROR_URL="https://registry.npmmirror.com/-/binary/python"
ENV PATH="/root/.pyenv/bin:$PATH"
RUN \
# Write to the image source
        echo 'deb [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie main' > /etc/apt/sources.list \
        && echo 'deb [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian-security jessie/updates main' >> /etc/apt/sources.list \
        && echo 'deb-src [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie main' >> /etc/apt/sources.list \
        && echo 'deb-src [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian-security jessie/updates main' >> /etc/apt/sources.list \
        && echo 'deb [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list \
        && echo 'deb-src [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie-backports main' >> /etc/apt/sources.list.d/backports.list \
        && echo 'deb [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie-backports-sloppy main' > /etc/apt/sources.list.d/backports-sloppy.list \
        && echo 'deb-src [trusted=yes] http://mirrors.aliyun.com/debian-archive/debian jessie-backports-sloppy main' >> /etc/apt/sources.list.d/backports-sloppy.list \
        && apt-get update \
# Build tools required for Python
# Reference https://github.com/pyenv/pyenv/issues/2426#issuecomment-1200430855
        && apt-get build-dep -y python3 \
# https://github.com/pyenv/pyenv/wiki/Common-build-problems#2-your-openssl-version-is-incompatible-with-the-python-version-youre-trying-to-install
        && apt-get -t jessie-backports install -y openssl \
        && apt-get install --no-install-recommends -y \
                ca-certificates \
                patchelf \
                ccache \
                curl \
                git \
# Build tools required for Python
# Using clang because of low gcc version
# Pyenv defaults to using gcc to compile Python, while Nuitka detects that the gcc version is too low and switches to g++, causing symbol mismatches, and Nuitka cannot compile (this is my guess)
# Clang does not have this issue and can be used with any version, all supporting c11
# See https://nuitka.net/doc/user-manual.html#c-compiler
                clang \
                build-essential \
                gdb \
                lcov \
                pkg-config \
                libbz2-dev \
                libffi-dev \
                libgdbm-dev \
                liblzma-dev \
                libncurses5-dev \
                libreadline6-dev \
                libsqlite3-dev \
                libssl-dev \
                lzma \
                lzma-dev \
                tk-dev \
                uuid-dev \
                zlib1g-dev
# Build and install Python 3.9.19 from source
RUN curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
RUN eval "$(pyenv init -)"
RUN PYTHON_CONFIGURE_OPTS="--enable-shared" CXX="clang++" CC="clang"  pyenv install 3.9.19 \
        && pyenv global 3.9.19

build:
FROM +deps
RUN /root/.pyenv/versions/3.9.19/bin/python -m pip install --force-reinstall ansible==7.7.0 nuitka==2.1.3 -i https://mirror.sjtu.edu.cn/pypi/web/simple
# During compilation, do not set CXX="clang++" CC="clang" again, otherwise the following error will occur
# TypeError: 'NoneType' object is not iterable:
#     File "/root/.pyenv/versions/3.9.19/lib/python3.9/site-packages/nuitka/build/Onefile.scons", line 314:
#         reportCCompiler(env, "Onefile", output_func=scons_logger.info)
#     FATAL: Error, onefile bootstrap binary build failed.
# Cannot create onefile
RUN /root/.pyenv/versions/3.9.19/bin/python -m nuitka \
        --clang \
        --onefile \
        --include-package=pty \
        --include-package=xml \
        --include-package-data=ansible:'*.py' \
        --include-package-data=ansible:'*.yml' \
        /root/.pyenv/versions/3.9.19/bin/ansible-playbook
    SAVE ARTIFACT ansible-playbook.bin AS LOCAL ansible-playbook.bin

Execute earthly +build to compile. After testing, this version of ansible-playbook.bin can even be used on Ubuntu 14.

As for other pitfalls encountered, they can also be seen in the comments of the code above.