Resolving Ansible Version Compatibility Issues with Containers

Ansible and Python have version dependencies; different versions of Python support different versions of Ansible.

Control Node Python Support

Target Node Python Support

Ansible and Python version incompatibility can lead to execution failures. For example, if the Ansible version is 3.18.2 and the controlled nodes have Python versions 3.12.0 and 3.6.8, executing tasks on the node with Python version 3.6.8 will result in Python errors causing task execution failures. To address this issue, Ansible proposes executing tasks through containers.

Demonstration of Ansible Execution Environment

Assuming there are two servers:

[root@study ansible]# ansible all --list-hosts
  hosts (2):
    servera
    serverb

The Python versions on these two servers are inconsistent, with <span>servera</span> being the current Ansible control node:

[root@study ansible]# ansible --version
ansible [core 2.18.2]
  config file = /root/ansible1/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/python/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/python/bin/ansible
  python version = 3.12.0 (main, Oct  8 2023, 15:41:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/local/python/bin/python3.12)
  jinja version = 3.1.2
  libyaml = True
[root@study ansible]# ssh servera
root@servera's password:
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Sun Feb  9 16:52:29 2025 from 10.0.164.64
[root@localhost ~]# python3 --version
Python 3.6.8
[root@localhost ~]# python
python3                                       python3.6m                                    python-argcomplete-tcsh
python3.6                                     python-argcomplete-check-easy-install-script

• • Ansible version is 2.18.2
• • <span>servera</span> has Python version 3.12.0
• • <span>serverb</span> has Python version 3.6.8

Testing the ansible command:

[root@study ansible]# ansible servera -m ping
[WARNING]: Platform linux on host servera is using the discovered Python interpreter at /usr/local/python/bin/python3.12, but future installation of another Python
interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more
information.
servera | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/python/bin/python3.12"
    },
    "changed": false,
    "ping": "pong"
}
[root@study ansible]# ansible serverb -m ping
[WARNING]: Unhandled error in Python interpreter discovery for host serverb: Expecting value: line 1 column 1 (char 0)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: future feature annotations is not defined
[WARNING]: Platform linux on host serverb is using the discovered Python interpreter at /usr/bin/python3, but future installation of another Python interpreter could
change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
serverb | FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "module_stderr": "Shared connection to serverb closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File "/root/.ansible/tmp/ansible-tmp-1739091843.7380877-172497-249791526669330/AnsiballZ_ping.py", line 107, in <module>\n    _ansiballz_main()\n  File "/root/.ansible/tmp/ansible-tmp-1739091843.7380877-172497-249791526669330/AnsiballZ_ping.py", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File "/root/.ansible/tmp/ansible-tmp-1739091843.7380877-172497-249791526669330/AnsiballZ_ping.py", line 44, in invoke_module\n    from ansible.module_utils import basic\n  File "<frozen importlib._bootstrap>", line 971, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 894, in _find_spec\n  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec\n  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec\n  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec\n  File "<frozen importlib._bootstrap>", line 441, in spec_from_loader\n  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location\n  File "/tmp/ansible_ping_payload_3__qh04p/ansible_ping_payload.zip/ansible/module_utils/basic.py", line 5\nSyntaxError: future feature annotations is not defined\n",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error",
    "rc": 1
}

As we can see, <span>serverb</span> failed to execute due to a low Python version.

A container execution environment for Ansible was created:

[root@study ansible]# podman images
REPOSITORY                     TAG         IMAGE ID      CREATED         SIZE
localhost/ansible_ee_rocky     8.10        516702fcfaf2  32 minutes ago  364 MB

Checking the version of the container execution environment:

[root@study ansible]# ansible-navigator exec -- ansible --version
ansible [core 2.15.13]
  config file = /root/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.20 (main, Oct 23 2024, 13:02:27) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3)
  jinja version = 3.1.5
  libyaml = True

As we can see, the Ansible version in the container execution environment is 2.15.13, and the Python version is 3.9.20.

Testing commands using the container execution environment:

[root@study ansible]# ansible-navigator exec -- ansible servera -m ping
servera | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[root@study ansible]# ansible-navigator exec -- ansible serverb -m ping
serverb | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}

As we can see, this execution was successful. Therefore, when dealing with multiple system versions, using a container execution environment can help avoid version compatibility issues. Additionally, different Ansible collections can be packaged into separate container execution environments for easier management of different modes, such as one container for system initialization and another for managing LDAP collections.

Configuring Ansible Container Execution Environment

Installing Ansible Container Execution Environment

To configure the Ansible execution environment, you need to install <span>ansible-navigator</span> and the container environment. If you are using a RHEL system with a RHEL subscription, the corresponding RPM packages will be available. For open-source environments, <span>ansible-navigator</span> can be installed using <span>pip</span>, and the container environment can be set up using <span>podman</span> or <span>docker</span>.

Since execution is done through containers, a container image is required. The open-source <span>ansible-navigator</span> defaults to using the image <span>ghcr.io/ansible/community-ansible-dev-tools:latest</span>.

# Install ansible-navigator
[root@ansible-controller ~]# python3 -m pip install ansible-navigator

# Install container environment
[root@ansible-controller ~]# dnf install podman

# Test
[root@ansible-controller ~]# cd ansible-navigator/
[root@ansible-controller ansible-navigator]# ansible-navigator welcome
-------------------------------------------------------------------------------------
Execution environment image and pull policy overview
-------------------------------------------------------------------------------------
Execution environment image name:     ghcr.io/ansible/community-ansible-dev-tools:latest
Execution environment image tag:      latest
Execution environment pull arguments: None
Execution environment pull policy:    tag
Execution environment pull needed:    True
-------------------------------------------------------------------------------------
Updating the execution environment
-------------------------------------------------------------------------------------
Running the command: podman pull ghcr.io/ansible/community-ansible-dev-tools:latest
Trying to pull ghcr.io/ansible/community-ansible-dev-tools:latest...

# Executing ansible-navigator welcome will display an interactive page as follows
 0│Welcome
 1│————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
 2│
 3│Some things you can try from here:
 4│- :collections                                    Explore available collections
 5│- :config                                         Explore the current ansible configuration
 6│- :doc <plugin>                                   Review documentation for a module or plugin
 7│- :help                                           Show the main help page
 8│- :images                                         Explore execution environment images
 9│- :inventory -i <inventory>                       Explore an inventory
10│- :log                                            Review the application log
11│- :lint <file or directory>                       Lint Ansible/YAML files (experimental)
12│- :open                                           Open current page in the editor
13│- :replay                                         Explore a previous run using a playbook artifact
14│- :run <playbook> -i <inventory>                  Run a playbook in interactive mode
15│- :settings                                       Review the current ansible-navigator settings
16│- :quit                                           Quit the application
17│
18│happy automating,
19│
20│-winston

In the interactive page above, entering <span>:collections</span> allows you to view the available collections (similar to saving a file in VIM with <span>:wq</span><span>).</span>

<span>ansible-navigator</span> will be discussed in more detail later.

Executing Ansible Tasks via Containers

• • <span>--pp</span> sets the image pull policy
• • <span>--eei</span> specifies which container image to use

[root@ansible-controller ~]# mkdir ansible-navigator/
[root@ansible-controller ~]# cd ansible-navigator/
[root@ansible-controller ansible-navigator]# ansible-navigator exec --pp missing --eei quay.io/ansible/awx-ee:24.6.1 -- ansible --version
ansible [core 2.15.12]
  config file = /root/ansible-navigator/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.19 (main, Jun 11 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True

# Prepare configuration file and host inventory for testing
[root@ansible-controller ansible-navigator]# cat ansible.cfg
[defaults]
inventory=./inventory
host_key_checking=False
[root@ansible-controller ansible-navigator]# cat inventory
master1 ansible_ssh_host=192.168.221.142 HOSTNAME=master1.example.com
worker1 ansible_ssh_host=192.168.221.143 HOSTNAME=worker1.example.com

[all:vars]
ansible_ssh_user=ansible
ansible_ssh_password=redhat

The controlled node system is Rocky 8.10, with Python version 3.6.8.

<span>ansible-navigator</span> can execute tasks in two ways:

• • Execute <span>Ansible Ad-Hoc</span> using the <span>exec</span> option
• • Execute <span>Ansible Playbook</span> using the <span>run</span> option

Executing Ansible Ad-Hoc via Containers

• • <span>-m stdout</span> sets the output mode to non-interactive

[root@ansible-controller ansible-navigator]# ansible-navigator exec --pp missing --eei quay.io/ansible/awx-ee:24.6.1 -- 'ansible all -m command -a "python3 --version"'
worker1 | CHANGED | rc=0 >>
Python 3.6.8
master1 | CHANGED | rc=0 >>
Python 3.6.8

Executing Playbook via Containers

• • <span>-m stdout</span> sets the output mode to non-interactive

[root@ansible-controller ansible-navigator]# ansible-navigator --pp missing --eei quay.io/ansible/awx-ee:24.6.1 -m stdout run test.yml

PLAY [Ansible-navigator test!] *************************************************

TASK [Gathering Facts] *********************************************************
ok: [master1]
ok: [worker1]

TASK [Print Hello World!] ******************************************************
ok: [master1] =>> {
    "msg": "Hello World!"
}
ok: [worker1] =>>
    "msg": "Hello World!"
}

PLAY RECAP *********************************************************************
master1                    : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
worker1                    : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0