5 Common Pitfalls to Avoid When Deploying Services in Bulk with Ansible

In the field of operations automation, Ansible has become a powerful tool for bulk service deployment due to its agentless architecture, declarative syntax, and rich modules. However, even experienced operations personnel often stumble over seemingly simple details during actual operations. This article, based on over 200 production environment deployment experiences in the past three years, dissects five frequently encountered pitfalls, each accompanied by real-world scenario restoration, error code analysis, and practical solutions validated through experience.

1. Template Variable Conflicts: The Culprit Behind Sudden “Garbage” in Configuration Files

Scenario Restoration: A certain e-commerce team used Ansible to deploy a payment gateway, where everything worked fine in the test environment, but the production environment frequently had the database password in the configuration file replaced with the plaintext {{ db_pass }}. Upon investigation, it was found that developers used both Jinja2 variables ({{ }}) and Nginx configuration variables (like ${host}), leading to parsing confusion during rendering.

Root Cause of the Problem:

The Jinja2 template engine in Ansible uses {{ }} as the default variable delimiter. If there are variable syntaxes from other tools (like Shell scripts’ $var or Python’s {{ }}), it can trigger unexpected variable replacements. This is especially problematic when the template contains multiple nested variables (like {{ app_config.{{ env }}.port }}), as the parser will throw a TemplateSyntaxError directly.

Solution:

1. Escape Variable Delimiters Use {% raw %} and {% endraw %} to wrap non-Ansible variables, for example, the reverse proxy address in Nginx configuration:

location / {
    {% raw %}
    proxy_pass http://
    ${upstream_server};
    {% endraw %}
}

1. Custom Variable Delimiters Modify the delimiters in the playbook using variable_start_string and variable_end_string to avoid conflicts:

- name: Deploy configuration with special variables
  template:
    src: app.conf.j2
    dest: /etc/app.conf
    variable_start_string: '[['
    variable_end_string: ']]'

1. Use|default to Avoid Empty Variables Add default values for potentially undefined variables to prevent empty values during rendering:

db_password: {{ db_pass | default('default_password') }}

Pitfall Avoidance Tip: Execute ansible-playbook --syntax-check before submitting the template to check syntax, and use ansible -m template -a "src=test.j2 dest=/tmp/test" for single-node rendering tests to detect variable conflicts in advance.

2. Asynchronous Task Timeout: Deployment Stuck in “Running” for 30 Minutes

Scenario Restoration: A gaming company used Ansible to deploy server patches in bulk, and out of 100 nodes, 30 always showed async task running, which were marked as failed after timing out. However, upon logging into the nodes, it was found that the patches had actually been installed. The issue was due to the poll parameter of the asynchronous task not being set correctly, causing Ansible to terminate checks prematurely.

Root Cause of the Problem:

The async parameter in Ansible defines the maximum runtime of a task (in seconds), while the poll parameter defines the check interval. By default, if poll: 0 is not set, Ansible will block and wait for the task to complete. If the task takes longer than the async value, it will be deemed a failure even if it actually succeeded. A common misuse is:

- name: Install large package
  yum:
    name: bigpackage
    state: present
    async: 3600  # Maximum runtime of 1 hour
    # Missing poll parameter, defaults to checking every 15 seconds, which may misjudge timeout due to network latency

Solution:

1. Non-blocking Asynchronous Mode For long-running tasks (like database migrations or large file transfers), set poll: 0 and use async_status to poll results:

- name: Asynchronously execute database migration
  command: /opt/db/migrate.sh
  async: 7200  # Maximum runtime of 2 hours
  poll: 0
  register: migrate_task

- name: Poll migration results
  async_status:
    jid: "{{ migrate_task.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 30  # Maximum retries of 30
  delay: 60    # Check every 60 seconds

1. Adjust Timeout by Task Type Different tasks require differentiated async values:

• Package installation: 300-900 seconds (depending on package size)
• System upgrades: over 3600 seconds
• Configuration file generation: usually does not require async (time < 10 seconds)

1. Capture Asynchronous Task Output The standard output of asynchronous tasks is not automatically saved; it must be captured using register along with the stdout parameter:

- name: Asynchronously execute script and capture output
  script: long_running_script.sh
  async: 1800
  poll: 0
  register: script_result

- name: Get output
  async_status:
    jid: "{{ script_result.ansible_job_id }}"
  register: final_result
  until: final_result.finished

Pitfall Avoidance Tip: Check the default timeout configuration using ansible-config dump | grep DEFAULT_TIMEOUT. It is recommended to set timeout = 60 (default is 10 seconds) in the ansible.cfg for production environments to avoid connection timeouts due to network fluctuations.

3. Permission Inheritance Trap: Using Become but Still Getting Permission Denied

Scenario Restoration: A certain team deployed Elasticsearch and set become: yes in the playbook, but still encountered /usr/share/elasticsearch/data: Permission denied. Upon checking, it was found that when using the copy module to copy configuration files, even though administrator privileges were used, the file owner was incorrectly set to root, while Elasticsearch required the data directory to be owned by the elasticsearch user.

Root Cause of the Problem:

The become mechanism in Ansible only elevates task execution privileges and does not automatically handle file permission inheritance. Common misconceptions include:

• Assuming become: yes will make all operations default to root privileges (it actually needs to be combined with become_user)
• Ignoring the owner / group parameters of the copy / template modules, leading to file permissions not matching the service running user
• Executing su - user in the shell module without enabling a login shell via args: executable: /bin/bash, resulting in lost environment variables

Solution:

1. Refined Permission Control Clearly define the become hierarchy and handle tasks requiring user-level permissions separately:

- name: Create data directory (root privileges)
  file:
    path: /var/lib/appdata
    state: directory
    mode: '0755'
    become: yes

- name: Deploy application configuration (application user privileges)
  template:
    src: app.conf.j2
    dest: /home/appuser/app.conf
    owner: appuser
    group: appuser
    become: yes
    become_user: appuser  # Switch to application user for execution

1. shell Module Permission Specification When needing to execute user-level commands in shell, use become_user + args to ensure the environment is correct:

- name: Execute initialization script as application user
  shell: |
    source ~/.bashrc
    ./init.sh --env prod
  args:
    chdir: /home/appuser
    executable: /bin/bash  # Enable login shell
    become: yes
    become_user: appuser

1. Permission Check Tasks Add permission verification after critical steps, such as checking directory permissions after deployment:

- name: Verify data directory permissions
  stat:
    path: /var/lib/appdata
  register: dir_stat

- name: Assert correct directory owner
  assert:
    that:
      - dir_stat.stat.pw_name == 'appuser'
    fail_msg: "Data directory owner is incorrect, should be appuser"

Pitfall Avoidance Tip: Use ansible-doc <module> to check if the module supports become (all core modules do, while third-party modules need additional testing for permission inheritance).

4. Module Idempotence Loss: Repeated Execution Causes Service Anomalies

Scenario Restoration: A financial institution used Ansible to deploy an API gateway, and due to a network interruption, after retrying the playbook, multiple Systemd services with the same name appeared on the server. Investigation revealed that the deployment task used shell: cp gateway.service /etc/systemd/system/ instead of the copy module, and did not check if the service already existed, leading to conflicts from repeated copies.

Root Cause of the Problem:

The core advantage of Ansible is idempotence (consistent results on repeated execution), but the following situations can break this feature:

• Misusing shell / command modules to perform non-idempotent operations (like cp, mv, rm)
• Not using creates / removes parameters to prevent repeated execution
• Using when conditions with changed_when: false to mask state changes
• Executing INSERT on databases instead of INSERT IGNORE (without primary key checks)

Solution:

1. Prioritize Using Core Modules Replace shell commands with idempotent modules, for example:

• Replace shell: echo "max_open_files=1024" >> /etc/security/limits.conf → lineinfile: path=/etc/security/limits.conf line="* hard nofile 1024"
• Replace shell: systemctl start nginx → service: name=nginx state=started (will not repeat execution if already started)

1. shell Module Idempotence Transformation If using shell is necessary, control execution conditions using creates / removes:

- name: Initialize database (only execute first time)
  shell: /opt/db/init_db.sh
  args:
    creates: /var/lib/db/initialized  # Do not execute if this file exists
    become: yes

1. State Judgment Optimization Use register and when to implement conditional execution, avoiding repeated operations:

- name: Check if service is already deployed
  stat:
    path: /usr/bin/appservice
  register: service_exists

- name: Install only if service is not deployed
  yum:
    name: appservice
    state: present
    when: not service_exists.stat.exists

Pitfall Avoidance Tip: Add the --check parameter before executing the playbook for a dry run, observing whether the changed state meets expectations. For critical tasks, it is recommended to implement a “change only if there is a change” mechanism in the handler using the listen mechanism (e.g., restart services after configuration file updates).

5. Inventory Dynamic Refresh Failure: New Nodes Always “Not Found”

Scenario Restoration: A cloud-native team managed AWS instances through Ansible dynamic inventory (ec2.py), and after expanding new nodes, executing ansible all -m ping always failed to recognize the new nodes. Investigation revealed that the dynamic inventory cache was not refreshed in time, and Ansible was still using the node list from an hour ago.

Root Cause of the Problem:

Ansible’s inventory caches host information by default (default cache time is 300 seconds). When using dynamic inventory (like interfacing with cloud vendor APIs or CMDB systems), if the cache is not cleared, it can lead to new nodes not being discovered or already offline nodes still being managed. Common errors include:

• Not setting inventory_cache_expiration, leading to long-term cache validity
• Dynamic inventory scripts lacking permissions to obtain the latest node information
• Switching between multi-environment inventories (like prod/test) without specifying the -i parameter, mistakenly using the default inventory

Solution:

1. Cache Strategy Configuration Set reasonable cache parameters in ansible.cfg:

[inventory]
cache_enabled = True
cache_type = jsonfile
cache_dir = ~/.ansible/inventory_cache
cache_expiration = 60  # Cache validity period of 60 seconds (set to 0 for high-frequency change environments)

1. Force Refresh Inventory Add the --flush-cache parameter when executing commands to clear the cache:

ansible-playbook -i ec2.py deploy.yml --flush-cache

1. Multi-environment Inventory Management Organize inventory using directory structures and specify environments through -i:

inventory/
├── prod/
│   ├── hosts
│   └── group_vars/
└── test/
    ├── hosts
    └── group_vars/

Clearly specify the environment during deployment:

ansible-playbook -i inventory/prod deploy_prod.yml

1. Dynamic Inventory Health Check Add scheduled tasks to verify the validity of dynamic inventory:

- name: Check dynamic inventory availability
  command: ./ec2.py --list
  args:
    chdir: /etc/ansible/inventory
  register: inventory_check
  failed_when: "'error' in inventory_check.stderr"

Pitfall Avoidance Tip: Use ansible-inventory --list to view the currently loaded host list, comparing it with ansible-inventory --graph for topology structure to confirm that groups and variables are correctly applied. For cloud environments, it is recommended to combine dynamic inventory with the wait_for module to ensure new nodes are ready for SSH services before executing deployment.

Summary: Three Principles for Building a Reliable Ansible Deployment System

1. Layered Verification Principle: Embed assert, failed_when, and other assertions in the playbook to achieve full-link verification of “environment check before deployment → step validation during deployment → status confirmation after deployment”.
2. Least Privilege Principle: Avoid using become: yes globally; allocate permissions based on actual task needs. For sensitive operations (like database changes), it is recommended to add tags: risky for extra confirmation during execution.
3. Documentation as Code Principle: For complex playbooks, retain original comments using !unsafe tags, and combine with ansible-doc to generate module descriptions, ensuring team members can quickly understand task logic.

Many of the pitfalls in Ansible stem from neglecting the differences between “declarative syntax” and “actual execution logic”. Remember: good Ansible code should resemble an executable operations manual—clear and readable, while also resilient against the side effects of repeated execution.

Finally, I recommend two practical tools: ansible-lint (static code analysis) and molecule (automated testing framework), which can help intercept 80% of common issues before deployment, making bulk deployment truly a “worry-free task”.

Since you’ve read this far, if you found it helpful, please give it a like, share, and follow. If you want to receive updates promptly, you can also star me. Thank you for reading my article, and see you next time.

1. Template Variable Conflicts: The Culprit Behind Sudden “Garbage” in Configuration Files

2. Asynchronous Task Timeout: Deployment Stuck in “Running” for 30 Minutes

3. Permission Inheritance Trap: Using Become but Still Getting Permission Denied

4. Module Idempotence Loss: Repeated Execution Causes Service Anomalies

5. Inventory Dynamic Refresh Failure: New Nodes Always “Not Found”

Summary: Three Principles for Building a Reliable Ansible Deployment System

Related posts

Leave a Comment Cancel reply