Standardizing Package Management Across Hybrid Linux Environments Using Ansible Modules

Two years ago I inherited a fleet that consisted of a mix of Ubuntu 18.04, CentOS 7, and several Amazon Linux 2 boxes. Prior to our standards approach, each of these platforms had their own custom way of patching (so called “bug fixing”). This was typically done through multiple shell scripts that included an if [ -f /etc/debian_version ] conditional and nobody wanted to touch. Each time a new application was added to the mix, someone would fork one of the existing procedures to modify the package names. This resulted in creating yet another way to do what we were already doing – and resulted in creating multiple instances of the same code – and wasted us a ton of late-night maintenance periods.

I knew that Ansible would help me to solve this problem, but I didn’t want to trade 12 shell scripts for 12 new sets of distro-specific tasks. The objective was to stop worrying about which apt or yum parameter to use – just do an “install” regardless of the operating system. At this point in my Ansible journey, I planned to take a wide angle on how to accomplish package management across different Linux distributions in a cohesive way and how that will forever alter how I manage mixed fleets.

Quick Summary

• Single playbook that will execute against the “correct” package manager based on facts collected via Ansible
• Reduced duplicated code through the generic package module and some conditions
• No stale cache or missing repositories due to caching and prerequisite repositories performed before the main installation process.
• Reusable logic through reflected package mappings of application dependencies and distribution-level package representations through Ansible roles.

Prerequisites & Planning

Before writing one task, you should do an Audit of what the system has installed. You should know how many Operating systems you have in Inventory, what the package name Differences are, and if any of those boxes will require additional repositories just to find your applications.

Inventory OS Detection with ansible_os_family

The first thing I do on a new project is use the facts gathered from the Inventory and grep for the ansible_os_family fact. This fact is the foundation of performing cross-distro logic. The ansible_os_family groups your hosts into broad categories of like Operating Systems such as Debian, RedHat, or Suse, and it allows you to create cross-distro tasks without worrying about the specific release of the Operating System.

Here is an example of what that looks like when I run an ad-hoc command using facts to do a quick Inventory gather across a mixed group in a staging area of Production:

$ ansible -i inventory staging -m setup -a 'gather_subset=!all,!min,distribution' -vvv 2>&1 | grep -A2 ansible_os_family
# Output trimmed for brevity
web-ubuntu-01 | SUCCESS => {
    "ansible_facts": {
        "ansible_os_family": "Debian",         <-- Ubuntu lands here
        ...
    }
}
app-centos-02 | SUCCESS => {
    "ansible_facts": {
        "ansible_os_family": "RedHat",         <-- CentOS, RHEL, Rocky
        ...
    }
}
worker-amzn-03 | SUCCESS => {
    "ansible_facts": {
        "ansible_os_family": "RedHat",         <-- Amazon Linux 2 also shows RedHat
        ...
    }
}

What this output tells me is I will only have to create 2 conditions for my playbook logic for either Debian or RedHat based on the families reported. If you are using Suse, they would fall into their separate environment.

Mapping Application Dependencies to Distro-Specific Package Names

If you fail to complete this step early, it will bite you. There are many instances where the same dependency will have different names on Debian versus RedHat. For example: the python3-pip package is named the same on both ubuntu and Centos7; however, on Centos7 the python3-pip package is inside the EPEL repository, and on Centos8+ the name may differ as python39-pip. SSL Development Libraries can also have two different names: on Debian it is named libssl-dev, and on RedHat it is named openssl-devel. If you do not create this mapping before writing your playbooks you will have silent errors in your playbooks where you will encounter a No package matching error at 2AM.

Designing Ansible Package Management Across Different Linux Distros

Once you have established the environment and package requirements you create a playbook that creates the necessary controls for developing out your packages. There were several planning phases before I could produce a playbook that I did not have an issue with using.

Understanding the apt vs yum Modules and Their Limitations

Also, the apt and yum modules can cause errors if you do not take this into consideration when you define your inventory. While these are great tools, if you are attempting to deploy a large number of packages over several distros, you end up writing your deploy processes in a parallel fashion. For example, if you need to deploy 5 packages that are based on three distinct distro types, your playbook becomes very repetitive.

Another area of difficulty with the apt and yum modules is parameter mismatch. The apt module defines an update_cache parameter as a boolean, whereas the yum module also defines the update_cache parameter as a boolean, except that in yum, there is no need to redefine the cache when you install a package. While this is a small difference in how the modules operate, it can become exponentially larger when you try to maintain idempotency and keep a clean playbook.

Introducing the generic package module

In my opinion, the Ansible package module is the answer. It abstracts away the package backend and utilizes the package manager exposed by the target system. You only specify the package you want to install and the state of the package (e.g. rng as present) and the Ansible engine determines whether to invoke a package using the apt, yum, dnf, or zypper package managers.

Although you will still have to deal with the package name differences between the various distributions with the package module, at least you will gain the benefit of not having to remember which module to use with which distribution when you are implementing automated deployment playbooks.

Implementing Conditional Logic with ansible_os_family

You still need to map your package list out by OS Family even if you’re using the generic module. I created a variable that contains the list of packages, keyed by OS Family, then I looped through that variable; this way the condition can live in the variable structure keeping your task clean.

Ensuring Idempotency: state: present, latest, absent

Idempotency in package management allows Ansible to avoid any modification to a given package if it has already been installed or has the state you are expecting. The state: present will install the package if it does not exist. The state: latest will install the package at the latest version available at each run (which I hardly ever do in production without a hard-coded version specified). The state: absent removes the package from the system. For about 90% of what you will do with configuration management stick to using state: present unless you have a concrete reason to keep bumping packages automatically.

Managing Repository Cache with update_cache

Repositories become stale over time. If you run a playbook that installs nginx and the cache has not been updated for week(s), you may get an old version, or even get a 404 error due to the repository metadata expiring prematurely. The update_cache: yes will run apt-get update prior to performing the install when using a Debian-based OS. On RedHat-based systems, this is generally a no-op since yum typically already pulls the meta-data fresh, however it does not hurt to include the update_cache: yes for consistency’s sake.

Here is an abbreviated task that uses the package module in conjunction with a distribution-specific conditional and a separate cache update task.

- name: Update apt cache on Debian family
  apt:
    update_cache: yes
    cache_valid_time: 3600
  when: ansible_os_family == "Debian"
  changed_when: false

- name: Install required packages across distros
  package:
    name: "{{ distro_packages[ansible_os_family] }}"
    state: present
  vars:
    distro_packages:
      Debian:
        - nginx
        - python3-pip
        - libssl-dev
      RedHat:
        - nginx
        - python3-pip
        - openssl-devel

This code snippet first check to see if the Debian apt cache is up to date, then passes the list of installations to the generic module.The cache task’s limitation is that it only executes on Debian machines; whereas, the install task has a dictionary lookup that provides the corresponding list for the corresponding distribution family. As a result, I won’t need to create a large number of when conditions for each applicable package, and I can add new OS families to distro_packages by adding to its dictionary.

Handling Dependency Mapping Across Distros

Since the package manager is implemented in ansible, the process of mapping a dependency across distributions goes far beyond merely changing the pkg name. Although some applications will require additional repositories, some will require either specific versions or system libraries that exist only in “universe” or EPEL. I manage this requirement by referencing a list of package dependencies per family and creating a prereq task prior to executing the install task.

Optimization & Best Practices

Once you’ve created the basic logic, the next step is to make it scalable. There shouldn’t be repeated copies of the distro_packages dictionary across your playbooks.

Migrating to the generic package module for Unified Code

The reason for eventually using the package module versus the separate apt and yum tasks was for long-term maintainability purposes. I have already had to on-board multiple members who do not have advanced knowledge of apt vs. yum options. By providing a single module that indicates to the user, “install this,” it significantly reduces cognitive load, and I can easily replace an installer from Ubuntu to Rocky Linux without refactoring the core logic. All that I would need to do is change the pkg mapping.

Aligning Package Management with the Software Lifecycle

The lifecycle of your application is not independent from the other products in your organization or the entire lifecycle of the product.
There are also many different ways we set up a development environment versus a production environment:

Development environments usually include debug symbols (latest) whereas production environments use pinned versions (present) along with repo priority settings.

As a result, I’ve created group variables to allow me to specify the desired state based on type of environment, while keeping the role itself generic, so that I can use the same role throughout the full software lifecycle without duplicating work.

Using Ansible Roles for Reusable Package Installation Logic

According to Ansible documentation on roles, Roles provide a way to group all variables/tasks/handlers in a single directory structure. I created a Role called “base_packages”, which includes the distro mappings, prereq repos, and the main install task; thus, I can include it whenever I want to ensure that I have the same system tools installed on each system.

The structure of that Role is as follows:

roles/base_packages/
├── defaults
│   └── main.yml
├── tasks
│   └── main.yml
└── vars
    └── main.yml

I loop through a variable that contains the distro-specific list of packages in ‘tasks/main.yml’:

- name: Install base packages
  package:
    name: "{{ item }}"
    state: present
  loop: "{{ base_packages[ansible_os_family] }}"

The ‘base_packages’ variable is defined in ‘vars/main.yml’ as a dictionary with the same structure as the example above. There are no conditionals in the task list; if I wanted to add Alpine and/or Suse later; I would simply add a key with a list to the dictionary.

Common Mistakes & Edge Cases

Even with clean playbooks, there are many opportunities for this setup to fail undetectably.

Forgetting to Install Required Repositories (EPEL, Ubuntu Universe)

If you are starting from a minimal install of CentOS, you can’t install ‘python3-pip’ without first enabling EPEL.Similar situation applies to nginx when you are using Ubuntu within the universe repo. If you have skipped the step required to add EPEL, you will receive an error indicating that an error has occurred while processing the package task.

TASK [Install required packages across distros] ********************************
fatal: [app-centos-02]: FAILED! => {"changed": false, "msg": "No package matching 'python3-pip' found available, installed or updated", "rc": 126, "results": ["No package matching 'python3-pip' found available, installed or updated"]}

To fix the issue, you must add EPEL before running the primary package task. My experience has shown me using the raw module for this problem often works better because the package module does not have access to the repository until after the associated repository exists in the package manager.

- name: Ensure EPEL is available on RedHat family
  raw: yum install -y epel-release
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version | int == 7
  changed_when: false

- name: Install base packages
  package:
    name: "{{ base_packages[ansible_os_family] }}"
    state: present

The raw task executes a shell command directly. The benefit is that since it runs independent of Ansible’s abstraction layer, it will run regardless of whether or not Python or the package module are installed. After running the raw task, you will successfully complete the package task.

Misconfiguring update_cache Leading to Stale Metadata

One major issue I have seen is when users set update_cache: no across their installs in an attempt to speed up installs. During this time, I have seen apt error out on install due to expired or unusable repository metadata. If you intend to maintain your Debian boxes for more than a day, allow them to refresh their metadata cache at least once per day. You can use the cache_valid_time parameter of the apt module to restrict how frequently the cache refresh occurs, so your playbook runs remain efficient.

Workaround: Using the raw Module for Unsupported Package Managers

If you wish to manage systems using non-standard package managers, such as Alpine’s apk, you cannot expect the package module to accomplish this unless you are using a recent version of Ansible with the proper collection installed on your system. If you happen to run into this situation, use the raw module with a simple install command instead of the [package] module or use the command module to call the apk add command. While this approach may not be very visually appealing in a pipeline, it provides a means for continuing your pipeline.

Frequently Asked Questions

Can the Ansible package module install packages from an alternative repository on Ubuntu?

Although it is possible, it is important to understand that the package module was not designed for that purpose. The package module just communicates to the package manager that you want to install a package name. To install a package from an alternate repository, you must create and configure the alternate repository before using the package module, whether it is with the apt_repository module or by adding a .list file. When the repository is configured, the package module will use that repository to install the specified package.

How can I remove packages while managing different Linux distributions?

The state must be set to absent, and the package’s name must also be specified. The package module abstracts the command to execute the removal for you. The only issue you may encounter when removing packages across different distributions is that package names can be different. You could accomplish this issue by using a dictionary-based mapping of packages_to_remove defined for each family, and then iterate through it, with state set to `absent, to remove the appropriate package name.

What is the best practice to manage kernel packages with Ansible on all Linux distributions?

Kernel package management should be used at your own risk. The risk of inadvertently removing the running kernel or causing an unplanned reboot while the playbook is executing is far too high. Therefore, the recommend practice for managing kerne packages is to specify the kernel version using the kernel name (linux-image-5.15.0-76-generic on an Ubuntu system) and set it to state: present, and on RedHat systems you would do it by specifying just kernel' but possibly kernel-devel’ as well. Having set that, you can also configure a reboot handler that gets called when you have set a flag requesting a reboot. You should never set any production kernel packages to state: latest. If you do, you should plan to experience an unplanned reboot.