Designing Secure Credential Storage with Ansible Vault for Production Environments

I mistakenly shared an unencrypted production database password on a shared git repository while deploying at 2 am several years ago. This caused panic to ensue as I frantically tried to replace passwords, look through logs, and consume copious amounts of caffeine! Following this experience, I started to use ansible vault appropriately.

Once you start to grow your use of ansible vault beyond a single file for one password, issues start popping up in the form of mismatch in decryption, CI/CD difficulties, and teammates mistakenly committing plaintext secrets. I will share the exact process I currently use, the mistakes I have observed, and how to have securely encrypted sensitive data using ansible vault work in a real-world production environment.

There are four topics I plan on discussing:

1. Secure the control node and defining a vault-id strategy before encrypting anything
2. Creating encrypted files, editing encrypted files in-place and using inline variable encryption with ansible-vault
3. Implementing file system ACLs, no_log decryption at run-time, and CI/CD automation to prevent leaking secrets
4. Identifying common silent mistakes, including overriding vaulted vars from unencrypted host_vars, missing passwords needed for re-keying, and missing vault-ids

Prerequisites & Planning

Environment assumptions for a production control node

In my assumption, there should be one dedicated Linux control node (not your daily laptop) with a non-root service account (e.g. ansible-deploy user), ssh key access has been established with all target machines, and the home directory of the ansible deploy user has been locked down (with chmod 750). Ideally, you are installing ansible-core using whatever package manager is native to your OS, or through pip. Most importantly, your control node should NOT be serving ansible configuration files through a web server.

Planning vault-id labels and password file separation

At the beginning of my development process, I relied on one password across all environments for my vault and all environments that existed across the various secrets stored therein. Secrets in my Staging and my Production environments were located in different group_vars files but they both used the same master passphrase; this resulted in security concerns for two reasons: (1) Any developer who had access to the Staging password could decrypt all Production sealed files if he/she guessed the location of the files on disk, and (2) when the passphrase was changed/rotated, all encrypted resources would need to be re-keyed at once, which poses a risk.

I decided to move from a single passphrase to using Vault IDs. The Ansible tool allows you to assign a vault password source (i.e., the password) with a label of prod, staging, db_admin, or any other descriptor of your choice to a password source. You can then use the --vault-id prod@prompt or --vault-id prod@/secure/path method for the vault source password. The use of vault IDs allows each environment to have its own credentials without requiring you to manage separateansible.cfg files. This means that if a team member has the password for Staging, he/she cannot access encrypted vault information for Production.

Structuring inventory and group_vars for encrypted content

I have my inventory located in the ./inventory directory located at the top of my project. Under the group_vars/ directory, I create one subdirectory for each environment group (prod and staging). Inside each environment subdirectory there is a plain text vars.yml file that contains normal (non-secured) configuration values and a vault.yml file that only includes encrypted values. Because of this separation, team members can easily read the logic of how code works without having to provide the associated vault password. The CI/CD pipeline can skip the decryption step when executing a dry run that does not access any secrets.

.
├── inventory
│   ├── production
│   │   ├── hosts
│   │   └── group_vars
│   │       ├── prod
│   │       │   ├── vars.yml
│   │       │   └── vault.yml
│   └── staging
│       ├── hosts
│       └── group_vars
│           └── staging
│               ├── vars.yml
│               └── vault.yml
├── ansible.cfg
└── playbooks

Why I ended up choosing this method – Once I separated vault passphrase across environments with Vault ID labeling, I no longer had to worry about getting secure content from one environment to another. The directory structure and the automatic loading of group_vars//vault.yml by ansible meant that I didn’t clutter my playbooks with vars_files references.It simply worked and the mental model was very easy for newcomers on the team.

encrypting sensitive data with ansible vault: Architecture & Core Workflow

Initializing vault password files and vault-id configuration

I store my vault passwords in a secure directory on the control node owned by the ansible-deploy user with permissions set to 0700. No passwords are ever typed into terminal history. For a production environment, I generate a 64‑byte random passphrase.

mkdir -p ~/.ansible/vault
openssl rand -base64 64 > ~/.ansible/vault/prod-password
chmod 600 ~/.ansible/vault/prod-password

Next I check that my vault ID is correctly tied to the password file using the --vault-id command with a file as a source. The label prod is tied to that password file.

ansible-vault view --vault-id prod@~/.ansible/vault/prod-password secret.yml

If you want to bind a file as a vault-to-ID configuration but the file is not on the filesystem yet, you will get an error when running the above command, but it will still have been configured as a vault-to-ID binding. The end result of this method of configuration is that you will never pass a password in plaintext on your command line.

Creating encrypted variable files with ansible-vault create

I create my initial ###creating encrypted variable files with vault using the vault-create command from the command line, and by referencing my vault ID to do so:

ansible-vault create --vault-id prod@~/.ansible/vault/prod-password inventory/production/group_vars/prod/vault.yml

I open an editor when I run this command, and drop in any of my secrets, e.g., db_password: superSecretProdDB. Once I save and close the editor, the entire file containing my secrets is encrypted. Later I can view it with the ansible-vault view command and reference the same --vault-id flag. Note that I did not specify an ANSIBLE_VAULT_PASSWORD_FILE environment variable or a default vault ID password in the ansible.cfg file. This forces you to explicitly think about the password that you are currently using, as the vault ID is included with every command.

Selective variable‑level encryption inside plaintext YAML

Whole text based files can be encrypted, however sometimes you want an isolated secret value stored in an otherwise readable vars file. With ansible, you can inline store secret values, with the encrypt_string command to do this.

ansible-vault encrypt_string --vault-id prod@~/.ansible/vault/prod-password \
  --name 'api_token' 's3cret-abc123'

api_token: !vault |
          $ANSIBLE_VAULT;1.2;AES256;prod
          35346130643034653937346238373661306334356231636437386330336338653764323261333230
          3064393330663030616462393138663564323230393839630a383532373565623337356264643034
          ...

The API token that I used will typically reside in alongside the rest of its regional endpoints using a !vault block pasted directly into my vars.yml. This way, much of the file remains readable, while using variable encryption for the sensitive piece of data. Only the tasks that reference the api_token will require the vault password at runtime.

To rotate an existing database password after several weeks of use without needing to recreate the entire file, I simply run:

ansible-vault edit --vault-id prod@~/.ansible/vault/prod-password \
  inventory/production/group_vars/prod/vault.yml

This opens the file in my editor decrypted. I can make the change and save it, and it will automatically be re‑encrypted. I have gotten into the habit of using the ansible-vault edit process and it also preserves the old permissions and vault ID metadata of the original file — and no temporary plaintext copies are ever placed on disk during this process.

Integrating encrypted files into Ansible’s automatic decryption pipeline

Because of my placement of vault.yml in group_vars/prod/, when I target hosts in the prod group, Ansible will automatically find it based on the provided vault‑id at runtime:

ansible-playbook -i inventory playbooks/site.yml \
  --vault-id prod@~/.ansible/vault/prod-password

As a result, the variable db_password is available to my roles without requiring any additional tasks to include it. Ansible will provide runtime decryption on the fly; it never writes the secret to disk on the control node.

Optimization & Best Practices

Implementing security layers with filesystem ACLs and gitignore rules

Vault password files are your keys to the kingdom. On my control node, I set strict POSIX permissions of chmod 600 and chown ansible-deploy:.Then I put the directory that holds the passwords in my .gitignore, as password files should never be versioned. Since I also sometimes have unencrypted YAML that looks like secrets and don’t want to accidentally commit this data, I add a rule that ignores vault.yml unless it has an Ansible Vault header.

# .gitignore
**/vault-password*
**/*.vault-pass

Then, using a pre-commit hook (which I’ll describe later), I scan for ANSIBLE_VAULT headers to prevent any commit from containing plain-text passwords.

Runtime decryption strategies for sensitive output and no‑log enforcement

Whenever you encrypt data at rest, if a task outputs that secret to the screen (stdout) or a log file, you have an information leak. Therefore, I set no_log: true for every task that deals with a vaulted variable, such as in a playbook:

- name: Apply database migration
  command: "mysql -u {{ db_user }} -p{{ db_password }} -e '...'"
  no_log: true

This prevents Ansible from providing the command-line arguments, which would otherwise disclose the password. To take this approach further, I also set the environment variable ANSIBLE_DISPLAY_ARGS_TO_STDOUT=false in CI pipelines to prevent any command argument logging.

Using vault-id for multi‑environment credential isolation

As noted above, vault-ids are Best Practices for managing secrets across multiple environments. I use different vault-ids such as prod, staging, and ci-test, each with its own password file. When I run my staging playbook, I pass --vault-id staging@~/.ansible/vault/staging-password to Ansible. If I accidentally reference a variable in a task that only exists in prod, the play will fail instead of being silently decrypted with the wrong key. This level of separation is significantly more secure and robust than simply looking up the vault password file or storing it in environment variables.

Leveraging password files in CI/CD without human interaction

In my pipeline run by Jenkins, I’m using the vault password file from a secure credential store like Kubernetes Secrets or the Jenkins credentials plugin. The ansible-playbook command is provided the option –vault-password-file so that it does not prompt the user for a password while executing playbooks.

stage('Deploy') {
    steps {
        withCredentials([file(credentialsId: 'ansible-vault-prod', variable: 'VAULT_PASS')]) {
            sh """
                ansible-playbook -i inventory playbooks/deploy.yml \
                  --vault-id prod@${VAULT_PASS}
            """
        }
    }
}

The output of the pipeline will never contain the plaintext version of the password. The file itself is resolved after execution. This technique keeps password files out of automatically generated logs and scripts.

Common Mistakes & Edge Cases After Setup

Accidentally committing plaintext secrets to version control

This happens most commonly when someone adds a new secret into group_vars/prod/vars.yml without encrypting it first. To help catch these by avoiding the accidental usage of plaintext passwords, I have added a pre-commit hook that will stop any commit that has plaintext passwords.

#!/bin/bash
# .git/hooks/pre-commit (simplified)
if git diff --cached --name-only | grep -E 'group_vars.*\vault\.yml'; then
  echo "ERROR: vault.yml must be encrypted. Use ansible-vault encrypt."
  exit 1
fi
# Scan staged files for common secret patterns
if git grep -E 'password:|secret:|api_key:' $(git diff --cached --name-only); then
  echo "ERROR: Plaintext secret detected. Encrypt inline or move to vault.yml."
  exit 1
fi

While it is true that regular expressions are not bulletproof, they will help provide validation that a plaintext password was not copied and pasted during the creation of the commit. I also have added a CI process step that runs ansible-vault view on the vault file to confirm that it can be decrypted with the CI vault password; otherwise, the pipeline will fail quickly.

Decryption failures with mismatched vault-IDs

When you run your playbook, you will get the following error:

ERROR! Vault password file not found for vault id prod

This means that Ansible found an encrypted file that was created with a different vault-ID or that you didn’t provide the matching –vault-id. To fix this, verify what vault ID was used at the time of encryption. You may check the value located at the top of the vault file by executing: “head -1 vault.yml.” If the vault prefix is “prod”, then you need to supply a vault-id of “prod@…”Certain times a vault label may appear blank (ANSIBLE_VAULT;1.1;AES256), meaning it was only encrypted and not assigned a label. If you pass --vault-id @prompt alone (with no provided label) to the command, a label could be introduced at the time of the encrypt operation.

Overriding encrypted variables with unencrypted host_vars precedence

According to the rules that apply to Ansible variables, if you have defined a plain text/unencrypted variable in the host_vars/webserver01.yml, it will replace any previously encrypted variable in the group_vars/prod/vault.yml file. Therefore, if you put the db_password variable as SuperSecret in your host_vars for a quick test, this will result in the use to the host_vars database password, and no error will be reported. To mitigate this risk, I have established a policy whereby all secret keys must be kept in vault-only files in group_vars. I have implemented a CI linting tool that will identify all occurrences of password: in non-vault yaml files and fail the build.

Re‑keying vault files without knowing the old password (edge case)

You lost the original password for the vault, but you may have another vault-id (for example, an administrative credential that was used to encrypt a copy of the file). In this case, you can decrypt with the known secondary vault-id, and then re-encrypt using a new primary password.

ansible-vault rekey --vault-id secondary@~/.ansible/vault/admin-password \
  --new-vault-password-file ~/.ansible/vault/new-prod-password \
  inventory/production/group_vars/prod/vault.yml

The original password is not needed because you have a valid secondary vault-id to decrypt. Once the file has been successfully re-encrypted with a different primary password, all references to the original primary password file should be updated to use the newly generated password. If the secondary vault-id is no longer needed, it should be removed. The ability to do this saved my company when all that was left of my company was an encrypted backup copy.

Frequently Asked Questions

Can I use different vault passwords for staging and production environments?

Yes! To have different passwords for the production and staging environments, use separate vault-ids (prod/staging) and use different password files. Your inventory would have the group_vars/prod/vault.yml file encrypted using --vault-id prod@... and the group_vars/staging/vault.yml file encrypted using --vault-id staging@.... When running your playbook, simply pass the vault-id of the environment you would like to run against. According to the Ansible Vault documentation, you can supply multiple instances of the --vault-id parameter if your playbook needs to decrypt both, but keeping separate vault-ids allow for a much cleaner separation.

How do I securely share the vault password file with multiple team members?

Never mail or slack share the unencrypted file. Use a shared secrets manager: Encrypt the file using GPG using each team member’s key, store it in a HashiCorp Vault Key-Value storage system, and issue temporary wrapped tokens. Alternatively, use a team password manager that supports file attachment capabilities, and at the time of running the playbook retrieve the password through an authenticated API call, not the unencrypted file. If you are a small team, then ansible-vault with GPG is straightforward – simply encrypt the vault password files using the team’s GPG keys and decrypt them only when needed.

Can I decrypt vault‑encrypted content at runtime and still hide it from logs?

Yes, it is critical that you set no_log: true on any tasks that utilize a vaulted variable. If it is necessary for debugging, temporarily set ANSIBLE_DEBUG=1, but do not do this in production. If you need output to be displayed but only display the masked value, create either a custom callback or utilize the ansible.builtin.debug module using var: my_secret and verbosity: 2 – combined with no_log: false on that debugging task. The no_log documentation provides additional information. The key to remember is that Ansible decrypts the variable in memory and that the decision to log the variable is handled individually on a task by task basis.