Lab 9 – Implementing Ansible Vault

Introduction

Ansible Vault is a feature that allows users to encrypt values and data structures within Ansible projects. This provides the ability to secure any sensitive data that is necessary to successfully run Ansible plays but should not be publicly visible, like passwords or private keys. Ansible automatically decrypts vault-encrypted content at runtime when the key is provided.

In this lab, we will demonstrate how to use Ansible Vault and explore some recommended practices to simplify its use. We will be using RHEL7 server for the Ansible control machine. No remote hosts are needed.

1. Ansible Vault

Vault is a mechanism that allows encrypted content to be incorporated transparently into Ansible workflows. A utility called ansible-vault secures confidential data by encrypting it on disk. To integrate these secrets with regular Ansible data, both the ansible and ansible-playbook commands, for executing ad hoc tasks and structured playbook respectively, have support for decrypting vault-encrypted content at runtime.

Vault is implemented with file-level granularity, meaning that files are either entirely encrypted or unencrypted. It uses the AES256 algorithm to provide symmetric encryption keyed to a user-supplied password.

1.1 Login as “root” user on pod0-master.origin.com node:

ssh root@pod0-master.origin.com

1.2 Create and change to the directory ~/test-ansible-vault

mkdir ~/test-ansible-vault && cd ~/test-ansible-vault

2. How to manage sensitive files with ansible-vault

The ansible-vault command is the main interface for managing encrypted content within Ansible. This command is used to initially encrypt files and is subsequently used to view, edit, or decrypt the data.

2.1 Creating new encrypted files

To create a new file encrypted with Vault, use the ansible-vault create command. Pass in the name of the file you wish to create.

ansible-vault create vault.yml

You will be prompted to enter and confirm a password:

Output:

New Vault password: onecloud
Confirm New Vault password: onecloud

When you have confirmed your password, Ansible will immediately open an editing window where you can enter your desired contents.

2.2 To test the encryption function, enter some test text:

i
Secret information

2.3 Ansible will encrypt the contents when you close the file. If you check the file, instead of seeing the words you typed, you will see an encrypted block:

cat vault.yml

Output:

$ANSIBLE_VAULT;1.1;AES256
38373565373330623062303235306663353533363630613334356162623233373338643630616466
3537303261613166356235656536656165323065316534340a653837373463383935303163366337
64373039306564656530643932333962356530386634656430343436366161666563623866613862
3535646336326163340a666631303166646235616539346232353830316138646265306238343165
33623933623663326531623933666132666233623863376663633735376138333234

We can see some header information that Ansible uses to know how to handle the file, followed by the encrypted contents, which display as numbers.

3. Encrypting Existing Files

If you already have a file that you wish to encrypt with Vault, use the ansible-vault encrypt command instead.

3.1 For testing, we can create an example file by typing:

echo 'unencrypted stuff' > encrypt_me.txt

3.2 Now, you can encrypt the existing file by typing:

ansible-vault encrypt encrypt_me.txt

You will be prompted to provide and confirm a password. Afterwards, a message will confirm the encryption:

Output:

New Vault password: onecloud
Confirm New Vault password: onecloud
Encryption successful

3.3 Instead of opening an editing window, ansible-vault will encrypt the contents of the file and write it back to disk, replacing the unencrypted version.

If we check the file, we should see a similar encrypted pattern:

cat encrypt_me.txt

Output:

$ANSIBLE_VAULT;1.1;AES256
31663737306634336238656439383634613163356632303537623466366561373935663664393038
6336343562363439363737313864383163623934666265380a656563386135386162353234326366
39613862313835656266623032636236333661666131393732663231656261393438373334396331
3862626333636437610a373863626566383865346164386666306330616430616635646532383366
32623962623230303765613066393135366233346465626134363636346436353731

4. Viewing Encrypted Files

Sometimes, you may need to reference the contents of a vault-encrypted file without needing to edit it or write it to the filesystem unencrypted. The ansible-vault view command feeds the contents of a file to standard out. By default, this means that the contents are displayed in the terminal.

4.1 Pass the vault encrypted file to the command:

ansible-vault view vault.yml

You will be asked for the file’s password. After entering it successfully, the contents will be displayed:

Output:

Vault password: onecloud
Secret information

As you can see, the password prompt is mixed into the output of file contents. Keep this in mind when using ansible-vault view in automated processes.

5. Editing Encrypted Files

5.1 When you need to edit an encrypted file, use the ansible-vault edit command:

ansible-vault edit vault.yml

Output:

Vault password: onecloud

5.3 You will be prompted for the file’s password. After entering it, Ansible will open the file an editing window, where you can make any necessary changes.

Append the below line of text

i
Modified data

Upon saving, the new contents will be encrypted using the file’s encryption password again and written to disk.

6. Manually Decrypting Encrypted Files

6.1 To decrypt a vault encrypted file, use the ansible-vault decrypt command.

ansible-vault decrypt vault.yml

6.2 You will be prompted for the encryption password for the file. Once you enter the correct password, the file will be decrypted:

Output:

Vault password:onecloud
Decryption successful

6.3 If you view the file again, instead of the vault encryption, you should see the actual contents of the file:

cat vault.yml

Output:

Modified data
Secret information

Your file is now unencrypted on disk. Be sure to remove any sensitive information or re-encrypt the file when you are finished.

7. Changing the Password of Encrypted Files

7.1 If you need to change the password of an encrypted file, use the ansible-vault rekey command:

ansible-vault rekey encrypt_me.txt

When you enter the command, you will first be prompted with the file’s current password:

Output:

Vault password: onecloud

After entering it, you will be asked to select and confirm a new vault password:

Output:

New Vault password:ansible
Confirm New Vault password:ansible

When you have successfully confirmed a new password, you will receive a message indicating success of the re-encryption process:

Output:

Rekey successful

The file should now be accessible using the new password. The old password will no longer work.

8. Running Ansible with Vault-Encrypted Files

After you’ve encrypted your sensitive information with Vault, you can begin using the files with Ansible’s conventional tooling. The ansible and ansible-playbook commands both know how to decrypt vault-protected files given the correct password. There are a few different ways of providing passwords to these commands depending on your needs.

8.1 To follow along, you will need a vault-encrypted file. You can create one by typing:

ansible-vault create secret_key

New Vault password:onecloud
Confirm New Vault password:onecloud

8.2 Select and confirm a password. Fill in whatever dummy contents you want:

i
confidential data

8.3 We can also create a temporary hosts file as an inventory:

cat > hosts <<EOF
[database]
localhost ansible_connection=local
EOF

8.4 Create an ansible.cfg file in the current directory

cat > ansible.cfg <<EOF
[defaults]
inventory = ./hosts
EOF

When you are ready, continue on.

9. Using an Interactive Prompt

The most straightforward way of decrypting content at runtime is to have Ansible prompt you for the appropriate credentials. You can do this by adding the ––ask-vault-pass to any ansible or ansible-playbook command. Ansible will prompt you for a password which it will use to try to decrypt any vault-protected content it finds.

9.1 we needed to copy the contents of a vault-encrypted file to a host, we could do so with the copy module and the ––ask-vault-pass flag. If the file actually contains sensitive data, you will most likely want to lock down access on the remote host with permission and ownership restrictions.

ansible --ask-vault-pass -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost

Our task specifies that the file’s ownership should be changed to root, so administrative privileges are required. You will then be asked for the Vault password:

Output:

Vault password: onecloud

When the password is provided, Ansible will attempt to execute the task, using the Vault password for any encrypted files it finds. Keep in mind that all files referenced during execution must use the same password:

Output:

localhost | SUCCESS => {
    "changed": true,
    "checksum": "021cc7334a5c10bb36174319482f3932741fc155",
    "dest": "/tmp/secret_key",
    "gid": 0,
    "group": "root",
    "md5sum": "30171f39abe34206b2e56d893aa91296",
    "mode": "0600",
    "owner": "root",
    "secontext": "unconfined_u:object_r:admin_home_t:s0",
    "size": 20,
    "src": "/root/.ansible/tmp/ansible-tmp-1531788518.95-35953169726371/source",
    "state": "file",
    "uid": 0
}

Prompting for a password is secure, but can be tedious, especially on repeated runs, and also hinders automation. Thankfully, there are some alternatives for these situations.

10. Using Ansible Vault with a Password File

10.1 If you do not wish to type in the Vault password each time you execute a task, you can add your Vault password to a file and reference the file during execution.

Apply your password in a .vault_pass file like this:

echo 'onecloud' > .vault_pass

10.2 The ––vault-password-file flag is available on the command line. We could complete the same task from the last section by typing:

ansible --vault-password-file=.vault_pass -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost

You will not be prompted for the Vault password this time.

Output:

localhost | SUCCESS => {
    "changed": false,
    "checksum": "021cc7334a5c10bb36174319482f3932741fc155",
    "dest": "/tmp/secret_key",
    "gid": 0,
    "group": "root",
    "mode": "0600",
    "owner": "root",
    "path": "/tmp/secret_key",
    "secontext": "unconfined_u:object_r:admin_home_t:s0",
    "size": 20,
    "state": "file",
    "uid": 0
}

11. Reading the Password File Automatically

11.1 To avoid having to provide a flag at all, you can set the ANSIBLE_VAULT_PASSWORD_FILE environment variable with the path to the password file:

export ANSIBLE_VAULT_PASSWORD_FILE=./.vault_pass

11.2 Execute the command without the ––vault-password-file flag for the current session:

ansible -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost

Output:

localhost | SUCCESS => {
    "changed": false,
    "checksum": "021cc7334a5c10bb36174319482f3932741fc155",
    "dest": "/tmp/secret_key",
    "gid": 0,
    "group": "root",
    "mode": "0600",
    "owner": "root",
    "path": "/tmp/secret_key",
    "secontext": "unconfined_u:object_r:admin_home_t:s0",
    "size": 20,
    "state": "file",
    "uid": 0
}

11.3 To make Ansible aware of the password file location across sessions, you can edit your ansible.cfg file.

Open the local ansible.cfg file we created earlier:

cat >> ansible.cfg <<EOF
vault_password_file = ./.vault_pass
EOF

Now, when you run commands that require decryption, you will no longer be prompted for the vault password. As a bonus, ansible-vault will not only use the password in the file to decrypt any files, but it will apply the password when creating new files with ansible-vault create and ansible-vault encrypt.

12. Reading the Password from an Environment Variable

You may be worried about accidentally committing your password file to your repository. Unfortunately, while Ansible has an environment variable to point to the location of a password file, it does not have one for setting the password itself.

However, if your password file is executable, Ansible will run it as a script and use the resulting output as the password.

12.1 Open up your .vault_pass file in your editor:

cat >> .vault_pass <<EOF
import os
print os.environ['VAULT_PASSWORD']
EOF

12.2 You can then set and export the VAULT_PASSWORD environment variable, which will be available for your current session:

export VAULT_PASSWORD=my_vault_password

echo $VAULT_PASSWORD

Output:

my_vault_password

You will have to do this at the beginning of each Ansible session, which may sound inconvenient. However, this effectively guards against accidentally committing your Vault encryption password, which could have serious drawbacks.

13. Using Vault-Encrypted Variables with Regular Variables

While Ansible Vault can be used with arbitrary files, it is most frequently used to protect sensitive variables. We will work through an example to show you how to transform a regular variables file into a configuration that balances security and usability.

13.1 When you created the hosts file earlier, you placed the localhost entry in a group called database to prepare for this step.

Databases usually require a mixture of sensitive and nonsensitive variables. These can be assigned in a group_vars directory in a file named after the group:

mkdir -p group_vars

13.2 Inside the group_vars/database file, set up some variables. Some variables, like the MySQL port number, are not secret and can be freely shared. Other variables, like the database password, will be confidential:

cat > group_vars/database <<EOF
---
# nonsensitive data
mysql_port: 3306
mysql_host: 10.0.0.3
mysql_user: fred

# sensitive data
mysql_password: supersecretpassword
EOF

13.3 We can test that all of the variables are available to our host with Ansible’s debug module and the hostvars variable:

ansible -m debug -a 'var=hostvars[inventory_hostname]' database

Output:

localhost | SUCCESS => {
    "hostvars[inventory_hostname]": {
        "ansible_check_mode": false,
        "ansible_connection": "local",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 5,
        "ansible_inventory_sources": [
            "/root/test-ansible-vault/hosts"
        ],
        "ansible_playbook_python": "/usr/bin/python2",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.6.1",
            "major": 2,
            "minor": 6,
            "revision": 1,
            "string": "2.6.1"
        },
        "group_names": [
            "database"
        ],
        "groups": {
            "all": [
                "localhost"
            ],
            "database": [
                "localhost"
            ],
            "ungrouped": []
        },
        "inventory_dir": "/root/test-ansible-vault",
        "inventory_file": "/root/test-ansible-vault/hosts",
        "inventory_hostname": "localhost",
        "inventory_hostname_short": "localhost",
        "mysql_host": "10.0.0.3",
        "mysql_password": "supersecretpassword",
        "mysql_port": 3306,
        "mysql_user": "fred",
        "omit": "__omit_place_holder__61b455126974829d8835bf3b4379124c58282465",
        "playbook_dir": "/root/test-ansible-vault"
    }
}

The output confirms that all of the variables we set up are applied to the host. However, our group_vars/database file currently holds all of our variables. This means we can either leave it unencrypted, which is a security concern because of the database password variable, or we encrypt all of the variables, which creates usability and collaboration issues.

14. Moving Sensitive Variables into Ansible Vault

To solve this issue, we need to make a distinction between sensitive and nonsensitive variables. We should be able to encrypt confidential values and at the same time easily share our nonsensitive variables. To do so, we will split our variables between two files.

14.1 It is possible to use a variable directory in place of an Ansible variable file in order to apply variables from more than one file. We can refactor to take advantage of that ability. First, rename the existing file from database to vars. This will be our unencrypted variable file:

mv group_vars/database group_vars/vars

14.2 Next, create a directory with the same name as the old variable file. Move the vars file inside:

mkdir group_vars/database
mv group_vars/vars group_vars/database/

14.3 We now have a variable directory for the database group instead of a single file and we have a single unencrypted variable file. Since we will be encrypting our sensitive variables, we should remove them from our unencrypted file. Edit the group_vars/database/vars file to remove the confidential data:

cat > group_vars/database/vars <<EOF
---
# nonsensitive data
mysql_port: 3306
mysql_host: 10.0.0.3
mysql_user: fred
EOF

14.4 Create a vault-encrypted file within the directory that will live alongside the unencrypted vars file:

ansible-vault create group_vars/database/vault

14.5 In this file, define the sensitive variables that used to be in the vars file. Use the same variable names, but prepend the string vault_ to indicate that these variables are defined in the vault-protected file:

i
---
vault_mysql_password: supersecretpassword

14.6 The resulting directory structure looks this:

tree

.
.
├── ansible.cfg
├── encrypt_me.txt
├── group_vars
│   └── database
│       ├── vars
│       └── vault
├── hosts
├── secret_key
└── vault.yml

At this point, the variables are separate and only the confidential data is encrypted. This is secure, but our implementation has affected our usability. While our goal was to protect sensitive values, we’ve also unintentionally reduced visibility into the actual variable names. It is not clear which variables are assigned without referencing more than one file, and while you may wish to restrict access to confidential data while collaborating, you still probably want to share the variable names.

To address this, the Ansible project generally recommends a slightly different approach.

a. Referencing Vault Variables from Unencrypted Variables

When we moved our sensitive data over to the vault-protected file, we prefaced the variable names with vault_ (mysql_password became vault_mysql_password). We can add the original variable names (mysql_password) back to the unencrypted file. Instead of setting these to sensitive values directly, we can use Jinja2 templating statements to reference the encrypted variable names from within our unencrypted variable file. This way, you can see all of the defined variables by referencing a single file, but the confidential values will remain in the encrypted file.

14.7 To demonstrate, open the unencrypted variables file again:

cat > group_vars/database/vars <<EOF
---
# nonsensitive data
mysql_port: 3306
mysql_host: 10.0.0.3
mysql_user: fred

# sensitive data
mysql_password: "{{ vault_mysql_password }}"
EOF

The mysql_password variable will be set to the value of the vault_mysql_password variable, which is defined in the vault file.

With this method, you can understand all of the variables that will be applied to hosts in the database group by viewing the group_vars/database/vars file. The sensitive parts will be obscured by the Jinja2 templating. The group_vars/database/vault only needs to be opened when the values themselves need to be viewed or changed.

14.8 You can check to make sure that all of the mysql_* variables are still correctly applied using the same method as last time.

ansible -m debug -a 'var=hostvars[inventory_hostname]' database

Output:

localhost | SUCCESS => {
    "hostvars[inventory_hostname]": {
        "ansible_check_mode": false,
        "ansible_connection": "local",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 5,
        "ansible_inventory_sources": [
            "/root/test-ansible-vault/hosts"
        ],
        "ansible_playbook_python": "/usr/bin/python2",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.6.1",
            "major": 2,
            "minor": 6,
            "revision": 1,
            "string": "2.6.1"
        },
        "group_names": [
            "database"
        ],
        "groups": {
            "all": [
                "localhost"
            ],
            "database": [
                "localhost"
            ],
            "ungrouped": []
        },
        "inventory_dir": "/root/test-ansible-vault",
        "inventory_file": "/root/test-ansible-vault/hosts",
        "inventory_hostname": "localhost",
        "inventory_hostname_short": "localhost",
        "mysql_host": "10.0.0.3",
        "mysql_password": "{{ vault_mysql_password }}",
        "mysql_port": 3306,
        "mysql_user": "fred",
        "omit": "__omit_place_holder__988df6642d052dadf03ab02b114fbea605ed6ef6",
        "playbook_dir": "/root/test-ansible-vault",
        "vault_mysql_password": "supersecretpassword"
    }
}

Both the vault_mysql_password and the mysql_password are accessible. This duplication is harmless and will not affect your use of this system.

14.9 Exit from the pod18-master.origin.com node to be reset from vault:

exit

Output:

logout
Connection to pod18-master.origin.com closed.