There are two types of SSH key distribution discussed in this post: private keys on local hosts and public keys on remote hosts. SSH private key distribution is best used for setting up your own workstation or possibly an Ansible Tower server. In general, you should not be distributing private keys widely; with a good SSH tunneling configuration and SSH public key distribution, there should be no need for the private keys to be installed in more than few places. This configuration will show off a technique for configuring an SSH jump host bastion that allows you to keep your private key on your own workstation; there is no need to have the SSH private key on the bastion host.

For the purpose of this post, I have generated a new SSH key pair to demonstrate this technique; this keypair is used nowhere. Part of the trick to making this work is that the private key needs to be base64 encoded so that line breaks are preserved when it is stored as a yaml string in the vars_files. Template files are created for the public and private keys to preserve file change detection.

Generate a new key pair:

$ ssh-keygen -b 2048 -f junk_key -C junk
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in junk_key.
Your public key has been saved in junk_key.pub.
The key fingerprint is:
94:94:ae:ac:c3:5e:ee:7d:fa:2c:cb:0f:ae:19:c8:99 junk
The key's randomart image is:
+--[ RSA 2048]----+
|        ..       |
|       ...       |
|       .o        |
|       ..        |
|     . .S        |
|   . +o          |
|   .E.o .        |
|    +o *.o.      |
|   ..o=.*B+      |
+-----------------+

Base64 encode the private key:

base64 -i junk_key > junk_key.b64

Create an Ansible vars_files yaml data file named ssh_keys/ssh_key_vault.yml. The ssh_private_key variable should contain the base64 encoded private key and the ssh_public_key variable should contain the public key. Encrypt the file with ansible-vault:

ssh_private_key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeUtIRlhKWFJweXlCV2FobGExM2I5S2t1aGlwSHNkVkR4dDhaZnJrMWpqd1NnNEhCCmEzMzBnQnBxUFk1SkVxeEtPV0F3WWZleExwZ3VFcHk0Z2o5S1JxZGxTb0lsYllWbEtaUnY4RmhRSC9iT3lIKzAKVytJb0VzZ096MjR6U1ZQRU9ybWV6d3QzMzN0OWh0NDFsWVBBTHpzbkVaem9vVWE4ZTVKc1RzT0YzQzdmaUh3NApBSXZTOStWVUp5Mm8wUnZQN2ZMQkttV0FBN2dvWVA3d1Z6aVNQbEVrVVJIRGEyNXBVTmRTU1lxQzI2Y0c0UWNPCk14Q3VOeXdFRks0TGl5Q21zcHNXSnkzV3BkQ1FYQ0k1Q0J0SUVVTnR6Y1FpTFQvd0ZwbzRpRnp4NEREbkRsZWcKdkc2L1JHelFqMVJyeWRFdCtTNVdHenMzYkJHbDgxOWMvSmpPNVFJREFRQUJBb0lCQUYyQXZ5a3lEWDVheUlIUApjRXpFZG5Fa3M3RUZYVnBzcU9Tekx2K1hNM1Z4VzdOOE1uZDFRUkMrdnNxbldEamlvTWp5b2puV0pQWXhLQysyCmFHc1RNZnVSb2l4Q1VVMGtnUXdLeU14N2JBUXBreDl3SE05QnJDbHNvVEpkQ252ZkZUSEZObFVKNURqOEpYbEkKY0RLWkwyVVRyVmFSQ1AyNHFManllWldQbkFBTDZPc1JwSUc1Ukp1ays2QTVmVEppL0FVTmp4a2FIM1VOUklmTwpMZjYwOVJIUHZKUEtPNkNnNWVzK3RSY0VlbnR6ZVJxeENkYkh1b2NESjluUWNRQjVIVVBaeVdYOGwzODhJQ1hhCm5oaCt6VUhlYWY4cEM1dE9STGh0aDdsR1FFN3NOQ1FQRkovMHVCVWhleEVWREwxcU1Vd2JRZUpFU2orUmMrMi8KazRZeFhEVUNnWUVBNjhaNzRGUlJuUmViZy8xT2Z2K3ZlY0p2akEyRi9oWTBDVkpBOHdTcHdpNTlHTUpUS3g1TQpFWCtwNHBhMlY3NDZTZFFjY2l4K3hlWlhyZXkvbXhLWlByZnE0bVl1ZkJRRmVPT2tuWWdXTEhXUjV2cW1zUkFwCmZMQlhTbGdZNzV6SGFzSWJmOVlQZDhZQytLV1J4RlZyQml4eEtPQ2o1WFlrZmhoZkFnaDJsNnNDZ1lFQTJkZU4KM3FvR1lDM09GdllmWTJKVTQ4a2RvejNuT09uN09rd1pHNTFZOW5GM3JTYWpUeW9XRHpLTzc5MUNtK0hGNmhFWgpBWFlzeDlTcXJER1JYT2lvUi9ZMEpNVDZsS0ZuTUJpWERmRWROUVFCYStQQ3RFNWhqdTFoS1dPOHlIN21pZk1DCnQ0OHZBbGk5NEQxZjNxa2FjREtmRWVpa2VaazZaWWFhUWFTZ1k2OENnWUVBb0cxb3dzWjgxZWhIVURNZW96bDAKKytONkpSRGFtSDRoSUNxUXVRcjJPNE9JYVQxb2U5RmNyeGR2MEJiK3NZdGxlL0RRL2pzYWM2djlBd0l4aWVISQoxaTBzcktvY2ZSN2VibGh2SFNXSStPMXl2bmpVeld3UzNwM2FkMktrYlAzL2pydlBIRmZhSklSZVp6TzVrSjhTCmVKdnF6NGF5M3FKWnlGYnE1cVk5azRzQ2dZQmlzNVhtTTJkY0lLVG1KbkltVjZGYTYvN3Z2ZGFNSlFmZGJDbGMKSjdqdFFKQVc5aEM4aDdjaS82ZGY2d0tKR296UDl4czdYRTRCNU12SDVWV1ZvUnpPTGpHR0QzSHg4Z2VNOVRkTAo2OWx0OGZpcTU3R0tmSkViYjFhOHFDSWJQZFE2NE01MFdQM1Z0RnVqeEdzeHViRHU4U0M5dm9qM1I0UDhDRGJRClUwVVFwUUtCZ1FDc0pyMlBrdFl0QWJteHdCbUMrT1FVOFd4dHQwZ2ZoNGluZHpVV2J3MFZWY3Ivb3A5aDliekoKWG9TSVVSenQwUjZmNVVDK1RwQUdxY3ZPKzhtTUgwbnZpZ2VuYkhXdk01WFBuSUtwR2RLZmc4QkxUMUZnR2t3Ugpma2tydnpwWjM1dWpCTkRWdnl4ZWVCTyt2MTVqc0N1YTFTS0FsaXpzaWJrdE9lM1F1VTkwS3c9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

ssh_public_key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIocVcldGnLIFZqGVrXdv0qS6GKkex1UPG3xl+uTWOPBKDgcFrffSAGmo9jkkSrEo5YDBh97EumC4SnLiCP0pGp2VKgiVthWUplG/wWFAf9s7If7Rb4igSyA7PbjNJU8Q6uZ7PC3ffe32G3jWVg8AvOycRnOihRrx7kmxOw4XcLt+IfDgAi9L35VQnLajRG8/t8sEqZYADuChg/vBXOJI+USRREcNrbmlQ11JJioLbpwbhBw4zEK43LAQUrguLIKaymxYnLdal0JBcIjkIG0gRQ23NxCItP/AWmjiIXPHgMOcOV6C8br9EbNCPVGvJ0S35LlYbOzdsEaXzX1z8mM7l junk
ansible-vault encrypt ssh_keys/ssh_key_vault.yml
Vault password: 
Confirm Vault password: 
Encryption successful

Create an inventory file named inventory, showing off the SSH jump host connection capability:

[localhost]
localhost ansible_connection=local

[group-all:children]
group-01
group-02

[group-01]
i-00000001 ansible_ssh_host=bastion+192.168.1.1 ansible_ssh_user=remoteuser
i-00000002 ansible_ssh_host=bastion+192.168.1.2 ansible_ssh_user=remoteuser

[group-02]
i-00000003 ansible_ssh_host=bastion+192.168.1.3 ansible_ssh_user=remoteuser
i-00000004 ansible_ssh_host=bastion+192.168.1.4 ansible_ssh_user=remoteuser

Create a template file for the private key named templates/HOME_.ssh_junk:

{{ssh_private_key_decoded.stdout}}

Create a template file for the public key named templates/HOME_.ssh_junk.pub:

{{ssh_public_key}}

Create a template file for the SSH jump host configuration named templates/HOME_.ssh_config:

Host *
    ServerAliveInterval 30
    ServerAliveCountMax 5

Host bastion
    User {{remote_user}}
    IdentityFile ~/.ssh/junk
    Hostname bastion
 
Host bastion+*
    User {{remote_user}}
    IdentityFile ~/.ssh/junk
    ProxyCommand ssh -T -a bastion nc $(echo %h |cut -d+ -f2) %p 2>/dev/null
    StrictHostKeyChecking no

This configuration assumes that you have a consistent remote username defined on the bastion server and your protected hosts.

Write a playbook to install the SSH key and configuration on your local workstation named config_local-ssh.yml:

- name: configure local ssh
  hosts:
  - localhost
  gather_facts: false
  sudo: false
  vars:
    local_home: "{{ lookup('env','HOME') }}"
    local_user: "{{ lookup('env','USER') }}"
    remote_user: remoteuser
  vars_files:
  - ssh_keys/ssh_key_vault.yml
  tasks:
  - file: path={{local_home}}/.ssh state=directory mode=0700 owner={{local_user}}

  - template: src=templates/HOME_.ssh_config dest={{local_home}}/.ssh/config mode=0644 owner={{local_user}} backup=yes

  - shell: echo {{ssh_private_key}} |base64 --decode
    register: ssh_private_key_decoded

  - template: src=templates/HOME_.ssh_junk dest={{local_home}}/.ssh/junk mode=0600 owner={{local_user}}

  - template: src=templates/HOME_.ssh_junk.pub dest={{local_home}}/.ssh/junk.pub mode=0644 owner={{local_user}}

Run the playbook to setup your local workstation with SSH keys and configuration:

ansible-playbook -i inventory config_local-ssh.yml --ask-vault-pass
Vault password:

Test your SSH tunneling access to a remote host behind the bastion server:

ssh bastion+192.168.1.1
ssh bastion+192.168.1.1 "date; date > /tmp/date.out"
scp bastion+192.168.1.1:/tmp/date.out .

Notice that the hosts behind the bastion server are referenced in the SSH command the same way that they are referenced in the Ansible inventory file. The “+” character used as a separator was selected explicitly for its ability to be used interchangeably at the command line and in the Ansible inventory. IP addresses are being used on the right hand side of the expression since the secondary connection to the protected host relies on the name resolution capabilities of the first host in the tunnel. If you had a reliable dynamic DNS service that was keeping up with changes to the protected hosts and was accessible to the bastion host, then you could use host names instead, such as bastion+webserver01. This host selection technique can be extended to an Ansible dynamic inventory script, if you were running instances at a cloud provider such as AWS. When you write a dynamic inventory script, the data format should look like this:

{
    "group-01": {
        "hosts": [
            "i-00000001", 
            "i-00000002"
        ]
    }, 
    "group-02": {
        "hosts": [
            "i-00000003", 
            "i-00000004"
        ]
    }, 
    "group-all": {
        "children": [
            "group-01", 
            "group-02"
        ]
    },
    "localhost": [
        "localhost"
    ], 
    "_meta": {
        "hostvars": {
            "i-00000001": {
                "ansible_ssh_host": "bastion+192.168.1.1",
                "ansible_ssh_user": "remoteuser"
            },
            "i-00000002": {
                "ansible_ssh_host": "bastion+192.168.1.2",
                "ansible_ssh_user": "remoteuser"
            }, 
            "i-00000003": {
                "ansible_ssh_host": "bastion+192.168.1.3",
                "ansible_ssh_user": "remoteuser" 
            }, 
            "i-00000004": {
                "ansible_ssh_host": "bastion+192.168.1.4",
                "ansible_ssh_user": "remoteuser"
            }, 
            "localhost": {
                "ansible_connection": "local"
            },
        }
    }
}                        

The nice thing about this style of SSH configuration is that you can have multiple bastion hosts in different locations and target the hosts behind each of them, provided that you give your bastion hosts different names. The method of accessing them is the same between direct SSH connections and Ansible execution. Once you have this infrastructure in place, you can start distributing public SSH keys to your protected hosts.

Write a templates/etc_sudoers file that grants NOPASSWD access to the sudo group:

Defaults    env_reset
Defaults    mail_badpass
Defaults    secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=NOPASSWD: ALL

#includedir /etc/sudoers.d

Write a playbook update_remote-ssh.yml to configure NOPASSWD sudo access for your remote user and distribute SSH public keys on your remote hosts. This will allow subsequent playbook execution to operate more easily against your remote hosts. In order for this to work, the paramiko connection type must be used initially, so that the password can be requested once and re-used across all hosts.

- name: update remote ssh
  hosts:
  - group-all
  gather_facts: false
  sudo: true
  connection: paramiko
  vars_files:
  - ssh_keys/ssh_key_vault.yml
  tasks:
  - copy: src=templates/etc_sudoers dest=/etc/sudoers mode=0440 owner=root group=root

  - user: name=remoteuser groups=sudo shell=/bin/bash state=present

  - authorized_key: user=remoteuser state=present key={{ssh_public_key}}

Run an SSH configuration playbook against remote hosts through the SSH tunnel, providing the SSH password, sudo password and vault password:

ansible-playbook -i inventory update_remote-ssh.yml --ask-pass --ask-sudo-pass --ask-vault-pass
SSH password:
sudo password [defaults to SSH password]: 
Vault password:

PLAY [update ssh] ************************************************************* 

TASK: [copy src=templates/etc_sudoers dest=/etc/sudoers mode=0440 owner=root group=root] *** 
ok: [i-00000001]
ok: [i-00000002]
ok: [i-00000003]
ok: [i-00000004]

TASK: [user name=remoteuser groups=sudo shell=/bin/bash state=present] *** 
ok: [i-00000001]
ok: [i-00000002]
ok: [i-00000003]
ok: [i-00000004]

TASK: [authorized_key user=remoteuser state=present key="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIocVcldGnLIFZqGVrXdv0qS6GKkex1UPG3xl+uTWOPBKDgcFrffSAGmo9jkkSrEo5YDBh97EumC4SnLiCP0pGp2VKgiVthWUplG/wWFAf9s7If7Rb4igSyA7PbjNJU8Q6uZ7PC3ffe32G3jWVg8AvOycRnOihRrx7kmxOw4XcLt+IfDgAi9L35VQnLajRG8/t8sEqZYADuChg/vBXOJI+USRREcNrbmlQ11JJioLbpwbhBw4zEK43LAQUrguLIKaymxYnLdal0JBcIjkIG0gRQ23NxCItP/AWmjiIXPHgMOcOV6C8br9EbNCPVGvJ0S35LlYbOzdsEaXzX1z8mM7l junk"] *** 
ok: [i-00000001]
ok: [i-00000002]
ok: [i-00000003]
ok: [i-00000004]


PLAY RECAP ******************************************************************** 
i-00000001               : ok=3    changed=0    unreachable=0    failed=0   
i-00000002               : ok=3    changed=0    unreachable=0    failed=0   
i-00000003               : ok=3    changed=0    unreachable=0    failed=0   
i-00000004               : ok=3    changed=0    unreachable=0    failed=0   

The best way to keep Ansible output concise is to run without verbosity – only crank this up if you need it to diagnose a problem.



blog comments powered by Disqus

Published

30 June 2014

Category

ansible

Tags