foobars.log

Development

Automation instead of Documentation when Provisioning Servers

Chef and Puppet each have a bit of a learning curve, and if you only need to provision a few servers, you might think the effort to learn either isn’t worth it. I’ve found translating manual provisioning steps to Ansible Playbooks to be easy and a great step towards automated provisioning.

You can get far with just writing down your provisioning steps, manually executing them, imaging the resulting server and using that image as the basis for your testing, staging, production and failover. Besides, now you know that these environments are exactly the same, because they are from the same image.

Using images for your deployments is not the problem. In fact, you should be making images and skipping the provisioning step when deploying multiple instances of the same application on a service like EC2. The problem is with relying solely on documentation for building an image from scratch.

With an automation tool, you can destroy and rebuild your local development and test environments quickly and make sure they match your production and staging environments command-for-command without any chance of “fat-fingering” or accidentally skipping a step. This can also make bootstrapping a new developer easier.

There are also a few scenarios where rebuilding your production server from scratch might be needed or is the better option:

  • Distribution upgrades – apt-get dist-upgrade works, but it’s probably best to start from scratch when upgrading the distro
  • Moving to a new hosting provider – not impossible to move an image, but sometimes can cause issues
  • Tweaks to your installed libraries or programs – a lot of time it’s best to isolate your tweaks with fresh rebuilds in case of conflicts
  • Re-architecting your cluster – started off with the DB on the same box as the app and now need the DB on its own server

Ansible could be an answer

Ansible is an automation tool that can be as involved or as simple as you want it to be. It stays installed on your local machine and communicates to the servers via SSH, so there is nothing to install on the servers. It’s like you logged in and performed the tasks yourself, and I feel that makes it easy to understand.

Another thing that makes Ansible easy to use is the fact that tasks are written in YAML instead of a block-based DSL like Chef. Take the following Ansible task for example:

copy: src=http://foobars.com/myfiles/foo.conf dest=/etc/foo.conf owner=foo group=foo mode=0644

An Ansible module has options passed to it much like a command-line tool has flags. The copy module will take a local file src and copy it to a remote server location dest and set the ownership and permissions mode on it. This one line replaces what could be three steps (scp, chown and chmod).

This post is just a preview

Without going too much in to how tasks, roles, playbooks, and modules all work together, I want to show you how you can turn a multi-step provisioning guide in to a simple playbook that can be run on a server with single command. Hopefully this will encourage you to learn more about Ansible and ditch the provisioning documentation for automation.

The steps I will be turning in to an Ansible playbook are from the DigitalOcean tutorial on how to add swap to Ubuntu. I will be using the shell, file, and lineinfile Ansible modules as well as registering a conditional.

Step 1: check for swap space

To check for swap, run the command sudo swapon -s. It should output the results as either a empty table (with only headers) or a list of swaps.

Empty:

Filename     Type   Size    Used    Priority

With a swap:

Filename     Type   Size    Used    Priority
/swapfile    file   262140  0       -1

To make this command more boolean friendly, we can run it through grep. If there is no output, there is no swap.

sudo swapon -s | grep -E "^/"

In an Ansible task, we can use the shell module to run this command. Ansible tasks can also “register” the result of the module to a variable we can use elsewhere. However, this particular case is a little different. Since there is a possibility of the command failing to return any output (remember, no output means no swap currently on the system), we need to ignore the failing error.

shell: 'sudo swapon -s | grep -E "^/"'
register: swapfile
ignore_errors: yes

Now we only have to check for a swap once. We are doing this because we don’t want to run the following steps if there is already a swap on the system.

Step 2: create and enable a swap file

The next command to run is sudo dd if=/dev/zero of=/swapfile bs=1024 count=256k which will create an empty swapfile at the location /swapfile.

With Ansible, we will use the shell module again to run this command, but with one catch: we only want to run the command “when” ansible failed to find a swap file (using the variable swapfile we created in the previous task).

shell: 'sudo dd if=/dev/zero of=/swapfile bs=1024 count=256k'
when: swapfile|failed

Next is to prepare the swap file with the command sudo mkswap /swapfile and we will use the shell module yet again and with the same “when” conditional.

when: swapfile|failed
shell: 'sudo mkswap /swapfile'

Notice that the when is now on top. Order doesn’t matter.

And finally enable the swapfile with sudo swapon /swapfile, which we will translate to an Ansible task with the shell module again.

when: swapfile|failed
shell: 'sudo swapon /swapfile'

Step 3: mount the swap

Next, we want to add the following line to the file /etc/fstab.

/swapfile      none    swap    sw      0      0

For this, we will use a different module, the lineinfile module. This module will let us look through the file using a regular expression and make sure it’s present (or absent). If you provide a line= option to the module (required when checking if a line is present), it will insert/replace that line in the file. Since we only want one entry for /swapfile in /etc/fstab, this is the module we will use.

when: swapfile|failed
lineinfile: dest=/etc/fstab
  regexp="^/swapfile"
  state=present
  line="/swapfile       none    swap    sw      0       0"

Modules that take options can be written on new lines (as above) or all in one line (as below). The indention is not required in the above example, but I personally like the way the indention reads.

lineinfile: dest=/etc/fstab regexp="^/swapfile" state=present line="/swapfile       none    swap    sw      0       0"

Step 4: set the system “swappiness”

This will determine how aggressive the system will be about hitting the swapfile. With a setting of “1”, the system will swap only to avoid an out of memory condition (kernel 3.5+). If you run a kernel older than 3.5, use “0” for the same behavior. The kernel default value is “60”. Setting the value higher results in more aggressive swapping.

To set the swappiness temporarily we will use the shell module.

when: swapfile|failed
shell: 'echo 1 | sudo tee /proc/sys/vm/swappiness'

And to make it permanent so the preference will survive a reboot, we will use the lineinfile module again.

when: swapfile|failed
lineinfile: dest=/etc/sysctl.conf
  regexp="^vm.swappiness"
  state=present
  line="vm.swappiness = 1"

Step 5: set permissions on swap

Lastly, we need to make sure the permissions are set correctly on the swapfile. While we could just use the shell module again to execute the command in the DigitalOcean tutorial, I’d like to introduce the file module. This module sets attributes of files, symlinks, and directories, or removes files, symlinks, or directories. With the file module we can set the owner, group and permissions mode the file at the path /swapfile all in one task.

when: swapfile|failed
file: path=/swapfile
  owner=root
  group=root
  mode=0600

Putting it in a playbook

If you want to try out the following playbook, install Ansible with brew install ansible on a Mac, or with pip (more details about installation can be found in the Ansible documentation).

An Ansible playbook is just a YAML file with a little bit of ceremony. At the very least you need need a host and list of tasks. Also since the tasks in this playbook require sudo access, we need to note that.

- hosts: all
  sudo: yes

  tasks:
    - name: 'test for swap partition'
      shell: 'sudo swapon -s | grep -E "^/"'
      register: swapfile
      ignore_errors: yes

    - name: 'create swapfile'
      when: swapfile|failed
      shell: 'sudo dd if=/dev/zero of=/swapfile bs=1024 count=256k'

    - name: 'set swapfile permissions'
      when: swapfile|failed
      file: path=/swapfile
        owner=root
        group=root
        mode=0600

    - name: 'prepare swapfile'
      when: swapfile|failed
      shell: 'sudo mkswap /swapfile'

    - name: 'enable swap'
      when: swapfile|failed
      shell: 'sudo swapon /swapfile'

    - name: 'add swapfile'
      when: swapfile|failed
      lineinfile: dest=/etc/fstab
        regexp="^/swapfile"
        state=present
        line="/swapfile       none    swap    sw      0       0"

    - name: 'set swappiness (temporarily)'
      when: swapfile|failed
      shell: 'echo 1 | sudo tee /proc/sys/vm/swappiness'

    - name: 'set swappiness (permanent)'
      when: swapfile|failed
      lineinfile: dest=/etc/sysctl.conf
        regexp="^vm.swappiness"
        state=present
        line="vm.swappiness = 1"

The three dashes at the top of the file are not really required. They help some editors and processors note that it is a YAML file. I put them in out of habit. While YAML is simple format, it does have arrays (items in a YAML array start with a -). An Ansible playbook is an array (note - hosts: and everything else is indented.), and tasks:, within a playbook, is an array of tasks (each new tasks is denoted by - name: ... in the above playbook). The name for each task is not required, but I do like to add a name to the top of each task for documentation. This is what will scroll by when Ansible is running through the tasks.

There are a few thing we could do to DRY up this playbook such as extract variables like the swappiness value, moving all tasks to a role and redo how the conditional works, but that’s for another post.

Running the play

Normally, you would need a few other things to run a playbook such as an inventory file and a configuration file, but this playbook can be run with a single command-line.

ansible-playbook swapfile.yml -i 192.168.1.123,

Note the -i for passing in a server IP address or domain name instead of using an inventory file, and make sure there is a comma after the IP address or domain name (it’s a quirk).

Also, in the playbook above, the hosts value is set to “all.” That means when it runs, it will run the tasks on all the servers in your inventory list. For adding more servers inline, write them out in a comma separated list.

ansible-playbook swapfile.yml -i 192.168.1.123,192.168.1.124

The above command works if you SSH in to the server with the same username that log in to your local machine with, use ssh keys for authentication and have sudo access with no password required. You can add flags to the command if this ideal scenario isn’t the case.

ansible-playbook swapfile.yml -i 192.168.1.123, -u jonathan --ask-pass --ask-sudo-pass

The take away

Ansible has many built-in modules to make tasks simpler, more readable, and more uniform across different distribution families, but you can always fallback on directly translating your actions to one of a small handful of modules like shell, lineinfile and file. We could’ve used the mount module or the sysctl module in this guide. Be sure to skim through the Ansible module index to see if there is anything that will make your task creation simpler.

With a nice collection of modules and a very readable and rememberable syntax, Ansible can replace provisioning notes and command snippets with something almost as readable but with the power to be executed across many servers all at once with a single command, thus reducing the amount of time provisioning would normally take and facilitating in keeping all those servers in sync.