August 31, 2017

Use Ansible to apply rolling updates to Docker swarm nodes

Even with all those elastic superscaling whatsoever buzzword stuff, sometimes it just comes down to simple upgrading a component like Docker daemon itself or a new kernel.

With swarm, this means draining the node, do our stuff, probably reboot and then set it active again.

Being ephemeral is cool, so we could spawn new (updated) swarm members using Ansible and then destroy the old hard-working bees afterwards. But to me, this seems more like a “just because we can”, even more then running small or medium sized swarms/clusters.

Using my dynamic Ansible inventory for Docker swarm, we could just run any playbook against an existing swarm. Upgrading now is just simple playbook like this (please consider this as an exmaple only…):

---

- hosts: '{{ target }}'
  serial: 1

  tasks:

   - name: docker info
     command: docker info
     register: docker_info

  - name: drain docker node
    environment:
      DOCKER_HOST: "tcp://my-swarm.example.com:2375"
    local_action: shell docker node update --availability drain '{{ inventory_hostname }}'

  - name: apt dist-upgrade
    apt:
      update_cache: yes
      upgrade: dist
      autoremove: yes

  - name: reboot
    become: yes
    shell: sleep 2 && /sbin/shutdown -r now "Ansible reboot"
    async: 1
    poll: 0
    ignore_errors: true

  - name: waiting for server to come back
    local_action: wait_for host={{ ansible_default_ipv4.address }} port=22 state=started delay=30 timeout=240
    become: false

   - name: wait for swarm manager
     wait_for:
       host: "{{ inventory_hostname }}"
       port: "2377"
       delay: 5
       state: started
       timeout: 1200
     when: "docker_info.stdout | search('Is Manager: true')"

  - name: activate docker node
    environment:
      DOCKER_HOST: "tcp://my-swarm.example.com:2375"
    local_action: shell docker node update --availability active '{{ inventory_hostname }}'

The serial: 1 takes care of just one host after another, so the swarm can take care of spawning new services on other hosts to fulfil the requirements.

I like using a variable for the target hosts, so calling this one using my dynamic inventory with all workers only would be:

ansible-playbook -i /etc/ansible/docker-swarm-nodes.py -e target=worker update.yml

As always, this is just a hint, of course you should have secured access to your swarm.

Also one could use pre/post tasks in Ansible!