Today i was stumbling upon ZFS in Docker Docs.

The ZFS on Linux (ZoL) port is healthy and maturing. However, at this point in time it is not recommended to use the zfs Docker storage driver for production use unless you have substantial experience with ZFS on Linux.

I do have, so let’s try things out. I’m not installting any Ubuntu ZFS thingy, just grabbed one of my lab Proxmox installations with ZFS builtin. In this test, there was no running Docker daemon before the next steps. If you already are running Docker, you should stop the service, rename (or delete) /var/lib/docker and continue.

Create a dataset for Docker

zfs create -o mountpoint=/var/lib/docker -p rpool/data/docker

I’m using the default rpool, as this is just a test setup. For production use, you might think about another dedicated zpool.

Configure Docker to use ZFS

root@lab:~# mkdir -p /etc/systemd/system/docker.service.d
root@lab:~# cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --storage-driver=zfs
EOF

Adjust to your needs.

Start Docker and check storage driver

root@lab:~# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: zfs
 Zpool: rpool
 Zpool Health: ONLINE
 Parent Dataset: rpool/data/docker
 Space Used By Parent: 98304
 Space Available: 82527678464
 Parent Quota: no
 Compression: lz4
...

Pull an image and check ZFS

root@lab:~# docker pull ubuntu:xenial
xenial: Pulling from library/ubuntu
aed15891ba52: Pull complete 
773ae8583d14: Pull complete 
d1d48771f782: Pull complete 
cd3d6cd6c0cf: Pull complete 
8ff6f8a9120c: Pull complete 
Digest: sha256:35bc48a1ca97c3971611dc4662d08d131869daa692acb281c7e9e052924e38b1
Status: Downloaded newer image for ubuntu:xenial
root@lab:~# zfs list -r rpool/data/docker 
NAME                                                                                 USED  AVAIL  REFER  MOUNTPOINT
rpool/data/docker                                                                   90.4M  76.7G   852K  /var/lib/docker
rpool/data/docker/03cbcae48dd4a769bd9709791710f22f6ae594a4e6210fd06cf4a689e1db3f4c   112K  76.7G  83.1M  legacy
rpool/data/docker/2e06fa8a3967dff7453fda15fdfdcaf60189bf91a372b782c002a9341c6dae2c   160K  76.7G  83.1M  legacy
rpool/data/docker/38207bb42aebd11dee6a3883263f8c29fb21817d90de0ad7e5db9360dc6a5729   228K  76.7G  88.9M  legacy
rpool/data/docker/c20becbd3210cd4884ccf89561f0f8d05fb2fc1e4e1d2881f9de321cc9153f41   132K  76.7G  83.1M  legacy
rpool/data/docker/f9c1b3b86b5a8fed9be094874fd8b759cdad095b64b7680162cdeb2bcdcddc48  88.9M  76.7G  88.9M  legacy

As you can see, each single layers gets an own dataset.

Run a container, do something, check ZFS

root@lab:~# docker run -it ubuntu:xenial /bin/bash
# ... do something inside container and exit
root@lab:~# zfs list -r rpool/data/docker 
NAME                                                                                      USED  AVAIL  REFER  MOUNTPOINT
rpool/data/docker                                                                         399M  76.5G   952K  /var/lib/docker
rpool/data/docker/03cbcae48dd4a769bd9709791710f22f6ae594a4e6210fd06cf4a689e1db3f4c        112K  76.5G  83.1M  legacy
rpool/data/docker/2e06fa8a3967dff7453fda15fdfdcaf60189bf91a372b782c002a9341c6dae2c        160K  76.5G  83.1M  legacy
rpool/data/docker/38207bb42aebd11dee6a3883263f8c29fb21817d90de0ad7e5db9360dc6a5729        228K  76.5G  88.9M  legacy
rpool/data/docker/576a9aa247d01a90ec15ad397be9317bd910e20e3f8479c5190881662f82ff8e        308M  76.5G   383M  legacy
rpool/data/docker/576a9aa247d01a90ec15ad397be9317bd910e20e3f8479c5190881662f82ff8e-init   152K  76.5G  83.1M  legacy
rpool/data/docker/c20becbd3210cd4884ccf89561f0f8d05fb2fc1e4e1d2881f9de321cc9153f41        132K  76.5G  83.1M  legacy
rpool/data/docker/f9c1b3b86b5a8fed9be094874fd8b759cdad095b64b7680162cdeb2bcdcddc48       88.9M  76.5G  88.9M  legacy

Now you can see the container dataset 576a9aa247d01a90ec15ad397be9317bd910e20e3f8479c5190881662f82ff8e. If you remove the container instance, it will disappear.

Awesome!