December 24, 2024

Clustered multiarchitecture docker buildx nodes

When building multiarchitecture container images, the most common setup is using QEMU. While this works quite well, it also performs quite slow due to the required virtualization layer involved.

Most builds today are targeting linux/amd64 and linux/arm64 thanks to the availability of client and server hardware getting better (MacBooks, Ampere, Graviton, …) and of course the Raspberry Pi 😀.

If you have access to multiple nodes using different architecture but want to create a single multiarch image without stitching 2 independent images together (docker buildx imagetools create), you can use a cluster of multiple nodes to do so.

Setup

  1. Setup secure remote access and expose1 the docker daemon on the target node
  2. Create a new context using the certs from previous step:
    docker context create remote --docker "host=tcp://${DOCKER_REMOTE}:2376,ca=${DOCKER_CA},cert=${DOCKER_CERT},key=${DOCKER_KEY}"
    
  3. Create a new build instance on the local machine:
    docker buildx create --name cluster --node local --platform linux/amd64
    
  4. Attach the remote node to the local builder using it’s context
    docker buildx create --name cluster --node remote --use remote --append
    
  5. To use the clustered builder:
    docker buildx use cluster
    
  6. Prepare the build nodes w/ buildkit:
    docker buildx inspect --bootstrap
    
  7. Check setup:
    docker buildx ls
    
    Resulting output should look like this:
    NAME/NODE        DRIVER/ENDPOINT                   STATUS    BUILDKIT   PLATFORMS
    cluster*         docker-container                                       
     \_ local         \_ unix:///var/run/docker.sock   running   v0.18.2    linux/amd64* (+3)
     \_ remote        \_ remote                        running   v0.18.2    linux/arm64, linux/arm (+2)
    default          docker                                                 
     \_ default       \_ default                       running   v0.17.3    linux/amd64 (+3)
    remote           docker                                                 
     \_ remote        \_ remote                        running   v0.17.3    linux/arm64, linux/arm (+2)
    
  8. Build the image (example):
    docker buildx build \
    --provenance=false \
    --platform linux/amd64,linux/arm64 \
    --push \
    -t your-multiarch-image
    .
    

  1. Add systemd dropin:

    mkdir -p /etc/systemd/system/docker.service.d/
    cat > /etc/systemd/system/docker.service.d/00-tls.conf<<EOF
    [Service]
    ExecStart=
    ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem -H=0.0.0.0:2376
    EOF
    
     ↩︎