Managing a Jenkins Docker Infrastructure: Docker Garbage Collector

Using Docker for Continuous Delivery is great. It brings development teams an impressive flexibility, as they can manage environments and test resources by themselves, and, at same time, enforce clean isolation with other teams sharing the same host resources.

But a side effect on enabling Docker on build infrastructure is disk usage, as pulling various Docker images consumes hundreds megabytes. The layered architecture of Docker images ensures that you'll share the lower level layers as much as possible. However, as those layers get updated with various fixes and upgrades, the previous ones remain on disk, and can result, after few months, in huge disk usage within

/var/lib/docker

Jenkins monitors can alert on disk consumption on build executors. However, a more proactive solution should be implemented versus simply making the node offline until administrator handle the issue "ssh-ing" to the server.

Docker does not offer a standard way to address image garbage collection, so most production teams have created their own tool, including folks at Spotify who open-sourced docker-gc script .

On a Jenkins infrastructure, a scheduled task can be created to run this maintenance script on all nodes. I did it for my own usage (after I had to handle filesystem full error). To run the script on all docker enabled nodes, I'm using a workflow job. Workflow make it pretty trivial to setup such a GC.

The script I'm using relies on a "docker" label to be used on all nodes with docker support.

Jenkins.instance.getLabel("docker").nodes 

returns all the build nodes with this label, so I can iterate on them and run a workflow

node() 

block to execute the docker-gc script within a sh shell script command:

def nodes = Jenkins.instance.getLabel("docker").nodes
for (n in nodes) {
node (n.nodeName) {
      sh 'wget -q -O - https://raw.githubusercontent.com/spotify/docker-gc/master/docker-gc | bash'
    }
 }

docker-gc script do check images not used by a container. When an image existed last run of the script, but is not used by a container,

I hope that the Docker project will soon release an official docker-gc command. This will benefit to infrastructure teams, eliminating the need to re-invent custom solutions to the same common issue.

Nicolas De Loof
Hacker
CloudBees