Containers for continuous integration
(in science)

Frederick “Erick” Matsen

http://matsen.fredhutch.org/
@ematsen

Welcome to
the future

✔ Code under version control
✔ Example inputs included in repository
✗ Code compiles
✗ Code runs on example inputs
✗ Code does the right thing on example inputs

What do the
pros do?

Check out pull requests for Rails
and click on the little ✔s and ✗s.

Uh oh, one of my builds is failing.

No cowsay????

Scientific workflows are complex

Docker enables us to bring together lots of working parts, unusual languages, and heavy dependencies.

Dockerfile

FROM ubuntu:trusty

RUN apt-get update -q && \
    apt-get install -y -q --no-install-recommends \
        ca-certificates \
        cowsay \
        git \
        make

RUN ln -s /usr/games/cowsay /usr/bin

CMD git clone https://github.com/matsen/p-neq-np.git && \
    cd p-neq-np && \
    make

Try this

git clone https://github.com/matsen/cowsay-build-env.git
cd cowsay-build-env/
docker build -t $USER/cowsay-build-env .
docker images
docker run -t $USER/cowsay-build-env
docker run -it $USER/cowsay-build-env /bin/bash

Exercise:

Exit from your container
Install nyancat with apt-get in your Dockerfile
Build your container (same command as before)
Run it from within your container (same as before)

Docker hub

Either push directly:

docker push $USER/cowsay-build-env

Or set up an automated build as I have for this project:

https://hub.docker.com/r/matsen/cowsay-build-env/

We could stop here!

In fact, I wrote a little shim that would post to our Slack channel with the results of an automated build.

But, there are tools that make the process smoother and more logical, such as Wercker.

wercker.yml

Specify the Docker image used for the build:

box: matsen/cowsay-build-env

Specify what the build should do:

build:
    steps:
        - script:
            name: build
            code: make

When pushed to Wercker, you get a build page like this.

More fun

Notify:

build:
    after-steps:
        - slack-notifier:
            url: $SLACK_URL
            channel: microbiome
            username: pplacer build

Upload docs to Github pages:

deploy:
    steps:
        - ematsen/gh-pages:
            token: $GITHUB_TOKEN
            repo: matsen/pplacer
            path: docs/_build/html

YAML is great*

* and a little fussy to debug. Use yamllint.com.

Containers for continuous integration (in science)