Builds aren't isolated

[rafecolton]

Builds on the same build server aren't isolated from each other. This could definitely be a problem if we build a bunk version of a container upon which another container depends. Addressing this will be quite involved and may involve a re-architect of the whole build server system.

Welcoming suggestions on the design for this.

[rafecolton]

Possible solution: docker inside of docker, run the builder in the sub container, run the worker in the main container (with --privileged)

[rafecolton]

use https://github.com/jpetazzo/dind

[rafecolton]

So here is a proposition for a solution may work:

When run inside of a container, docker-builder could have two different components: a leader and a follower. The leader runs on the host machine, unprivileged. It connects to a docker socket by $DOCKER_HOST (mounting /var/run if necessary). When a request comes in, the leader container starts up a follower container to handle the actual build. The follower container must run with --privileged and must run dind.

Both pieces of functionality have been tested manually, and they do indeed work, but some questions will need to be answered about the implementation.

• How does the leader communicate with the follower? The leader could be responsible for cloning down the repo and mounting the directory onto the follower, but this would require them to run on the same host, which would not be necessary otherwise.
• It could pass the equivalent data as a command-line argument (the follower containers wouldn't need to run the server, as they will be processing a one-time request). However, something about that doesn't feel quite right. Maybe that's just me though.
• The two could communicate over HTTP. If we're communicating over HTTP, maybe the leader container could send the entire container as a tarball over HTTP. This would be good from the perspective of isolating the responsibilities between the leader and the follower. However, I don't know that there is a good, easy-to-use public implementation of tar/gz for Go other than the one in Docker itself, which doesn't look like it's meant to be extracted.

[meatballhat]

s/master/leader/g; s/slave/follower/g </nit>

[meatballhat]

If possible, I would want the follower to receive everything it needs to complete the work in a single payload, including any information about where to send status updates. It's REST mannnnn. To be clear, I do not think that it's necessary for the container to be sent as a tarball over HTTP. I'd say follow the model of Travis CI and other CI services and ensure the follower containers are single-job only, wiped from existence upon completion.

[rafecolton]

@meatballhat somebody's been reading django pull requests

Do you think the communication between the leader and follower should happen over HTTP, maybe with a flag to indicate that the follower should exit after the build? Or do you think the data should be passed as an arg to the follower ENTRYPOINT (i.e. docker run modcloth/docker-builder-follower '{"a_bunch": "of json"}')

[meatballhat]

I think the latter makes sense to me, as waiting for the follower to be ready to receive a payload over HTTP introduces another problem. If '{"a_bunch": "of json"}' gets to be too big, I think it would be fine to pass it via a file + volume mount.