Ross Patterson's Blog - Posts tagged docker-compose

Reproducing Deployments with Docker-in-Docker

2021-04-09T00:00:00+00:00

Debugging deployment issues locally by falling down the Docker-in-Docker (DinD) rabbit hole.

TL;DR:

You can run a # dockerd daemon inside a Docker container which can be used with $ docker-compose ... to (almost) completely reproduce hosted deployment environments on one’s development laptop.

As noted in a recent post, I seem to have become the Docker wrangler on the team for my recent projects. Also recently, I’m getting the distinct impression that containerization has or is beginning to reach that special point in the life-cycle of a technology where it is sufficiently widely adopted to be misused and has been in use long enough for that misuse to hurt. This is one of those ironic signs of success and utility but knowing that doesn’t lessen the pain or cost of the individual case one may run into.

Most recently, I was dealing with problematically fragile container-based deployments where the team was getting bitten over and over again by deployment details that weren’t causing problems in our local container-based development environments but were causing problems when CD pushed our changes to real hosted deployments. Specifically, the hosted deployments were failing on such things as filesystem paths and IP addresses. To make things worse, the internal CI/CD pipeline was under-resourced and very slow making the inner loop of testing changes painfully slow and wasteful.

Before you think it, this is a project where there are many things that need to be re-worked to be in alignment with current best practices and thus less fragile: from how containers are used, to keeping CI/CD responsive, to how deployment is orchestrated. Deployments should not be sensitive to changes in filesystem paths and IP addresses, to be sure. The project was a Python 3 upgrade, however, and we’d done just about all there was appetite for in terms of cleanup work that wasn’t directly related to the upgrade. As software developers, it’s our job to figure out how to solve difficult technical problems, including the difficulty of getting sub-optimal technology usage to work.

At any rate, the tax we were paying for this deployment sensitivity was so high and was happening often enough that I decided it was past time to work on a way to try and reproduce the real hosted deployment environment more completely, preferably locally. Reproducing deployment hosts as thoroughly as possible without turning my development laptop in to the deployment hosts feels like an isolation issue, so I reached for containers and Docker. Specifically, I wanted to start with a container image as close to the VM image used by the deployment hosts as possible and figure out how to run the actual application container images within those “host containers”. IOW, can I run Docker containers within a Docker container? Yup, for some time now.

It turns out that Docker now provides an official Docker-in-Docker (DinD) image. After reading it’s ./Dockerfile and some blog posts detailing how to run DinD, I decided it would be more efficient and maintainable to use the official docker:dind image as my base image and extend it to match the project’s deployment hosts rather than the other way around, start with a base image close to the deployment hosts and get DinD working in those. A quick ./Dockerfile reproduced the deployment host content in an image, including deployment host filesystem paths and users, e.g. /home/ec2-user/:

FROM docker:dind
# Defensive shell settings, avoid silent failures
SHELL ["/bin/ash", "-xeu", "-c"]

# Install required OS host packages
RUN \
        apk --no-cache add 'py-pip' 'python3-dev' 'libffi-dev' 'openssl-dev' 'gcc' \
        'libc-dev' 'rust' 'cargo' 'make' && \
        pip3 install 'docker-compose'

# Duplicate the AWS EC2 user
RUN adduser --disabled-password --gecos 'EC2 User,,,' ec2-user

A ./docker-compose.yml file reproduces the running deployment hosts including network topology such as subnet and host IP addresses. I also abuse the ./docker-compose.yml file to build the application images:

version: "3.8"

networks:
  default:
    ipam:
      driver: "default"
      config:
        # Match the hosted deployment network
        - subnet: "10.81.82.0/24"

services:

# Build custom container image, to be loaded into the nested host Docker daemons.
  foo-app:
    build:
      context: "./foo-app/"
    image: "bar-company/foo-app:dev"
    # Build only, don't actually run as a service
    entrypoint: ["true"]

# Emulate deployment hosts running Docker daemons

  foo-host:
    build:
      context: "./"
    image: "bar-company/foo-host:dev"
    # Just build the image, don't actually run as a service
    entrypoint: ["true"]

  foo-host-corge:
    image: "bar-company/foo-host:dev"
    depends_on:
      - "foo-host"
    privileged: true
    networks:
      default:
        # Match the hosted deployment IP
        ipv4_address: "10.81.82.100"
    volumes:
      # Docker-in-Docker requires data on a real filesystem
      - "./var/lib/foo-host-corge/docker/:/var/lib/docker/"
      # Reproduce where project data is stored in hosted deployments
      - "./var/lib/foo-data/:/home/ec2-user/foo-data/"
      # Make source editable
      - "./foo-app/:/srv/foo-app/"
      # Reproduce deployment containers using a `$ docker-compose ...` configuration
      # specific to the deployment host
      - "./foo-host-corge/:/srv/foo-host-corge/"
    working_dir: "/srv/foo-host-corge/"

  foo-host-grault:
    image: "bar-company/foo-host:dev"
    depends_on:
      - "foo-host"
    privileged: true
    networks:
      default:
        # Match the hosted deployment IP
        ipv4_address: "10.81.82.101"
    volumes:
      # Docker-in-Docker requires data on a real filesystem
      - "./var/lib/foo-host-grault/docker/:/var/lib/docker/"
      # Reproduce where project data is stored in hosted deployments
      - "./var/lib/foo-data/:/home/ec2-user/foo-data/"
      # Make source editable
      - "./foo-app/:/srv/foo-app/"
      # Reproduce deployment containers using a `$ docker-compose ...` configuration
      # specific to the deployment host
      - "./foo-host-grault/:/srv/foo-host-grault/"
    working_dir: "/srv/foo-host-grault/"

Nested ./foo-host-corge/docker-compose.yml and ./foo-host-grault/docker-compose.yml files reproduce how the application containers are run on the real deployment hosts, but run instead within the nested DinD containers that reproduce the deployment hosts:

version: "3.8"

services:

  postgres:
    image: "postgres"
    ports:
      - "5432:5432"
    env_file:
      - "./.env"

  foo-app:
    image: "bar-company/foo-app:dev"
    ports:
      - "80:80"
    extra_hosts:
       - "redis:10.81.82.101"
    volumes:
      # Make source editable
      - "/srv/foo-app/:/srv/foo-app/"
      # Reproduce where project data is stored in hosted deployments
      - "/home/ec2-user/foo-data/:/srv/foo-data/"

version: "3.8"

services:

  redis:
    image: "redis"
    ports:
      - "6379:6379"

  foo-app:
    image: "bar-company/foo-app:dev"
    ports:
      - "80:80"
    extra_hosts:
       - "postgres:10.81.82.100"
    volumes:
      # Make source editable
      - "/srv/foo-app/:/srv/foo-app/"
      # Reproduce where project data is stored in hosted deployments
      - "/home/ec2-user/foo-data/:/srv/foo-data/"

Use some $ docker ... commands and some shell to push the application images into the nested # dockerd daemons and some $ docker-compose ... commands to run containers using those images, along with the rest of the deployment configuration. I use a ./Makefile to stitch this all together but use what you like:

# Reproduce hosted deployments in nested Docker (DinD)

### Defensive settings for make:
#     https://tech.davis-hansson.com/p/make/
SHELL := bash
.ONESHELL:
.SHELLFLAGS:=-xeu -o pipefail -O inherit_errexit -c
.SILENT:
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules


# Top-level targets

.PHONY: all
all: var/log/docker-compose-build.log foo-host-corge/.env

.PHONY: run
run: all
	docker-compose exec -T "foo-host-corge" docker-compose up -d
	docker-compose exec -T "foo-host-grault" docker-compose up -d
	sleep 1
	docker-compose exec -T "foo-host-corge" docker-compose ps
	docker-compose exec -T "foo-host-grault" docker-compose ps

.PHONY: test
test: run
# Demonstrate that deployment host network topology and hostnames are reproduced
	docker-compose exec foo-host-corge \
	    docker-compose exec foo-app nc -vz redis 6379
	docker-compose exec foo-host-grault \
	    docker-compose exec foo-app nc -vz postgres 5432
# Demonstrate that deployment host filesystem paths and data are reproduced
	sudo rm -rfv ./var/lib/foo-data/*
	ls -al "./var/lib/foo-data/"
	docker-compose exec foo-host-corge \
	    ls -al "/home/ec2-user/foo-data/"
	docker-compose exec foo-host-grault \
	    ls -al "/home/ec2-user/foo-data/"
	docker-compose exec foo-host-corge docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    test ! -e "/srv/foo-data/bar.txt"
	sudo touch "./var/lib/foo-data/bar.txt"
	docker-compose exec foo-host-corge docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    test -e "/srv/foo-data/bar.txt"

.PHONY: clean
clean:
	docker-compose down --rmi local -v
	sudo rm -rf "./var/lib/"
	mkdir -pv "./var/log/backups/"
	test ! -e "./var/log/docker-compose-build.log" ||
	    mv --backup=numbered -v "./var/log/docker-compose-build.log" \
	        "./var/log/backups/"
	test ! -e "./foo-host-corge/.env" ||
	    mv --backup=numbered -v "./foo-host-corge/.env" \
	        "./var/log/backups/"

# Real targets

var/log/docker-compose-build.log: foo-app/Dockerfile Dockerfile docker-compose.yml
	mkdir -pv "./$(dir $(@))/"
	sudo docker-compose build --pull | tee -a "./$(@)"
# Wait for the nested deployment host `# dorckerd` daemons to become available
	docker-compose up -d
	for host in foo-host-corge foo-host-grault
	do
	    timeout --foreground -v 20 $(SHELL) $(.SHELLFLAGS) "\
		while ! docker-compose exec -T "$${host}" docker ps; do sleep 0.1; done"
# Load the built image into the deployment host Docker daemons
	    docker save "bar-company/foo-app:dev" |
		docker-compose exec -T "$${host}" docker load
	done

foo-host-corge/.env:
	echo "POSTGRES_PASSWORD=$$(apg -M NCL -n 1)" >"./$(@)"

Once it’ up and running we can see that the network topology, host names, and containerized services and applications are running as they would in the real hosted deployment:

$ make test
+ docker-compose exec -T foo-host-corge docker-compose up -d
foo-host-corge_foo-app_1 is up-to-date
foo-host-corge_postgres_1 is up-to-date
+ docker-compose exec -T foo-host-grault docker-compose up -d
foo-host-grault_redis_1 is up-to-date
foo-host-grault_foo-app_1 is up-to-date
+ sleep 1
+ docker-compose exec -T foo-host-corge docker-compose ps
          Name                         Command               State           Ports
-------------------------------------------------------------------------------------------
foo-host-corge_foo-app_1    python -m http.server --di ...   Up      0.0.0.0:80->80/tcp
foo-host-corge_postgres_1   docker-entrypoint.sh postgres    Up      0.0.0.0:5432->5432/tcp
+ docker-compose exec -T foo-host-grault docker-compose ps
          Name                         Command               State           Ports
-------------------------------------------------------------------------------------------
foo-host-grault_foo-app_1   python -m http.server --di ...   Up      0.0.0.0:80->80/tcp
foo-host-grault_redis_1     docker-entrypoint.sh redis ...   Up      0.0.0.0:6379->6379/tcp
+ docker-compose exec foo-host-corge docker-compose exec foo-app nc -vz redis 6379
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.81.82.101:6379.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
+ docker-compose exec foo-host-grault docker-compose exec foo-app nc -vz postgres 5432
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.81.82.100:5432.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
+ sudo rm -rfv './var/lib/foo-data/*'
+ ls -al ./var/lib/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 08:49 .
drwxr-xr-x 5 root root 4096 Apr  5 08:45 ..
+ docker-compose exec foo-host-corge ls -al /home/ec2-user/foo-data/
total 8
drwxr-xr-x    2 root     root          4096 Apr  5 15:49 .
drwxr-sr-x    1 ec2-user ec2-user      4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault ls -al /home/ec2-user/foo-data/
total 8
drwxr-xr-x    2 root     root          4096 Apr  5 15:49 .
drwxr-sr-x    1 ec2-user ec2-user      4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-corge docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:49 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:49 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault docker-compose exec foo-app test '!' -e /srv/foo-data/bar.txt
+ sudo touch ./var/lib/foo-data/bar.txt
+ docker-compose exec foo-host-corge docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:50 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
-rw-r--r-- 1 root root    0 Apr  5 15:50 bar.txt
+ docker-compose exec foo-host-grault docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:50 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
-rw-r--r-- 1 root root    0 Apr  5 15:50 bar.txt
+ docker-compose exec foo-host-grault docker-compose exec foo-app test -e /srv/foo-data/bar.txt

There you have it. The demo code is publicly available. I think we can find more uses for DinD. In particular, I have always found local development clean build issues to be a persistent plague: “It worked for me when I committed it!”. Next up I’m working on using DinD as a route to a generic local clean build test.

Docker Gotchas I’ve Encountered

2021-04-08T00:00:00+00:00

Time to share the snags/gotchas I’ve run into developing and deploying with Docker containers.

On my recent projects, I seem to have become the Docker wrangler on the team. Over that time, I’ve hit a number of snags or gotchas and thought, “Maybe I should write that spot down?”. It’s as good a time as any now that I’ve reduced the impedance of my blogging platform.

Disk Space

TL;DR: CAUTION! $ docker system prune -a --volumes

It’s up to you to clean up unused images, containers, volumes, networks, etc.. This one will likely seem obvious to anyone who understands now how Docker works, but it certainly bit me a few times when I was climbing the learning curve.

I’m not a big fan of this fact, but it’s important to understand that the $ docker ... CLI is built only to perform the underlying operations, the building blocks, required for containerized applications. The important thing to pay attention to there is what it is not. It is not a system for managing images, containers, volumes, etc., at least not across deployments, over time, or between different applications. It is not a declarative system for describing what images should be used to run which containers connected to which volumes and networks on a given host. Even $ docker-compose ... is really only a different syntax to write a related collection of $ docker ... CLI commands in a way that is much more readable, maintainable, and feels declarative.

Stopping a container does not remove it. Removing a container does not remove the image, volumes or networks. Removing the image may not remove it’s base image or Layers. And so on. To use other terminology, neither the $ docker ... nor the $ docker-compose ... CLIs are orchestration systems. As such, a developer must clean up the inevitable large quantities of Docker cruft themselves. If, heavens forfend, you’re deploying containers without an orchestration system, then you’ll also have to do the same there. A recent project was my first AWS ECS exposure, so maybe it was being used wrong, but I was surprised to learn that this cruft also accrued for that usage of ECS.

The easiest way to take care of this is to use the docker system prune command (or the prune sub-command under the other $ docker ... commands) but it’s important to understand what it does lest you destroy data or even code in a certain sense. When Docker says “prune” they mean “relative to all currently running containers”. Say you have a debug container hanging around into which you’ve installed a rich set of OS/dist packages, configured things just the way you like, and even written a few utility or introspection scripts you’ve come to rely on. This debug container is stopped most of the time and only run when you need it. Firstly, never do this! Always treat containers as ephemeral and able to be destroyed at any time, but lets continue with this example. If this debug container isn’t running when you run $ docker system prune, then the container, its filesystem, its image, everything will be destroyed irrevocably.

As such, only run $ docker system prune when you’re sure every container is currently running whose, filesystem, image, volumes, etc. you need to preserve. See also the options/flags to that command for more thorough cleanup. IMO, if it’s not always safe to run $ docker system prune then you’re doing something wrong, and that stands for both local development and deployments.

Publicly Exposed Ports

TL;DR: Always specify the host IP for port mappings, e.g. 127.0.0.1:80:80.

Unlike SSH port forwarding, which defaults to only binding to the localhost IP, 127.0.0.1, the docker run -p … option binds all IPs by default. This is also true for $ docker-compose ... since it’s really just an alternate syntax for the $ docker ... CLI. So to prevent exposing the top secret application you’re developing on your laptop to everyone at the cafe who can copy and paste a $ nmap ... command, just default to always specifying 127.0.0.1 as the bind address. It’s a cheap way to force yourself to think about it and to make the intention of all port mappings clear to anyone else who reads them while simultaneously making an audit of intentionally exposed ports as simple as $ git grep -r '0\.0\.0\.0:[0-9]+:[0-9]+'. As for deployments, I’d personally much rather have a release break a deployment than have a release expose a service to the internet unintentionally and silently.

YAML Needs Some Zen

TL:DR; Quote every YAML value except numbers, booleans, and null.

$ python -c 'import this'
The Zen of Python, by Tim Peters
...
Explicit is better than implicit.
...

I love YAML, so a quick rant. Personally, I find the tendency among developers to find a flaw in a technology and then develop a long lasting hatred because it cost them some time once. Our job is to solve difficult technical problems. Our job is also to collectively build fruitful ecosystems of tools, frameworks, and other technology. I personally think that embracing what’s good about a technology is more productive than dismissing a whole technology because of a set of flaws that are a subset of its features. Using something and complaining is much more fruitful than complaining and not using. Is what you’re saying productive criticism or just complaining? I know the programmer’s virtues are delightfully subversive, but I don’t think whining is among them.

In fact, I’d love to see the YAML parser libraries add options, if not defaults in new major versions, to disable the following problematic type conversions. That could put pressure on the standard to move such surprising behavior to an explicit opt-in whereby it could still be powerful for those that need it but no longer surprising for the rest of us. Or maybe just the standard first, I don’t really know how these things work. ;-) Enough ranting.

I lied, next rant but opposite tone. YAML does include some fairly distressing magical behavior. Try to digest this:

>>> yaml.safe_load('port: 22:22')
{'port': 1342}

I lost more time than I care to admit trying to figure out why a ./docker-compose.yml SFTP port mapping wasn’t working when everything started up with no error. I eventually did discover that port 1342:1342 was being mapped but it certainly didn’t help me understand. My other port mappings were working without any surprises:

>>> yaml.safe_load('port: 80:80')
{'port': '80:80'}

What the actual fork! This is the very definition of surprising behavior. The root cause here is that YAML has support for several obscure value types, and one of them is base 60 integers. I would really love to hear someone make the case that the use cases for sexagesimal are important and powerful enough for such a majority of developer users that is justifies this surprise for the rest of us. Who knew the Babylonians are such a significant constituency!?

There are, however, other such value types that can result in similar unaccountably surprising behavior. Just avoid these issues, always quote every YAML value except numbers, booleans, and null unless and until you need any of that black magic:

>>> yaml.safe_load('port: "22:22"')
{'port': '22:22'}

Build Context and Image Stowaways

TL;DR: $ ln -sv "./.gitignore" "./.dockerignore"

Docker uses what it calls the build context, the directory containing the ./Dockerfile by default, to determine what is available to COPY into an image while building. That makes sense to me. What doesn’t make sense to me is that it copies the whole build context somewhere before building an image. If past is prologue I’d end up agreeing if I understood the full reasons for that, but until then I remain dubious. Moving on. Regardless of how the build context is handled or managed, the notion of the build context does sensibly define what will make it into the image.

When the build context includes unnecessary files and directories, at best it just slows down your image builds and maybe also increases your built image size to no benefit. At worst, it leaks content that wasn’t intended into a deployed image, or worse a publicly released image. I’ll be honest, though, I mostly get concerned about the former, unnecessarily long times build hurt a lot when it’s in the inner loop of developing an image. The method Docker provides to control which files under the build context directory should not be included in the build context is the ./.dockerignore file.

So then just remember to add paths to ./.dockerignore every time you do something that might add something to the build context that you wouldn’t want included in an image. Simple, right? Don’t we all know that developers are really good at keeping a long list of rote considerations in mind? Well I’m not. I also have to say I’m not the only one, given how many other developers on teams I’ve worked on have made changes that should be accompanied by additions to ./.dockerignore.

You know what draws my attention to new files in my build context, and very reliably? Well a new file represents a change and in such cases the new file is a consequence of some other change in the source code. We use tools to manage changes that don’t depend on developers remembering such considerations, don’t we? Alright, enough patronizing sarcasm. VCS, good ol’ $ git status, is how I notice when some change I made resulted in new files in the build context.

This might be controversial, but in spite of the many posts I’ve found while searching which say they should be managed separately, I’ve been strongly advocating for making ./.dockerignore a symlink that points to ./.gitignore for ~2 years now. I haven’t yet regretted it once. In that same time, I’ve also worked on projects where ./.dockerignore is managed separately and inappropriate files leaking into the build context has been a recurring issue on all of them, and not just from my commits.

There are frequent cases where a build artifact should not be committed to VCS but should be included in built images. Here we can exploit one of the differences between ./.gitignore and ./.dockerignore, namely that $ git ... processes ./.gitignore files in each descendant directory under the checkout but $ docker build ... does not. So make sure your build artifacts go somewhere in sub-directories of the checkout (a good idea in any case) and place your ./.gitignore rules for those artifacts at the appropriate level down within that sub-directory.

Also note the other differences between ./.gitignore and ./.dockerignore and keep your rules to those that are interpreted the same by both $ git ... and $ docker build .... In particular, replace your ./.gitignore rules meant to apply to files that match in any sub-directory, e.g. foo.txt, with more explicit rules that work in ./.dockerignore as well, e.g.:

/foo.txt
**/foo.txt

While more verbose, I prefer this anyways as it’s more explicit and communicates to me what’s intended instantly and without ambiguity.

TTFN

Well those are the big gotchas I’ve encountered. I’m sure I won’t encounter any more. I’m sure there won’t be any forthcoming “Docker Gotcha: …” posts. ;-)