#############################################
Reproducing Deployments with Docker-in-Docker
#############################################

    Debugging deployment issues locally by falling down the `Docker-in-Docker`_ (DinD)
    rabbit hole.

TL;DR:

    You can run a ``# dockerd`` daemon inside a `Docker`_ container which can be used with
    ``$ docker-compose ...`` to (almost) completely reproduce hosted deployment
    environments on one's development laptop.

As noted in `a recent post`_, I seem to have become the `Docker`_ wrangler on the team
for my recent projects.  Also recently, I'm getting the distinct impression that
containerization has or is beginning to reach that special point in the life-cycle of a
technology where it is sufficiently widely adopted to be misused and has been in use
long enough for that misuse to hurt.  This is one of those ironic signs of success and
utility but knowing that doesn't lessen the pain or cost of the individual case one may
run into.

Most recently, I was dealing with problematically fragile container-based deployments
where the team was getting bitten over and over again by deployment details that weren't
causing problems in our local container-based development environments but were causing
problems when CD pushed our changes to real hosted deployments.  Specifically, the
hosted deployments were failing on such things as filesystem paths and IP addresses.  To
make things worse, the internal CI/CD pipeline was under-resourced and very slow making
the inner loop of testing changes painfully slow and wasteful.

Before you think it, this is a project where there are many things that need to be
re-worked to be in alignment with current best practices and thus less fragile: from how
containers are used, to keeping CI/CD responsive, to how deployment is orchestrated.
Deployments should *not* be sensitive to changes in filesystem paths and IP addresses,
to be sure.  The project was a `Python 3 upgrade`_, however, and we'd done just about
all there was appetite for in terms of cleanup work that wasn't directly related to the
upgrade.  As software developers, it's our job to figure out how to solve difficult
technical problems, including the difficulty of getting sub-optimal technology usage to
work.

At any rate, the tax we were paying for this deployment sensitivity was so high and was
happening often enough that I decided it was past time to work on a way to try and
reproduce the real hosted deployment environment more completely, preferably locally.
Reproducing deployment hosts as thoroughly as possible without turning my development
laptop in to the deployment hosts feels like an isolation issue, so I reached for
containers and Docker.  Specifically, I wanted to start with a container image as close
to the VM image used by the deployment hosts as possible and figure out how to run the
actual application container images within those "host containers".  IOW, can I run
Docker containers within a Docker container?  Yup, `for some time now`_.

It turns out that Docker now provides an `official Docker-in-Docker (DinD) image`_.
After reading it's ``./Dockerfile`` and some blog posts detailing how to run DinD, I
decided it would be more efficient and maintainable to use the official ``docker:dind``
image as my base image and extend it to match the project's deployment hosts rather than
the other way around, start with a base image close to the deployment hosts and get DinD
working in those.  A quick :download:`./Dockerfile <./Dockerfile>` reproduced the
deployment host content in an image, including deployment host filesystem paths and
users, e.g. ``/home/ec2-user/``:

.. literalinclude:: ./Dockerfile
   :language: dockerfile

A :download:`./docker-compose.yml <./docker-compose.yml>` file reproduces the running
deployment hosts including network topology such as subnet and host IP addresses.  I
also abuse the ``./docker-compose.yml`` file to build :download:`the application images
<./foo-app/Dockerfile>`:

.. literalinclude:: ./docker-compose.yml
   :language: yaml

Nested :download:`./foo-host-corge/docker-compose.yml
<./foo-host-corge/docker-compose.yml>` and
:download:`./foo-host-grault/docker-compose.yml <./foo-host-grault/docker-compose.yml>`
files reproduce how the application containers are run on the real deployment hosts, but
run instead within the nested DinD containers that reproduce the deployment hosts:

.. literalinclude:: ./foo-host-corge/docker-compose.yml
   :language: yaml

.. literalinclude:: ./foo-host-grault/docker-compose.yml
   :language: yaml

Use some ``$ docker ...`` commands and some shell to push the application images into
the nested ``# dockerd`` daemons and some ``$ docker-compose ...`` commands to run
containers using those images, along with the rest of the deployment configuration.  I
use a :download:`./Makefile <./Makefile>` to stitch this all together but use what you
like:

.. literalinclude:: ./Makefile
   :language: makefile

Once it' up and running we can see that the network topology, host names, and
containerized services and applications are running as they would in the real hosted
deployment:

.. literalinclude:: ./demo.sh-session
   :language: shell-session

There you have it.  `The demo code is publicly available`_.  I think we can find more
uses for DinD.  In particular, I have always found local development clean build issues
to be a persistent plague: "It worked for me when I committed it!".  Next up I'm working
on using DinD as a route to a generic local clean build test.

.. _`Docker`: https://docs.docker.com/
.. _`for some time now`: https://www.docker.com/blog/docker-can-now-run-within-docker/
.. _`Docker-in-Docker`: https://hub.docker.com/_/docker
.. _`official Docker-in-Docker (DinD) image`: `Docker-in-Docker`_

.. _`Python 3 upgrade`: https://docs.python.org/3/howto/pyporting.html

.. _`a recent post`: ../docker-gotchas/
.. _`The demo code is publicly available`: https://gitlab.com/rpatterson/dind-reproduce-deploys


.. meta::
   :description: Debugging deployment issues locally by falling down the
                 Docker-in-Docker (DinD) rabbit hole.
   :keywords: Docker, DinD, docker-compose

.. post:: Apr 9, 2021
   :author: Ross Patterson
   :tags: Docker, DinD, docker-compose