###############################
Docker Gotchas I've Encountered
###############################

    Time to share the snags/gotchas I've run into developing and deploying with Docker
    containers.

On my recent projects, I seem to have become the `Docker`_ wrangler on the team.  Over
that time, I've hit a number of snags or gotchas and thought, `"Maybe I should write
that spot down?"`_.  It's as good a time as any now that `I've reduced the impedance of
my blogging platform`_.

.. contents::
   :local:

**********
Disk Space
**********

    TL;DR: **CAUTION!** ``$ docker system prune -a --volumes``

It's up to you to clean up unused images, containers, volumes, networks, etc..  This one
will likely seem obvious to anyone who understands now how Docker works, but it
certainly bit me a few times when I was climbing the learning curve.

I'm not a big fan of this fact, but it's important to understand that the ``$ docker
...`` CLI is built only to perform the underlying operations, the building blocks,
required for containerized applications.  The important thing to pay attention to there
is what it is *not*.  It is not a system for *managing* images, containers, volumes,
etc., at least not across deployments, over time, or between different applications.  It
is not a declarative system for describing what images should be used to run which
containers connected to which volumes and networks on a given host.  Even ``$
docker-compose ...`` _`is really only a different syntax` to write a related collection
of ``$ docker ...`` CLI commands in a way that is much more readable, maintainable, and
*feels* declarative.

Stopping a container does not remove it.  Removing a container does not remove the
image, volumes or networks.  Removing the image may not remove it's base image or
Layers.  And so on.  To use other terminology, neither the ``$ docker ...`` nor the ``$
docker-compose ...`` CLIs are `orchestration systems`_.  As such, a developer must clean
up the inevitable large quantities of Docker cruft themselves.  If, heavens forfend,
you're deploying containers *without* an orchestration system, then you'll also have to
do the same there.  A recent project was my first AWS ECS exposure, so maybe it was
being used wrong, but I was surprised to learn that this cruft also accrued for that
usage of ECS.

The easiest way to take care of this is to use the `docker system prune command`_ (or
the ``prune`` sub-command under the other ``$ docker ...`` commands) but it's important
to understand what it does lest you destroy data or even code in a certain sense.  When
`Docker`_ says "prune" they mean "relative to all *currently* running containers".  Say
you have a debug container hanging around into which you've installed a rich set of
OS/dist packages, configured things just the way you like, and even written a few
utility or introspection scripts you've come to rely on.  This debug container is
stopped most of the time and only run when you need it.  Firstly, **never do this**!
Always treat containers as ephemeral and able to be destroyed at any time, but lets
continue with this example.  If this debug container isn't running when you run ``$
docker system prune``, then the container, its filesystem, its image, everything will be
destroyed irrevocably.

As such, only run ``$ docker system prune`` when you're sure every container is
currently running whose, filesystem, image, volumes, etc. you need to preserve.  See
also the options/flags to that command for more thorough cleanup.  IMO, if it's not
always safe to run ``$ docker system prune`` then you're doing something wrong, and that
stands for both local development and deployments.

**********************
Publicly Exposed Ports
**********************

    TL;DR: Always specify the host IP for port mappings, e.g. ``127.0.0.1:80:80``.

Unlike `SSH port forwarding`_, which defaults to only binding to the ``localhost`` IP,
``127.0.0.1``, the `docker run -p ...`_ option binds all IPs by default.  This is also
true for ``$ docker-compose ...`` since it's really just `an alternate syntax`_ for the
``$ docker ...`` CLI.  So to prevent exposing the top secret application you're
developing on your laptop to everyone at the cafe who can copy and paste a ``$ nmap
...`` command, just default to always specifying ``127.0.0.1`` as the bind address.
It's a cheap way to force yourself to think about it and to make the intention of all
port mappings clear to anyone else who reads them while simultaneously making an audit
of intentionally exposed ports as simple as ``$ git grep -r
'0\.0\.0\.0:[0-9]+:[0-9]+'``.  As for deployments, I'd personally much rather have a
release break a deployment than have a release expose a service to the internet
unintentionally and silently.

*******************
YAML Needs Some Zen
*******************

    TL:DR; Quote every `YAML`_ value except numbers, booleans, and ``null``.

.. code-block:: shell-session

   $ python -c 'import this'
   The Zen of Python, by Tim Peters
   ...
   Explicit is better than implicit.
   ...

I love `YAML`_, so a quick rant.  Personally, I find the tendency among developers to
find a flaw in a technology and then develop a long lasting hatred because it cost them
some time once.  Our job is to solve difficult technical problems.  Our job is also to
collectively build fruitful ecosystems of tools, frameworks, and other technology.  I
personally think that embracing what's good about a technology is more productive than
dismissing a whole technology because of a set of flaws that are a subset of its
features.  Using something *and* complaining is much more fruitful than complaining and
not using.  Is what you're saying productive criticism or just complaining?  I know `the
programmer's virtues`_ are delightfully subversive, but I don't think whining is among
them.

In fact, I'd love to see `the YAML parser libraries`_ add options, if not defaults in
new major versions, to disable the following problematic type conversions.  That could
put pressure on the standard to move such surprising behavior to an explicit opt-in
whereby it could still be powerful for those that need it but no longer surprising for
the rest of us.  Or maybe just the standard first, I don't really know how these things
work.  ;-)  Enough ranting.

I lied, next rant but opposite tone.  `YAML`_ does include some fairly distressing
magical behavior.  Try to digest this::

  >>> yaml.safe_load('port: 22:22')
  {'port': 1342}

I lost more time than I care to admit trying to figure out why a
``./docker-compose.yml`` SFTP port mapping wasn't working when everything started up
with no error.  I eventually did discover that port ``1342:1342`` was being mapped but
it certainly didn't help me *understand*.  My other port mappings were working without
any surprises::

  >>> yaml.safe_load('port: 80:80')
  {'port': '80:80'}

What the actual *fork*!  This is the very definition of `surprising behavior`_.  The
root cause here is that `YAML`_ has support for several obscure value types, and `one of
them is base 60 integers`_.  I would really love to hear someone make the case that the
use cases for sexagesimal are important and powerful enough for such a majority of
developer users that is justifies this surprise for the rest of us.  Who knew `the
Babylonians`_ are such a significant constituency!?

There are, however, `other such value types that can result in similar unaccountably
surprising behavior`_.  Just avoid these issues, always quote every `YAML`_ value except
numbers, booleans, and ``null`` unless and until you need any of that black magic::

  >>> yaml.safe_load('port: "22:22"')
  {'port': '22:22'}

*********************************
Build Context and Image Stowaways
*********************************

    TL;DR: ``$ ln -sv "./.gitignore" "./.dockerignore"``

Docker uses what it calls `the build context`_, the directory containing the
``./Dockerfile`` by default, to determine what is available to ``COPY`` into an image
while building.  That makes sense to me.  What doesn't make sense to me is that it
*copies* the whole build context somewhere before building an image.  If past is
prologue I'd end up agreeing if I understood the full reasons for that, but until then I
remain dubious.  Moving on.  Regardless of how the build context is handled or managed,
the notion of `the build context`_ does sensibly define what will make it into the image.

When `the build context`_ includes unnecessary files and directories, at best it just
slows down your image builds and maybe also increases your built image size to no
benefit.  At worst, it leaks content that wasn't intended into a deployed image, or
worse a publicly released image.  I'll be honest, though, I mostly get concerned about
the former, unnecessarily long times build hurt a lot when it's in the inner loop of
developing an image.  The method `Docker`_ provides to control which files under the
build context directory should *not* be included in the build context is `the
./.dockerignore`_ file.

So then just remember to add paths to ``./.dockerignore`` every time you do something
that might add something to `the build context`_ that you wouldn't want included in an
image.  Simple, right?  Don't we all know that developers are really good at keeping a
long list of rote considerations in mind?  Well I'm not.  I also have to say I'm not the
only one, given how many other developers on teams I've worked on have made changes that
should be accompanied by additions to ``./.dockerignore``.

You know what draws my attention to new files in my build context, and very reliably?
Well a new file represents a change and in such cases the new file is a consequence of
some other change in the source code.  We use tools to manage changes that don't depend
on developers remembering such considerations, don't we?  Alright, enough patronizing
sarcasm.  VCS, good ol' ``$ git status``, is how I notice when some change I made
resulted in new files in the build context.

`This might be controversial`_, but in spite of the many posts I've found while
searching which say they should be managed separately, I've been strongly advocating for
making ``./.dockerignore`` a symlink that points to ``./.gitignore`` for ~2 years now.
I haven't yet regretted it once.  In that same time, I've also worked on projects where
``./.dockerignore`` is managed separately and inappropriate files leaking into the build
context has been a recurring issue on all of them, and not just from my commits.

There are frequent cases where a build artifact should not be committed to VCS but
*should* be included in built images.  Here we can exploit one of the `differences
between ./.gitignore and ./.dockerignore`_, namely that ``$ git ...`` processes
``./.gitignore`` files in each descendant directory under the checkout but ``$ docker
build ...`` does not.  So make sure your build artifacts go somewhere in sub-directories
of the checkout (a good idea in any case) and place your ``./.gitignore`` rules for
those artifacts at the appropriate level down within that sub-directory.

Also note the other `differences between ./.gitignore and ./.dockerignore`_ and keep
your rules to those that are interpreted the same by both ``$ git ...`` and ``$ docker
build ...``.  In particular, replace your ``./.gitignore`` rules meant to apply to files
that match in any sub-directory, e.g. ``foo.txt``, with more explicit rules that work in
``./.dockerignore`` as well, e.g.:

.. code-block:: text

  /foo.txt
  **/foo.txt

While more verbose, I prefer this anyways as it's more explicit and communicates to me
what's intended instantly and without ambiguity.

****
TTFN
****

Well those are the big gotchas I've encountered.  I'm sure I won't encounter any more.
I'm sure there won't be any forthcoming "Docker Gotcha: ..." posts.  ;-)

.. _`SSH port forwarding`: https://man.openbsd.org/ssh_config#LocalForward

.. _`YAML`: https://yaml.org/
.. _`the YAML parser libraries`: https://hitchdev.com/strictyaml/why/implicit-typing-removed/
.. _`one of them is base 60 integers`: https://yaml.org/type/float.html
.. _`the Babylonians`: https://en.wikipedia.org/wiki/Sexagesimal#Babylonian_mathematics
.. _`other such value types that can result in similar unaccountably surprising behavior`: https://sidneyliebrand.medium.com/the-greatnesses-and-gotchas-of-yaml-5e3377ef0c55

.. _`Docker`: https://docs.docker.com/
.. _`orchestration systems`: https://docs.docker.com/get-started/orchestration/
.. _`docker system prune command`: https://docs.docker.com/engine/reference/commandline/system_prune/
.. _`docker run -p ...`: https://docs.docker.com/engine/reference/commandline/run/#publish-or-expose-port--p---expose
.. _`the build context`: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#understand-build-context
.. _`the ./.dockerignore`: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#exclude-with-dockerignore
.. _`differences between ./.gitignore and ./.dockerignore`: https://zzz.buzz/2018/05/23/differences-of-rules-between-gitignore-and-dockerignore/

.. _`This might be controversial`: http://risk-show.com/podcast/this-might-be-controversial/
.. _`"Maybe I should write that spot down?"`: https://www.google.com/search?q=%22far+side%22+%22maybe+we+should+write+that+spot+down%22&sxsrf=ALeKk03GSjfKEOtaGmJTX2YwOMbOL5NiNw:1617781214389&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiEge7U0OvvAhUKP30KHVOPAnsQ_AUoAXoECAIQAw&biw=1536&bih=785&dpr=1.25
.. _`the programmer's virtues`: http://threevirtues.com/
.. _`surprising behavior`: https://en.wikipedia.org/wiki/Principle_of_least_astonishment

.. _`I've reduced the impedance of my blogging platform`: ../migrating-from-plone-to-ablog/
.. _`an alternate syntax`: `is really only a different syntax`_

.. meta::
   :description: Time to share the snags/gotchas I've run into developing and deploying
                 with Docker containers.
   :keywords: Docker, docker-compose, YAML, containerization, gotchas

.. post:: Apr 8, 2021
   :author: Ross Patterson
   :tags: Docker, docker-compose, YAML, containerization, gotchas