https://www.rpatterson.net/Ross Patterson's Blog - Posts tagged gotchas2023-08-22T15:29:28.184048+00:00ABloghttps://www.rpatterson.net/blog/docker-gotchas/Docker Gotchas I’ve Encountered2021-04-08T00:00:00+00:00Ross Patterson<section id="docker-gotchas-i-ve-encountered">
<blockquote>
<div><p>Time to share the snags/gotchas I’ve run into developing and deploying with Docker
containers.</p>
</div></blockquote>
<p>On my recent projects, I seem to have become the <a class="reference external" href="https://docs.docker.com/">Docker</a> wrangler on the team. Over
that time, I’ve hit a number of snags or gotchas and thought, <a class="reference external" href="https://www.google.com/search?q=%22far+side%22+%22maybe+we+should+write+that+spot+down%22&sxsrf=ALeKk03GSjfKEOtaGmJTX2YwOMbOL5NiNw:1617781214389&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiEge7U0OvvAhUKP30KHVOPAnsQ_AUoAXoECAIQAw&biw=1536&bih=785&dpr=1.25">“Maybe I should write
that spot down?”</a>. It’s as good a time as any now that <a class="reference external" href="../migrating-from-plone-to-ablog/">I’ve reduced the impedance of
my blogging platform</a>.</p>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#disk-space" id="id1">Disk Space</a></p></li>
<li><p><a class="reference internal" href="#publicly-exposed-ports" id="id2">Publicly Exposed Ports</a></p></li>
<li><p><a class="reference internal" href="#yaml-needs-some-zen" id="id3">YAML Needs Some Zen</a></p></li>
<li><p><a class="reference internal" href="#build-context-and-image-stowaways" id="id4">Build Context and Image Stowaways</a></p></li>
<li><p><a class="reference internal" href="#ttfn" id="id5">TTFN</a></p></li>
</ul>
</nav>
<section id="disk-space">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Disk Space</a></h2>
<blockquote>
<div><p>TL;DR: <strong>CAUTION!</strong> <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">system</span> <span class="pre">prune</span> <span class="pre">-a</span> <span class="pre">--volumes</span></code></p>
</div></blockquote>
<p>It’s up to you to clean up unused images, containers, volumes, networks, etc.. This one
will likely seem obvious to anyone who understands now how Docker works, but it
certainly bit me a few times when I was climbing the learning curve.</p>
<p>I’m not a big fan of this fact, but it’s important to understand that the <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span>
<span class="pre">...</span></code> CLI is built only to perform the underlying operations, the building blocks,
required for containerized applications. The important thing to pay attention to there
is what it is <em>not</em>. It is not a system for <em>managing</em> images, containers, volumes,
etc., at least not across deployments, over time, or between different applications. It
is not a declarative system for describing what images should be used to run which
containers connected to which volumes and networks on a given host. Even <code class="docutils literal notranslate"><span class="pre">$</span>
<span class="pre">docker-compose</span> <span class="pre">...</span></code> <span class="target" id="is-really-only-a-different-syntax">is really only a different syntax</span> to write a related collection
of <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">...</span></code> CLI commands in a way that is much more readable, maintainable, and
<em>feels</em> declarative.</p>
<p>Stopping a container does not remove it. Removing a container does not remove the
image, volumes or networks. Removing the image may not remove it’s base image or
Layers. And so on. To use other terminology, neither the <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">...</span></code> nor the <code class="docutils literal notranslate"><span class="pre">$</span>
<span class="pre">docker-compose</span> <span class="pre">...</span></code> CLIs are <a class="reference external" href="https://docs.docker.com/get-started/orchestration/">orchestration systems</a>. As such, a developer must clean
up the inevitable large quantities of Docker cruft themselves. If, heavens forfend,
you’re deploying containers <em>without</em> an orchestration system, then you’ll also have to
do the same there. A recent project was my first AWS ECS exposure, so maybe it was
being used wrong, but I was surprised to learn that this cruft also accrued for that
usage of ECS.</p>
<p>The easiest way to take care of this is to use the <a class="reference external" href="https://docs.docker.com/engine/reference/commandline/system_prune/">docker system prune command</a> (or
the <code class="docutils literal notranslate"><span class="pre">prune</span></code> sub-command under the other <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">...</span></code> commands) but it’s important
to understand what it does lest you destroy data or even code in a certain sense. When
<a class="reference external" href="https://docs.docker.com/">Docker</a> says “prune” they mean “relative to all <em>currently</em> running containers”. Say
you have a debug container hanging around into which you’ve installed a rich set of
OS/dist packages, configured things just the way you like, and even written a few
utility or introspection scripts you’ve come to rely on. This debug container is
stopped most of the time and only run when you need it. Firstly, <strong>never do this</strong>!
Always treat containers as ephemeral and able to be destroyed at any time, but lets
continue with this example. If this debug container isn’t running when you run <code class="docutils literal notranslate"><span class="pre">$</span>
<span class="pre">docker</span> <span class="pre">system</span> <span class="pre">prune</span></code>, then the container, its filesystem, its image, everything will be
destroyed irrevocably.</p>
<p>As such, only run <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">system</span> <span class="pre">prune</span></code> when you’re sure every container is
currently running whose, filesystem, image, volumes, etc. you need to preserve. See
also the options/flags to that command for more thorough cleanup. IMO, if it’s not
always safe to run <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">system</span> <span class="pre">prune</span></code> then you’re doing something wrong, and that
stands for both local development and deployments.</p>
</section>
<section id="publicly-exposed-ports">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">Publicly Exposed Ports</a></h2>
<blockquote>
<div><p>TL;DR: Always specify the host IP for port mappings, e.g. <code class="docutils literal notranslate"><span class="pre">127.0.0.1:80:80</span></code>.</p>
</div></blockquote>
<p>Unlike <a class="reference external" href="https://man.openbsd.org/ssh_config#LocalForward">SSH port forwarding</a>, which defaults to only binding to the <code class="docutils literal notranslate"><span class="pre">localhost</span></code> IP,
<code class="docutils literal notranslate"><span class="pre">127.0.0.1</span></code>, the <a class="reference external" href="https://docs.docker.com/engine/reference/commandline/run/#publish-or-expose-port--p---expose">docker run -p …</a> option binds all IPs by default. This is also
true for <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker-compose</span> <span class="pre">...</span></code> since it’s really just <a class="reference internal" href="#is-really-only-a-different-syntax">an alternate syntax</a> for the
<code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span> <span class="pre">...</span></code> CLI. So to prevent exposing the top secret application you’re
developing on your laptop to everyone at the cafe who can copy and paste a <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">nmap</span>
<span class="pre">...</span></code> command, just default to always specifying <code class="docutils literal notranslate"><span class="pre">127.0.0.1</span></code> as the bind address.
It’s a cheap way to force yourself to think about it and to make the intention of all
port mappings clear to anyone else who reads them while simultaneously making an audit
of intentionally exposed ports as simple as <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">git</span> <span class="pre">grep</span> <span class="pre">-r</span>
<span class="pre">'0\.0\.0\.0:[0-9]+:[0-9]+'</span></code>. As for deployments, I’d personally much rather have a
release break a deployment than have a release expose a service to the internet
unintentionally and silently.</p>
</section>
<section id="yaml-needs-some-zen">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">YAML Needs Some Zen</a></h2>
<blockquote>
<div><p>TL:DR; Quote every <a class="reference external" href="https://yaml.org/">YAML</a> value except numbers, booleans, and <code class="docutils literal notranslate"><span class="pre">null</span></code>.</p>
</div></blockquote>
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>python<span class="w"> </span>-c<span class="w"> </span><span class="s1">'import this'</span>
<span class="go">The Zen of Python, by Tim Peters</span>
<span class="go">...</span>
<span class="go">Explicit is better than implicit.</span>
<span class="go">...</span>
</pre></div>
</div>
<p>I love <a class="reference external" href="https://yaml.org/">YAML</a>, so a quick rant. Personally, I find the tendency among developers to
find a flaw in a technology and then develop a long lasting hatred because it cost them
some time once. Our job is to solve difficult technical problems. Our job is also to
collectively build fruitful ecosystems of tools, frameworks, and other technology. I
personally think that embracing what’s good about a technology is more productive than
dismissing a whole technology because of a set of flaws that are a subset of its
features. Using something <em>and</em> complaining is much more fruitful than complaining and
not using. Is what you’re saying productive criticism or just complaining? I know <a class="reference external" href="http://threevirtues.com/">the
programmer’s virtues</a> are delightfully subversive, but I don’t think whining is among
them.</p>
<p>In fact, I’d love to see <a class="reference external" href="https://hitchdev.com/strictyaml/why/implicit-typing-removed/">the YAML parser libraries</a> add options, if not defaults in
new major versions, to disable the following problematic type conversions. That could
put pressure on the standard to move such surprising behavior to an explicit opt-in
whereby it could still be powerful for those that need it but no longer surprising for
the rest of us. Or maybe just the standard first, I don’t really know how these things
work. ;-) Enough ranting.</p>
<p>I lied, next rant but opposite tone. <a class="reference external" href="https://yaml.org/">YAML</a> does include some fairly distressing
magical behavior. Try to digest this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">yaml</span><span class="o">.</span><span class="n">safe_load</span><span class="p">(</span><span class="s1">'port: 22:22'</span><span class="p">)</span>
<span class="go">{'port': 1342}</span>
</pre></div>
</div>
<p>I lost more time than I care to admit trying to figure out why a
<code class="docutils literal notranslate"><span class="pre">./docker-compose.yml</span></code> SFTP port mapping wasn’t working when everything started up
with no error. I eventually did discover that port <code class="docutils literal notranslate"><span class="pre">1342:1342</span></code> was being mapped but
it certainly didn’t help me <em>understand</em>. My other port mappings were working without
any surprises:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">yaml</span><span class="o">.</span><span class="n">safe_load</span><span class="p">(</span><span class="s1">'port: 80:80'</span><span class="p">)</span>
<span class="go">{'port': '80:80'}</span>
</pre></div>
</div>
<p>What the actual <em>fork</em>! This is the very definition of <a class="reference external" href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">surprising behavior</a>. The
root cause here is that <a class="reference external" href="https://yaml.org/">YAML</a> has support for several obscure value types, and <a class="reference external" href="https://yaml.org/type/float.html">one of
them is base 60 integers</a>. I would really love to hear someone make the case that the
use cases for sexagesimal are important and powerful enough for such a majority of
developer users that is justifies this surprise for the rest of us. Who knew <a class="reference external" href="https://en.wikipedia.org/wiki/Sexagesimal#Babylonian_mathematics">the
Babylonians</a> are such a significant constituency!?</p>
<p>There are, however, <a class="reference external" href="https://sidneyliebrand.medium.com/the-greatnesses-and-gotchas-of-yaml-5e3377ef0c55">other such value types that can result in similar unaccountably
surprising behavior</a>. Just avoid these issues, always quote every <a class="reference external" href="https://yaml.org/">YAML</a> value except
numbers, booleans, and <code class="docutils literal notranslate"><span class="pre">null</span></code> unless and until you need any of that black magic:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">yaml</span><span class="o">.</span><span class="n">safe_load</span><span class="p">(</span><span class="s1">'port: "22:22"'</span><span class="p">)</span>
<span class="go">{'port': '22:22'}</span>
</pre></div>
</div>
</section>
<section id="build-context-and-image-stowaways">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Build Context and Image Stowaways</a></h2>
<blockquote>
<div><p>TL;DR: <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">ln</span> <span class="pre">-sv</span> <span class="pre">"./.gitignore"</span> <span class="pre">"./.dockerignore"</span></code></p>
</div></blockquote>
<p>Docker uses what it calls <a class="reference external" href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#understand-build-context">the build context</a>, the directory containing the
<code class="docutils literal notranslate"><span class="pre">./Dockerfile</span></code> by default, to determine what is available to <code class="docutils literal notranslate"><span class="pre">COPY</span></code> into an image
while building. That makes sense to me. What doesn’t make sense to me is that it
<em>copies</em> the whole build context somewhere before building an image. If past is
prologue I’d end up agreeing if I understood the full reasons for that, but until then I
remain dubious. Moving on. Regardless of how the build context is handled or managed,
the notion of <a class="reference external" href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#understand-build-context">the build context</a> does sensibly define what will make it into the image.</p>
<p>When <a class="reference external" href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#understand-build-context">the build context</a> includes unnecessary files and directories, at best it just
slows down your image builds and maybe also increases your built image size to no
benefit. At worst, it leaks content that wasn’t intended into a deployed image, or
worse a publicly released image. I’ll be honest, though, I mostly get concerned about
the former, unnecessarily long times build hurt a lot when it’s in the inner loop of
developing an image. The method <a class="reference external" href="https://docs.docker.com/">Docker</a> provides to control which files under the
build context directory should <em>not</em> be included in the build context is <a class="reference external" href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#exclude-with-dockerignore">the
./.dockerignore</a> file.</p>
<p>So then just remember to add paths to <code class="docutils literal notranslate"><span class="pre">./.dockerignore</span></code> every time you do something
that might add something to <a class="reference external" href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#understand-build-context">the build context</a> that you wouldn’t want included in an
image. Simple, right? Don’t we all know that developers are really good at keeping a
long list of rote considerations in mind? Well I’m not. I also have to say I’m not the
only one, given how many other developers on teams I’ve worked on have made changes that
should be accompanied by additions to <code class="docutils literal notranslate"><span class="pre">./.dockerignore</span></code>.</p>
<p>You know what draws my attention to new files in my build context, and very reliably?
Well a new file represents a change and in such cases the new file is a consequence of
some other change in the source code. We use tools to manage changes that don’t depend
on developers remembering such considerations, don’t we? Alright, enough patronizing
sarcasm. VCS, good ol’ <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">git</span> <span class="pre">status</span></code>, is how I notice when some change I made
resulted in new files in the build context.</p>
<p><a class="reference external" href="http://risk-show.com/podcast/this-might-be-controversial/">This might be controversial</a>, but in spite of the many posts I’ve found while
searching which say they should be managed separately, I’ve been strongly advocating for
making <code class="docutils literal notranslate"><span class="pre">./.dockerignore</span></code> a symlink that points to <code class="docutils literal notranslate"><span class="pre">./.gitignore</span></code> for ~2 years now.
I haven’t yet regretted it once. In that same time, I’ve also worked on projects where
<code class="docutils literal notranslate"><span class="pre">./.dockerignore</span></code> is managed separately and inappropriate files leaking into the build
context has been a recurring issue on all of them, and not just from my commits.</p>
<p>There are frequent cases where a build artifact should not be committed to VCS but
<em>should</em> be included in built images. Here we can exploit one of the <a class="reference external" href="https://zzz.buzz/2018/05/23/differences-of-rules-between-gitignore-and-dockerignore/">differences
between ./.gitignore and ./.dockerignore</a>, namely that <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">git</span> <span class="pre">...</span></code> processes
<code class="docutils literal notranslate"><span class="pre">./.gitignore</span></code> files in each descendant directory under the checkout but <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span>
<span class="pre">build</span> <span class="pre">...</span></code> does not. So make sure your build artifacts go somewhere in sub-directories
of the checkout (a good idea in any case) and place your <code class="docutils literal notranslate"><span class="pre">./.gitignore</span></code> rules for
those artifacts at the appropriate level down within that sub-directory.</p>
<p>Also note the other <a class="reference external" href="https://zzz.buzz/2018/05/23/differences-of-rules-between-gitignore-and-dockerignore/">differences between ./.gitignore and ./.dockerignore</a> and keep
your rules to those that are interpreted the same by both <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">git</span> <span class="pre">...</span></code> and <code class="docutils literal notranslate"><span class="pre">$</span> <span class="pre">docker</span>
<span class="pre">build</span> <span class="pre">...</span></code>. In particular, replace your <code class="docutils literal notranslate"><span class="pre">./.gitignore</span></code> rules meant to apply to files
that match in any sub-directory, e.g. <code class="docutils literal notranslate"><span class="pre">foo.txt</span></code>, with more explicit rules that work in
<code class="docutils literal notranslate"><span class="pre">./.dockerignore</span></code> as well, e.g.:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>/foo.txt
**/foo.txt
</pre></div>
</div>
<p>While more verbose, I prefer this anyways as it’s more explicit and communicates to me
what’s intended instantly and without ambiguity.</p>
</section>
<section id="ttfn">
<h2><a class="toc-backref" href="#id5" role="doc-backlink">TTFN</a></h2>
<p>Well those are the big gotchas I’ve encountered. I’m sure I won’t encounter any more.
I’m sure there won’t be any forthcoming “Docker Gotcha: …” posts. ;-)</p>
</section>
</section>
Time to share the snags/gotchas I’ve run into developing and deploying with Docker
containers.2021-04-08T00:00:00+00:00