Ross Patterson's Blog - Posted in 2021

Endorsing Conventional Commits

2021-04-12T00:00:00+00:00

Consider adopting a VCS commit message convention, Conventional Commits has made me a better developer.

Over the years when starting a new project, I usually found myself staring at the cursor in my editor when it prompts me to write the commit message for my first commit and thinking “Is there a convention for this?”. Sometimes the project would have it’s own convention, which was always refreshing. That has been, unfortunately, rare and each such project usually ended up being the only one that uses that particular convention. In the last few years, I’ve been using Conventional Commits here and there. When I went looking for a tool to automate versioning and releasing based on VCS history, the tool I found uses Conventional Commits which pushed me over the edge to endorsing and advocating for it.

The primary reason I reach for a commit message convention is how much easier it makes visually scanning and make sense of VCS logs as a human. It makes a world of difference. Trying to remember that cool application logic you wrote because you need something similar again now but all your $ git log -G... efforts have come to naught? Browsing the log narrowed to $ git log --grep'^feat' will most often get you there a lot faster when compared to iterating through the whole log. For everything from $ git blame ... to CI failure messages to picking out-of-place commits in a PR, having this extra metadata as well as declaration of intent and scope in every commit has a thousand little incremental benefits that really add up for the humans that have to interpret commit messages.

The second most important reason for a convention, IMO, is it makes us better developers. It certainly has made me a better developer. When we add more structure to our commit messages and there is meaning attached to that structure, then writing commits forces us to think about those aspects of development at the time we are committing changes. When my commit message convention includes a piece of structure that declares how this change relates to releasing, versioning, and the changelog, then I am forced to think about those things before committing. If the changes I was intending to include in this commit include some backwards incompatible refactoring and a new feature, the convention will force me to realize I need to refactor this commit into two separate commits. It may also help me to think of a way to rework the changes to be backwards compatible.

Finally, using a commit message convention allows us to use tools to automate versioning, the changelog, and releasing. Use a tool in your CD pipeline, so that when you merge develop into master and $ git push, the CD pipeline will create a new version tag using the convention to determine whether the next version should be a patch version for bug fixes, a minor version for backwards compatible features, or a major version for backwards incompatible changes. Next, the CD pipeline kicks off the language-specific process to build release artifacts and a language-specific add-on/library uses that new version tag to determine the version to use in the release artifacts. Finally, the CD pipeline will upload those release artifacts to the appropriate registry/index.

IOW, merge, push, and walk away. Release discipline becomes a part of code review, e.g.: “The commit message says this is a backwards compatible feature but this change is not backward compatible. Please rework this change to be backward compatible or, if not possible, change the type in the commit message.” No manual version bump or changelog update steps. No duplicated version information. There are far fewer mind numbing chores for the release manager leading to less burn out in that role and more frequent releases.

Since I started using Conventional Commits, I’ve just formatted my commit messages thusly and when working with a team I’ve only shared my enthusiasm when a teammate asked. Having done so for a while now, I think I’m ready to start taking a more active tack and proselytizing to my teammates going forward; for the use of a convention in general, and Conventional Commits in particular. I can’t imagine going back, certainly not back to commit messages without some convention. I’m pretty thoroughly sold on the benefits, even when not using the CD tools that take advantage, and I honestly can’t think of a downside. So give it a shot, embrace the discipline, and see if you feel the same.

P.S.: If you agree and you’re a Magit user, please +1 my feature request.

Reproducing Deployments with Docker-in-Docker

2021-04-09T00:00:00+00:00

Debugging deployment issues locally by falling down the Docker-in-Docker (DinD) rabbit hole.

TL;DR:

You can run a # dockerd daemon inside a Docker container which can be used with $ docker-compose ... to (almost) completely reproduce hosted deployment environments on one’s development laptop.

As noted in a recent post, I seem to have become the Docker wrangler on the team for my recent projects. Also recently, I’m getting the distinct impression that containerization has or is beginning to reach that special point in the life-cycle of a technology where it is sufficiently widely adopted to be misused and has been in use long enough for that misuse to hurt. This is one of those ironic signs of success and utility but knowing that doesn’t lessen the pain or cost of the individual case one may run into.

Most recently, I was dealing with problematically fragile container-based deployments where the team was getting bitten over and over again by deployment details that weren’t causing problems in our local container-based development environments but were causing problems when CD pushed our changes to real hosted deployments. Specifically, the hosted deployments were failing on such things as filesystem paths and IP addresses. To make things worse, the internal CI/CD pipeline was under-resourced and very slow making the inner loop of testing changes painfully slow and wasteful.

Before you think it, this is a project where there are many things that need to be re-worked to be in alignment with current best practices and thus less fragile: from how containers are used, to keeping CI/CD responsive, to how deployment is orchestrated. Deployments should not be sensitive to changes in filesystem paths and IP addresses, to be sure. The project was a Python 3 upgrade, however, and we’d done just about all there was appetite for in terms of cleanup work that wasn’t directly related to the upgrade. As software developers, it’s our job to figure out how to solve difficult technical problems, including the difficulty of getting sub-optimal technology usage to work.

At any rate, the tax we were paying for this deployment sensitivity was so high and was happening often enough that I decided it was past time to work on a way to try and reproduce the real hosted deployment environment more completely, preferably locally. Reproducing deployment hosts as thoroughly as possible without turning my development laptop in to the deployment hosts feels like an isolation issue, so I reached for containers and Docker. Specifically, I wanted to start with a container image as close to the VM image used by the deployment hosts as possible and figure out how to run the actual application container images within those “host containers”. IOW, can I run Docker containers within a Docker container? Yup, for some time now.

It turns out that Docker now provides an official Docker-in-Docker (DinD) image. After reading it’s ./Dockerfile and some blog posts detailing how to run DinD, I decided it would be more efficient and maintainable to use the official docker:dind image as my base image and extend it to match the project’s deployment hosts rather than the other way around, start with a base image close to the deployment hosts and get DinD working in those. A quick ./Dockerfile reproduced the deployment host content in an image, including deployment host filesystem paths and users, e.g. /home/ec2-user/:

FROM docker:dind
# Defensive shell settings, avoid silent failures
SHELL ["/bin/ash", "-xeu", "-c"]

# Install required OS host packages
RUN \
        apk --no-cache add 'py-pip' 'python3-dev' 'libffi-dev' 'openssl-dev' 'gcc' \
        'libc-dev' 'rust' 'cargo' 'make' && \
        pip3 install 'docker-compose'

# Duplicate the AWS EC2 user
RUN adduser --disabled-password --gecos 'EC2 User,,,' ec2-user

A ./docker-compose.yml file reproduces the running deployment hosts including network topology such as subnet and host IP addresses. I also abuse the ./docker-compose.yml file to build the application images:

version: "3.8"

networks:
  default:
    ipam:
      driver: "default"
      config:
        # Match the hosted deployment network
        - subnet: "10.81.82.0/24"

services:

# Build custom container image, to be loaded into the nested host Docker daemons.
  foo-app:
    build:
      context: "./foo-app/"
    image: "bar-company/foo-app:dev"
    # Build only, don't actually run as a service
    entrypoint: ["true"]

# Emulate deployment hosts running Docker daemons

  foo-host:
    build:
      context: "./"
    image: "bar-company/foo-host:dev"
    # Just build the image, don't actually run as a service
    entrypoint: ["true"]

  foo-host-corge:
    image: "bar-company/foo-host:dev"
    depends_on:
      - "foo-host"
    privileged: true
    networks:
      default:
        # Match the hosted deployment IP
        ipv4_address: "10.81.82.100"
    volumes:
      # Docker-in-Docker requires data on a real filesystem
      - "./var/lib/foo-host-corge/docker/:/var/lib/docker/"
      # Reproduce where project data is stored in hosted deployments
      - "./var/lib/foo-data/:/home/ec2-user/foo-data/"
      # Make source editable
      - "./foo-app/:/srv/foo-app/"
      # Reproduce deployment containers using a `$ docker-compose ...` configuration
      # specific to the deployment host
      - "./foo-host-corge/:/srv/foo-host-corge/"
    working_dir: "/srv/foo-host-corge/"

  foo-host-grault:
    image: "bar-company/foo-host:dev"
    depends_on:
      - "foo-host"
    privileged: true
    networks:
      default:
        # Match the hosted deployment IP
        ipv4_address: "10.81.82.101"
    volumes:
      # Docker-in-Docker requires data on a real filesystem
      - "./var/lib/foo-host-grault/docker/:/var/lib/docker/"
      # Reproduce where project data is stored in hosted deployments
      - "./var/lib/foo-data/:/home/ec2-user/foo-data/"
      # Make source editable
      - "./foo-app/:/srv/foo-app/"
      # Reproduce deployment containers using a `$ docker-compose ...` configuration
      # specific to the deployment host
      - "./foo-host-grault/:/srv/foo-host-grault/"
    working_dir: "/srv/foo-host-grault/"

Nested ./foo-host-corge/docker-compose.yml and ./foo-host-grault/docker-compose.yml files reproduce how the application containers are run on the real deployment hosts, but run instead within the nested DinD containers that reproduce the deployment hosts:

version: "3.8"

services:

  postgres:
    image: "postgres"
    ports:
      - "5432:5432"
    env_file:
      - "./.env"

  foo-app:
    image: "bar-company/foo-app:dev"
    ports:
      - "80:80"
    extra_hosts:
       - "redis:10.81.82.101"
    volumes:
      # Make source editable
      - "/srv/foo-app/:/srv/foo-app/"
      # Reproduce where project data is stored in hosted deployments
      - "/home/ec2-user/foo-data/:/srv/foo-data/"

version: "3.8"

services:

  redis:
    image: "redis"
    ports:
      - "6379:6379"

  foo-app:
    image: "bar-company/foo-app:dev"
    ports:
      - "80:80"
    extra_hosts:
       - "postgres:10.81.82.100"
    volumes:
      # Make source editable
      - "/srv/foo-app/:/srv/foo-app/"
      # Reproduce where project data is stored in hosted deployments
      - "/home/ec2-user/foo-data/:/srv/foo-data/"

Use some $ docker ... commands and some shell to push the application images into the nested # dockerd daemons and some $ docker-compose ... commands to run containers using those images, along with the rest of the deployment configuration. I use a ./Makefile to stitch this all together but use what you like:

# Reproduce hosted deployments in nested Docker (DinD)

### Defensive settings for make:
#     https://tech.davis-hansson.com/p/make/
SHELL := bash
.ONESHELL:
.SHELLFLAGS:=-xeu -o pipefail -O inherit_errexit -c
.SILENT:
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules


# Top-level targets

.PHONY: all
all: var/log/docker-compose-build.log foo-host-corge/.env

.PHONY: run
run: all
	docker-compose exec -T "foo-host-corge" docker-compose up -d
	docker-compose exec -T "foo-host-grault" docker-compose up -d
	sleep 1
	docker-compose exec -T "foo-host-corge" docker-compose ps
	docker-compose exec -T "foo-host-grault" docker-compose ps

.PHONY: test
test: run
# Demonstrate that deployment host network topology and hostnames are reproduced
	docker-compose exec foo-host-corge \
	    docker-compose exec foo-app nc -vz redis 6379
	docker-compose exec foo-host-grault \
	    docker-compose exec foo-app nc -vz postgres 5432
# Demonstrate that deployment host filesystem paths and data are reproduced
	sudo rm -rfv ./var/lib/foo-data/*
	ls -al "./var/lib/foo-data/"
	docker-compose exec foo-host-corge \
	    ls -al "/home/ec2-user/foo-data/"
	docker-compose exec foo-host-grault \
	    ls -al "/home/ec2-user/foo-data/"
	docker-compose exec foo-host-corge docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    test ! -e "/srv/foo-data/bar.txt"
	sudo touch "./var/lib/foo-data/bar.txt"
	docker-compose exec foo-host-corge docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    ls -al "/srv/foo-data/"
	docker-compose exec foo-host-grault docker-compose exec foo-app \
	    test -e "/srv/foo-data/bar.txt"

.PHONY: clean
clean:
	docker-compose down --rmi local -v
	sudo rm -rf "./var/lib/"
	mkdir -pv "./var/log/backups/"
	test ! -e "./var/log/docker-compose-build.log" ||
	    mv --backup=numbered -v "./var/log/docker-compose-build.log" \
	        "./var/log/backups/"
	test ! -e "./foo-host-corge/.env" ||
	    mv --backup=numbered -v "./foo-host-corge/.env" \
	        "./var/log/backups/"

# Real targets

var/log/docker-compose-build.log: foo-app/Dockerfile Dockerfile docker-compose.yml
	mkdir -pv "./$(dir $(@))/"
	sudo docker-compose build --pull | tee -a "./$(@)"
# Wait for the nested deployment host `# dorckerd` daemons to become available
	docker-compose up -d
	for host in foo-host-corge foo-host-grault
	do
	    timeout --foreground -v 20 $(SHELL) $(.SHELLFLAGS) "\
		while ! docker-compose exec -T "$${host}" docker ps; do sleep 0.1; done"
# Load the built image into the deployment host Docker daemons
	    docker save "bar-company/foo-app:dev" |
		docker-compose exec -T "$${host}" docker load
	done

foo-host-corge/.env:
	echo "POSTGRES_PASSWORD=$$(apg -M NCL -n 1)" >"./$(@)"

Once it’ up and running we can see that the network topology, host names, and containerized services and applications are running as they would in the real hosted deployment:

$ make test
+ docker-compose exec -T foo-host-corge docker-compose up -d
foo-host-corge_foo-app_1 is up-to-date
foo-host-corge_postgres_1 is up-to-date
+ docker-compose exec -T foo-host-grault docker-compose up -d
foo-host-grault_redis_1 is up-to-date
foo-host-grault_foo-app_1 is up-to-date
+ sleep 1
+ docker-compose exec -T foo-host-corge docker-compose ps
          Name                         Command               State           Ports
-------------------------------------------------------------------------------------------
foo-host-corge_foo-app_1    python -m http.server --di ...   Up      0.0.0.0:80->80/tcp
foo-host-corge_postgres_1   docker-entrypoint.sh postgres    Up      0.0.0.0:5432->5432/tcp
+ docker-compose exec -T foo-host-grault docker-compose ps
          Name                         Command               State           Ports
-------------------------------------------------------------------------------------------
foo-host-grault_foo-app_1   python -m http.server --di ...   Up      0.0.0.0:80->80/tcp
foo-host-grault_redis_1     docker-entrypoint.sh redis ...   Up      0.0.0.0:6379->6379/tcp
+ docker-compose exec foo-host-corge docker-compose exec foo-app nc -vz redis 6379
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.81.82.101:6379.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
+ docker-compose exec foo-host-grault docker-compose exec foo-app nc -vz postgres 5432
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.81.82.100:5432.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
+ sudo rm -rfv './var/lib/foo-data/*'
+ ls -al ./var/lib/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 08:49 .
drwxr-xr-x 5 root root 4096 Apr  5 08:45 ..
+ docker-compose exec foo-host-corge ls -al /home/ec2-user/foo-data/
total 8
drwxr-xr-x    2 root     root          4096 Apr  5 15:49 .
drwxr-sr-x    1 ec2-user ec2-user      4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault ls -al /home/ec2-user/foo-data/
total 8
drwxr-xr-x    2 root     root          4096 Apr  5 15:49 .
drwxr-sr-x    1 ec2-user ec2-user      4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-corge docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:49 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:49 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
+ docker-compose exec foo-host-grault docker-compose exec foo-app test '!' -e /srv/foo-data/bar.txt
+ sudo touch ./var/lib/foo-data/bar.txt
+ docker-compose exec foo-host-corge docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:50 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
-rw-r--r-- 1 root root    0 Apr  5 15:50 bar.txt
+ docker-compose exec foo-host-grault docker-compose exec foo-app ls -al /srv/foo-data/
total 8
drwxr-xr-x 2 root root 4096 Apr  5 15:50 .
drwxr-xr-x 1 root root 4096 Apr  5 15:45 ..
-rw-r--r-- 1 root root    0 Apr  5 15:50 bar.txt
+ docker-compose exec foo-host-grault docker-compose exec foo-app test -e /srv/foo-data/bar.txt

There you have it. The demo code is publicly available. I think we can find more uses for DinD. In particular, I have always found local development clean build issues to be a persistent plague: “It worked for me when I committed it!”. Next up I’m working on using DinD as a route to a generic local clean build test.

Docker Gotchas I’ve Encountered

2021-04-08T00:00:00+00:00

Time to share the snags/gotchas I’ve run into developing and deploying with Docker containers.

On my recent projects, I seem to have become the Docker wrangler on the team. Over that time, I’ve hit a number of snags or gotchas and thought, “Maybe I should write that spot down?”. It’s as good a time as any now that I’ve reduced the impedance of my blogging platform.

Disk Space

TL;DR: CAUTION! $ docker system prune -a --volumes

It’s up to you to clean up unused images, containers, volumes, networks, etc.. This one will likely seem obvious to anyone who understands now how Docker works, but it certainly bit me a few times when I was climbing the learning curve.

I’m not a big fan of this fact, but it’s important to understand that the $ docker ... CLI is built only to perform the underlying operations, the building blocks, required for containerized applications. The important thing to pay attention to there is what it is not. It is not a system for managing images, containers, volumes, etc., at least not across deployments, over time, or between different applications. It is not a declarative system for describing what images should be used to run which containers connected to which volumes and networks on a given host. Even $ docker-compose ... is really only a different syntax to write a related collection of $ docker ... CLI commands in a way that is much more readable, maintainable, and feels declarative.

Stopping a container does not remove it. Removing a container does not remove the image, volumes or networks. Removing the image may not remove it’s base image or Layers. And so on. To use other terminology, neither the $ docker ... nor the $ docker-compose ... CLIs are orchestration systems. As such, a developer must clean up the inevitable large quantities of Docker cruft themselves. If, heavens forfend, you’re deploying containers without an orchestration system, then you’ll also have to do the same there. A recent project was my first AWS ECS exposure, so maybe it was being used wrong, but I was surprised to learn that this cruft also accrued for that usage of ECS.

The easiest way to take care of this is to use the docker system prune command (or the prune sub-command under the other $ docker ... commands) but it’s important to understand what it does lest you destroy data or even code in a certain sense. When Docker says “prune” they mean “relative to all currently running containers”. Say you have a debug container hanging around into which you’ve installed a rich set of OS/dist packages, configured things just the way you like, and even written a few utility or introspection scripts you’ve come to rely on. This debug container is stopped most of the time and only run when you need it. Firstly, never do this! Always treat containers as ephemeral and able to be destroyed at any time, but lets continue with this example. If this debug container isn’t running when you run $ docker system prune, then the container, its filesystem, its image, everything will be destroyed irrevocably.

As such, only run $ docker system prune when you’re sure every container is currently running whose, filesystem, image, volumes, etc. you need to preserve. See also the options/flags to that command for more thorough cleanup. IMO, if it’s not always safe to run $ docker system prune then you’re doing something wrong, and that stands for both local development and deployments.

Publicly Exposed Ports

TL;DR: Always specify the host IP for port mappings, e.g. 127.0.0.1:80:80.

Unlike SSH port forwarding, which defaults to only binding to the localhost IP, 127.0.0.1, the docker run -p … option binds all IPs by default. This is also true for $ docker-compose ... since it’s really just an alternate syntax for the $ docker ... CLI. So to prevent exposing the top secret application you’re developing on your laptop to everyone at the cafe who can copy and paste a $ nmap ... command, just default to always specifying 127.0.0.1 as the bind address. It’s a cheap way to force yourself to think about it and to make the intention of all port mappings clear to anyone else who reads them while simultaneously making an audit of intentionally exposed ports as simple as $ git grep -r '0\.0\.0\.0:[0-9]+:[0-9]+'. As for deployments, I’d personally much rather have a release break a deployment than have a release expose a service to the internet unintentionally and silently.

YAML Needs Some Zen

TL:DR; Quote every YAML value except numbers, booleans, and null.

$ python -c 'import this'
The Zen of Python, by Tim Peters
...
Explicit is better than implicit.
...

I love YAML, so a quick rant. Personally, I find the tendency among developers to find a flaw in a technology and then develop a long lasting hatred because it cost them some time once. Our job is to solve difficult technical problems. Our job is also to collectively build fruitful ecosystems of tools, frameworks, and other technology. I personally think that embracing what’s good about a technology is more productive than dismissing a whole technology because of a set of flaws that are a subset of its features. Using something and complaining is much more fruitful than complaining and not using. Is what you’re saying productive criticism or just complaining? I know the programmer’s virtues are delightfully subversive, but I don’t think whining is among them.

In fact, I’d love to see the YAML parser libraries add options, if not defaults in new major versions, to disable the following problematic type conversions. That could put pressure on the standard to move such surprising behavior to an explicit opt-in whereby it could still be powerful for those that need it but no longer surprising for the rest of us. Or maybe just the standard first, I don’t really know how these things work. ;-) Enough ranting.

I lied, next rant but opposite tone. YAML does include some fairly distressing magical behavior. Try to digest this:

>>> yaml.safe_load('port: 22:22')
{'port': 1342}

I lost more time than I care to admit trying to figure out why a ./docker-compose.yml SFTP port mapping wasn’t working when everything started up with no error. I eventually did discover that port 1342:1342 was being mapped but it certainly didn’t help me understand. My other port mappings were working without any surprises:

>>> yaml.safe_load('port: 80:80')
{'port': '80:80'}

What the actual fork! This is the very definition of surprising behavior. The root cause here is that YAML has support for several obscure value types, and one of them is base 60 integers. I would really love to hear someone make the case that the use cases for sexagesimal are important and powerful enough for such a majority of developer users that is justifies this surprise for the rest of us. Who knew the Babylonians are such a significant constituency!?

There are, however, other such value types that can result in similar unaccountably surprising behavior. Just avoid these issues, always quote every YAML value except numbers, booleans, and null unless and until you need any of that black magic:

>>> yaml.safe_load('port: "22:22"')
{'port': '22:22'}

Build Context and Image Stowaways

TL;DR: $ ln -sv "./.gitignore" "./.dockerignore"

Docker uses what it calls the build context, the directory containing the ./Dockerfile by default, to determine what is available to COPY into an image while building. That makes sense to me. What doesn’t make sense to me is that it copies the whole build context somewhere before building an image. If past is prologue I’d end up agreeing if I understood the full reasons for that, but until then I remain dubious. Moving on. Regardless of how the build context is handled or managed, the notion of the build context does sensibly define what will make it into the image.

When the build context includes unnecessary files and directories, at best it just slows down your image builds and maybe also increases your built image size to no benefit. At worst, it leaks content that wasn’t intended into a deployed image, or worse a publicly released image. I’ll be honest, though, I mostly get concerned about the former, unnecessarily long times build hurt a lot when it’s in the inner loop of developing an image. The method Docker provides to control which files under the build context directory should not be included in the build context is the ./.dockerignore file.

So then just remember to add paths to ./.dockerignore every time you do something that might add something to the build context that you wouldn’t want included in an image. Simple, right? Don’t we all know that developers are really good at keeping a long list of rote considerations in mind? Well I’m not. I also have to say I’m not the only one, given how many other developers on teams I’ve worked on have made changes that should be accompanied by additions to ./.dockerignore.

You know what draws my attention to new files in my build context, and very reliably? Well a new file represents a change and in such cases the new file is a consequence of some other change in the source code. We use tools to manage changes that don’t depend on developers remembering such considerations, don’t we? Alright, enough patronizing sarcasm. VCS, good ol’ $ git status, is how I notice when some change I made resulted in new files in the build context.

This might be controversial, but in spite of the many posts I’ve found while searching which say they should be managed separately, I’ve been strongly advocating for making ./.dockerignore a symlink that points to ./.gitignore for ~2 years now. I haven’t yet regretted it once. In that same time, I’ve also worked on projects where ./.dockerignore is managed separately and inappropriate files leaking into the build context has been a recurring issue on all of them, and not just from my commits.

There are frequent cases where a build artifact should not be committed to VCS but should be included in built images. Here we can exploit one of the differences between ./.gitignore and ./.dockerignore, namely that $ git ... processes ./.gitignore files in each descendant directory under the checkout but $ docker build ... does not. So make sure your build artifacts go somewhere in sub-directories of the checkout (a good idea in any case) and place your ./.gitignore rules for those artifacts at the appropriate level down within that sub-directory.

Also note the other differences between ./.gitignore and ./.dockerignore and keep your rules to those that are interpreted the same by both $ git ... and $ docker build .... In particular, replace your ./.gitignore rules meant to apply to files that match in any sub-directory, e.g. foo.txt, with more explicit rules that work in ./.dockerignore as well, e.g.:

/foo.txt
**/foo.txt

While more verbose, I prefer this anyways as it’s more explicit and communicates to me what’s intended instantly and without ambiguity.

TTFN

Well those are the big gotchas I’ve encountered. I’m sure I won’t encounter any more. I’m sure there won’t be any forthcoming “Docker Gotcha: …” posts. ;-)

Reflections on TTW Programming from the Future

2021-04-07T00:00:00+00:00

Playing with IFTTT and Zapier has me remembering the TTW programming fallout and debate.

I’ve been working on feeding my ABlog posts to social media. I’ve been experimenting with some external services for this, primarily because implementing something self-hosted seems like overkill for this and it makes an excuse to explore something I’ve been wanting to. I’m intrigued by these services that claim to let you connect little pieces of functionality, through an intuitive UI, to create “custom” mini-“applications”.

I’m old enough to have cut my teeth on some of the first Through-The-Web (TTW) programming offerings and thus long enough to watch the problems that became of that. The consensus among the communities I’ve been involved in is that it’s a bad idea to invite users and tinkerers to build their own custom applications with the promise that it’s easier and achievable because they can do it in their browser. The issue is not that non-programmers can’t learn to program. They certainly can and I object to the kind of rarefied elitism that leads others to argue otherwise. The issue is that no one can build the kind of custom application one can build with a full programming language without also learning most, if not all, of the suite of programming best practices. Without those best practices, one is all but guaranteed to end up with a poorly engineered, unmaintainable mess. The best case scenario is that your application fails to become useful and thus never has a chance to become a burden. That leaves those who manage to create an actually useful, initially successful application that users come to depend on. These successful tinkerers are eventually rewarded by hitting multiple walls at once. Thinking about, planning for, and executing on maintainability and other best practices are simply costs that must be paid. The apparent savings of not having to think about such things is an illusion and those costs have been compounding all along behind your back.

While the 10th feature was as easy to implement as the 1st, the 100th feature is 10 times as difficult as the 10th and breaks features 20-23; you can infer the curve. You took a stab at a major version upgrade of the framework months ago, hit another wall, and set it aside. Now the security/LTS of your major version is about to expire meaning your custom application will soon be vulnerable to multiple publicly available exploits. Change management is becoming a nightmare with duplicated code all over the place in an effort to compensate. A while back, you learned about VCS and the current best-of-breed in that domain but you also realized that you couldn’t use those tools to manage your TTW code. There are better ways to build custom applications even within the framework you’re using, but not TTW, the effort is comparable to re-implementing the application, and those that can do it justly charge much more. BTW, having learned so much from this, you’re now worth a lot more yourself and are recruited into your first junior programming job. Those last two are good things, but the end result is that a team of users is now dependent on a custom application that’s going to cost them a lot to either replace or maintain. They became a company in the business of developing software without ever intending to, planning on or budgeting for.

That whole process breeds resentment in a way that doesn’t come from even an equally blundering and costly but conscious effort to develop one’s own custom software. Like much of human happiness, it’s expectations, the surprise, that’s the issue. That’s how I understand the issue with TTW programming. If a framework provides anything approaching a full programming language, and offers or promises to save users-come-developers some portion of software development “hassles”, then everyone is almost certainly in for some bad-will inducing surprises.

So that means that any system that allows users to assemble “applications” through a UI meant to spare them from some portion of software development discipline needs to have clear, rigid boundaries on expectations. It needs to narrowly define what can be done, place firm limits on the size of what can be built (roughly, the quantity of components). It doesn’t hurt to provide users an eject button so that they can export what they’ve done to the filesystem and continue developing it incrementally using a full software development tool chain and process. These are the lessons the Plone community tried to learn from our Zope TTW roots. The Plone community built web UI based on these conclusions for both custom content types with custom schemata and for custom themes.

Given my involvement in that long (for software) history, I’ve been wanting to check out IFTTT. Such custom applet “development” and hosting platforms are solving different problems, very simple workflow applications connecting various internet services, but I imagine theses same principles apply. IFTTT seems to employ these principles quite firmly. I’m not aware of any eject button that could meaningfully allow one to incrementally build software on top of IFTTT applets, but that wouldn’t be terribly useful anyways given how firmly they restrict the scope of what one can do. I don’t have any current need for anything beyond IFTTT’s simple, two-step composition model, the this component and the that component in “If this then that”, but I could well imagine wanting more than that. There’s also the limit on 3 applets on the free tier of their freemium pricing plan, I am already using 2. Finally, I couldn’t find a then component for posting to LinkedIn as a service, that’s the one that really lead me to seek other options.

I ran across Zapier a while back and was similarly intrigued, so when that showed up in my search for something that could post to LinkedIn, I eagerly tried it out. They offer 5 applets, which they call “Zaps” (bit of a groaner of a term), on the free tier of their freemium pricing plan, so point for Zapier. I haven’t read much of their docs, but it’s apparent in the web UI that there’s support for at least a somewhat arbitrary chaining of more than 2 components, another win for Zapier. I also found their options for dealing with different “fields” in data, atom feed items in this case, to be significantly more rich than those offered by IFTTT, Zapier streaks ahead. They do indeed have components that interact with LinkedIn, so Zapier FTW in this very cursory and superficial scoring.

In the current moment, I’m very curious to see how the more restrictive IFTTT compares in the long term to the more feature-rich and composible Zapier in terms of user experience, user maintainability, and internal maintainability. Even if I were to pay enough attention to keep up on this, not very likely, I’m even less likely to get access to the kind of proprietary internal discussion that could yield such insight. C’est la capitalisme. At any rate, It’s been fun to reflect on the issue while automating some things.

Using Directories for Sphinx Pages

2021-04-05T00:00:00+00:00

Creating Sphinx pages as ./foo/index.rst has a number of benefits over ./foo.rst including path consistency and organizing content.

When I first started playing with ABlog, and thus learning more about Python’s wonderful documentation tool Sphinx, I observed that Sphinx actually creates a directory in the built HTML output for each reStructuredText ./**.rst file in the source. For example, if the source for this post had been ./blog/sphinx-page-dirs.rst, Sphinx would render that to ./_website/blog/sphinx-page-dirs/index.html.

One consequence of this is that a relative link to another area of the site will be broken in the rendered HTML build output if it’s correct in the ./**.rst source file. For example, if ./about.rst contains a link such as:

See `the blog area to see all posts <./blog/>`_.

The Sphinx HTML build output will render that ./about.rst file to ./about/index.html and thus the link will point to /about/blog/ instead of the intended /blog/. I dimly recall encountering such path issues with other types of reST path references as well. Path consistency is also particularly handy for editors or other tools that can infer enough to open files from path references in other files, such as FFAP in Emacs. Writing the source reST for a page as ./about/index.rst in the first place nicely avoids those problems and empowers such tools.

I’m not a big fan of embedding code of one sort into content (or code) of another sort. For example, as big an Org-mode fan as I am, I’ve never been able to get into using Babel. In this case, when writing documentation or other text content, I resist embedding code into source text files and prefer to have such code in separate files next to the text. If the code I want to include in documentation is actual used code, as was the case with the export/import scripts that migrated this blog, then it saves time copying and pasting and prevents out-of-date content when forgetting to do so. Even if it is example or demonstration code of more than a few lines, having it in separate sibling files supports using all our favorite tools (static analysis, IDEs, editors, etc.) which further prevents incorrect or misleading examples. So instead of reStructuredText’s :: literal blocks or Markdown’s indented code blocks I’d much rather include code from separate files. Using a directory for such pages, e.g. ./blog/migrating-from-plone-to-ablog/, also allows us to keep such related files organized neatly together.

Finally, using a directory for a Sphinx page supports putting all related content into a separate VCS repository checkout, such as via $ git submodule .... This can be very useful for sharing code related to one page without having to share the source and VCS history for the whole site. For example, I have an upcoming post with a non-trivial amount of working and tested demo code I want to share. I can use this approach to make that code and VCS history public while still preserving the option of having private content or VCS history for the rest of this site.