Wether you are operating cattle or pets, you will probably want to automate Docker resource cleanup, and using prune is a point of no return, meaning it could even lead to possible data loss when pruning volumes. The open-source docker-prune project tries to provide an alterative that is more conservative in the pruning decisions that it takes. For recurring operations, I recommend dockron.

docker-prune is a POSIX shell script that will prune exited containers, dangling volumes and dangling images with the following twist:

Containers

All exited, dead and stale containers will be removed, and this provides filtering capabilities similar to the prune command. Containers that have a name that was automatically generated by Docker at creation time are automatically selected. In addition, when removing containers, the script can consider only a subset of the containers.

Exited and dead containers are as reported by Docker. Stale containers are containers that are created but have not moved to any other state after a given timeout.

In addition, it is possible to forcedly remove ancient, but still running containers. This might be a dangerous operation, and it is turned off by default.

Images

All dangling and orphan images will be removed. This also provides filtering capabilities similar to the prune command. When removing images, the script will only consider images that were created a long time ago (6 months by default).

Dangling images are layers that have no relationship to any tagged images. Orphan images are images that are not used by any container, whichever state the container is in (including created or exited state).

Volumes

All “empty” dangling volumes will be removed. The script will count the files inside the volumes, only removing the ones which have less than an optional number of files, which defaults to 0. In addition, the script will is able to focus on subsets of the dangling volumes. Volumes that have a name that was automatically generated are automatically selected. File count is achieved through mounting the volumes into a temporary busybox container.

Fun Fact

The script dynamically parses the official go implementation to detect containers which names were automatically generated. Out of this implementation, this is probably the best line of code ever written!

Capabilities

The first step to manage and understand is how to give away enough rights to the container that will perform the mount so that other processes on the host (outside that container) will be able to access the files and directories. This can be achieved through using the following options for docker run:

• Give the fuse device to your container through --device /dev/fuse
• Raise its capabilities to bypass some of the security layers through: --cap-add SYS_ADMIN and --security-opt "apparmor=unconfined"

• Mount the local host file system into the container in a way that it can be shared back with other processes and containers using rshared.

Unmounting

You also want to properly unmount when the container gracefully terminates. This is achieved through a combination of tini and proper trap in the shell. The reason for this is that your are likely to interface the fuse implementation through some sort of entrypoint

tini

Use tini as the main entrypoint in your Dockerfile and arrange to give it the -g option so that it properly propagates signals to the entire mounting system in the container. tini exists as an Alpine package which can be helpful in keeping down the size of the image.

Cleanup

In order to clean properly, you can arrange for a shell to be called at the end of your entrypoint implementation, perhaps after having checked that the mount performed as it should. This shell will trap the INT and TERM signals, unmount the volume (lazily) and propagate these to the mount process. Without tini, you would never have received these signals, by Docker construction and design.

Examples

I have made available two example images following these principles:

• webdav-client mounts WebDAV resources and leverages the full potential of davfs2, being able to leverage all the options supported by davfs2. This is in contrast to the davfs volume which also uses davfs2 under the hood, but misses some of the configuration options.

• s3fs mounts a remote S3 bucket using the fuse implementation with the same name. It supports all the official versions of the original fuse projects through tags

# This is the image that you wish to list the tags for
im="abiosoft/caddy"

if [ -z "$(echo "$im" | grep -o '/')" ]; then
    hub="https://registry.hub.docker.com/v2/repositories/library/$im/tags/"
else
    hub="https://registry.hub.docker.com/v2/repositories/$im/tags/"
fi

# Get number of pages
if [ -z "$(command -v curl)" ]; then
    first=$(wget -q -O - $hub)
else
    first=$(curl -sL $hub)
fi
count=$(echo $first | sed -E 's/\{\s*"count":\s*([0-9]+).*/\1/')
pagination=$(echo $first | grep -Eo '"name":\s*"[a-zA-Z0-9_.-]+"' | wc -l)
pages=$(expr $count / $pagination + 1)

# Get all tags one page after the other
tags=
i=0
while [ $i -le $pages ] ;
do
    i=$(expr $i + 1)
    if [ -z "$(command -v curl)" ]; then
        page=$(wget -q -O - "$hub?page=$i")
    else
        page=$(curl -sL "$hub?page=$i")
    fi
    ptags=$(echo $page | grep -Eo '"name":\s*"[a-zA-Z0-9_.-]+"' | sed -E 's/"name":\s*"([a-zA-Z0-9_.-]+)"/\1/')
    tags="${ptags} $tags"
done

# Once here, the variable tags should contain the list of all tags for the image.

This code can be used in hooks to write complex build/push instructions. Such instructions can be used to automatically enhanced a standard image with additional features in a future-proof way: for every new version of the original image, watching it will ensure that your enhanced version will get built.

Let’s run roughly through the example of running grafana in a scalable way, for example using Redis and Postgres for, respectively, runtime and persistent data. To make this more complex, let’s suppose that the TOML configuration file of grafana is the result of another service. A regular Grafana Docker installation neither has an entrypoing, nor a run command as it is mostly configured using a combination of environment variables and the TOML configuration file. To arrange for the other services to be ready, you could have the following (snippet) entrypoint:

    entrypoint: >-
      wait-for.sh pg_grafana:5432 -t 120 -v --
        wait-for.sh redis:6379 -t 120 -v --
          wait-for.sh grafana-init:8080 -t 120 -v --
            /run.sh

In that example, pg_grafana, redis and grafana-init are the name of the services that implement (respectively) the postgres database, Redis and the initialisation of the TOML configuration. Creating them is (almost) out of the scope of this post… The implementation for wait-for.sh is available as a gist. It deviates slightly from the original as it prefers nc for trying to establish connection to remote servers, but is also able to use pure bash constructs whenever bash is available.

grafana-init is a bit special as it typically generates a configuration file depending on a number of parameters. In a regular installation, such a service would “die” once it had created the configuration file. However, instead it can wait forever once it has performed its initialisation job. I have experimented with relaying another external service with socat, as this is available in most distributions. I typically call the TOML generator with a commane-line option similar to -r 8080:icanhazip.com:80, you will notice that 8080 is the port that the grafana service was waiting for. The implementation is as follows, provided the content of the -r switch is contained in the variable RELAY. Even if the remote service was not available, initialisation would still work as socat would still respond on 8080 on incoming client requests.

if [ -n "$RELAY" ]; then
    lport=$(echo "${RELAY}"|cut -d: -f1)
    remote=$(echo "${RELAY}"|cut -d: -f2)
    rport=$(echo "${RELAY}"|cut -d: -f3)
    socat TCP4-LISTEN:${lport},su=nobody,fork,reuseaddr TCP4:${remote}:${rport}
fi

version: '3.3'

services:
  grafana:
    image: grafana/grafana:5.2.0-beta1
    environment:
      - GF_SECURITY_ADMIN_PASSWORD_FILE=/run/secrets/admin.pwd
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      replicas: 1
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "10"
    healthcheck:
      test: curl --fail http://localhost:3000/ || exit 1
      interval: 1m
      timeout: 10s
      retries: 3
    secrets:
      -
        source: admin-passwd
        target: /run/secrets/admin.pwd
        mode: 0444

secrets:
  admin-passwd:
    file: config/grafana/admin.pwd

For any environment variable that starts with GF_ and ends with _FILE, the Grafana Docker image will read the content of the file that it points at and arrange for the environment variable with the same name but without the trailing _FILE to be set before the main grafana process is started. Using a trailing _FILE is in line with other official images such as postgres or wordpress for example.

Exporting the Service

Start by jumping into your container using docker exec.

$ docker exec -it <containerID> ash

Then from within the container, make sure to install an SSH client. No need for a server here.

$ apk add --no-cache openssh-client

Once installation has succeeded, establish a reverse tunnel onto your host. Provided that you have a server, within your container, that listens on port 8088, a command similar to the following will make the same port available at the remote host and return back to the command line of the container (this is the meaning of -fNT). The command will require you to login at the remote host, etc. Note that if you did not want to use the same port at the remote host, you would change the first port number in the argument to -R below.

$ ssh -fNT -R8088:localhost:8088 emmanuel@<yourhost>

It is a good idea to keep the session into the container running, so you can easily return when cleaning up.

Accessing the Service

Now, login to the remote SSH host. You should be able to access the service running in the container directly under the port 8088. You can check that it works using something like nc for example:

$ nc -v localhost 8088

Provided that you have a working Docker environment on that host, you can even use the port for developing from within a container running on that remote SSH host. This involves sharing the host network with the container and, perhaps, mounting your development directory into the container. Running the following command would give you an Alpine based container with your development directory mounted on /data and where you are able to access 8088 from the container that you wanted to debug.

$ docker run -it --rm --network=host -v `pwd`:/data alpine

Cleaning Up

To clean up, apart from leaving the debugging container described in the previous section, you should return to the initial container, kill the ssh process running there are remove the ssh client package from the distribution.

Rationale

The rationale of concocter is to acquire variables from a number of remote of local sources, to dynamically generate (configuration) files with the content of these variables and to launch one or several processes once the files have been generated. concocter can be placed in the background to continuously perform these tasks, thus being able to regenerate the file as soon as a variable has changed. In that case, concocter will restart the process, or request it to reload its configuration using regular signals.

concocter has support for a range of sources for these variables. This includes information about other containers running on the same Docker host, but also the content of files (good for integration with Docker secrets), or the content of a external HTTP(S) resources. The latter faciliates the use of, for example, an internal key-value store with an RESTish interface for the storage and access of cluster or project wide configuration variables. Using concocter together with a capable reverse-proxy server such as nginx or caddy provides ways to automatically proxy containers carrying “instructions” as Docker labels or environment variables.

Discussion and Community

The github project provides support for issue tracking and enhancement proposals.

Installing on Ubuntu

To install LFS on Ubuntu 16.04+, perform the following. This is almost as advertised on the LFS home page and at PackageCloud, with the slight tweak that you need to actually install the extra package, since the bash script will only add the repository.

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install

Why

My repository of tclkits uses LFS storage to store binaries that were (cross-)compiled by the excellent KitCreator. This permit other projects to depend on those binaries in a lightweight form. For an example, have a look how this make script uses a contract to download these very binaries into a location that is under the git repository, but kept out of revision control.

The rationale for hosting those externally to the concocter project is to be able to offer the binaries and downloads as a service to the community as there seem to be few remotely hosted binaries with support for TLS.

Inside the wifi is a RealTek chipset 8821AU. Here is a rough guide for making it working with the latest kernel version. These have been tested with a beta of Ubuntu 18.04 LTS, but should be ok for prior versions as well:

1. Prepare for being able to compile additional stuff for your kernel: sudo apt install dkms build-essentials linux-headersXXX where XXX matches the kernel that you have.
2. Install git if you don’t already have it, you might want to run this from some git UI if you prefer.
3. Clone this repository. Note that there are a number of “competing” repos at github trying to solve the same problems, and not all of them seem to be working.
4. Change directory to the one of the repository above.
5. Issue sudo make -f Makefile.dkms install.
6. Load the driver sudo modprobe -i rtl8812au.
7. Verify that have a new network interface: sudo lshw -c network.

You should now be able to connect to any nearby wifi using the regular gnome tools (or from the command-line if you prefer). These instructions install DKMS so your drivers will be recompiled for each (new) version of the kernel.

The implementation was lagging behind and a recent restructuring have started to bring it to par with the currernt state of the Docker API itself. The restructuring matches the Docker CLI restructuring that happened with the 1.13.0 version. Recent additions to the Tcl command set provide bridges to the following sub-command trees of the API and CLI:

• container
• image
• service
• secret
• config
• node
• volume

There still remain a number of commands, but these new implementations provide a consistent API that is hopefully easier to work with than the previous uncategorised API, itself loosely modelled after the previous CLI model. Work with the integration of these new API calls is driven by the necessity to integrate some of these calls in dockron so as to be able to perform regular tasks on services within a Swarm, e.g. scale up/down at known times, restart services, etc.

(Funny fact of the day… Apparently, this API implementation made its way to HN soon after I announced it the first time!)