Allow for the exclusion of files from backups (#100 )

* Hoist walking of files so it can be used for features other than archive creation * Add option to ignore files from backup using glob patterns * Use Regexp instead of glob for exclusion * Ignore artifacts * Add teardown to test * Allow single Re for filtering only * Add documentation * Use MatchString on re, add bad input to message in case of error
Add documentation on using multiple configs for complex retention schemes
2025-12-05 09:08:02 +01:00 · 2022-05-08 11:20:38 +02:00 · 2022-04-20 18:01:41 +02:00 · 2022-04-20 14:16:59 +02:00 · 2022-04-20 10:56:26 +02:00 · 2022-04-15 11:05:52 +02:00
45 changed files with 2548 additions and 803 deletions
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -3,7 +3,7 @@ version: 2.1
 jobs:
  canary:
    machine:
-      image: ubuntu-1604:202007-01
+      image: ubuntu-2004:202201-02
    working_directory: ~/docker-volume-backup
    steps:
      - checkout
@@ -19,6 +19,7 @@ jobs:
          name: Run tests
          working_directory: ~/docker-volume-backup/test
          command: |
+            export GPG_TTY=$(tty)
            ./test.sh canary

  build:
@@ -47,6 +48,7 @@ jobs:
            if [[ "$CIRCLE_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
              # prerelease tags like `v2.0.0-alpha.1` should not be released as `latest`
              tag_args="$tag_args -t offen/docker-volume-backup:latest"
+              tag_args="$tag_args -t offen/docker-volume-backup:$(echo "$CIRCLE_TAG" | cut -d. -f1)"
            fi
            docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 \
               $tag_args . --push
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -0,0 +1,20 @@
+* **I'm submitting a ...**
+  - [ ] bug report
+  - [ ] feature request
+  - [ ] support request
+
+* **What is the current behavior?**
+
+* **If the current behavior is a bug, please provide the configuration and steps to reproduce and if possible a minimal demo of the problem.**
+
+* **What is the expected behavior?**
+
+* **What is the motivation / use case for changing the behavior?**
+
+* **Please tell us about your environment:**
+  
+  - Image version:
+  - Docker version: 
+  - docker-compose version:
+
+* **Other information** (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
--- a/13
+++ b/13
@@ -1,21 +1,22 @@
 # Copyright 2021 - Offen Authors <hioffen@posteo.de>
 # SPDX-License-Identifier: MPL-2.0

-FROM golang:1.17-alpine as builder
+FROM golang:1.18-alpine as builder

 WORKDIR /app
 COPY go.mod go.sum ./
 RUN go mod download
-COPY cmd/backup/main.go ./cmd/backup/main.go
-RUN go build -o backup cmd/backup/main.go
+COPY cmd/backup ./cmd/backup/
+WORKDIR /app/cmd/backup
+RUN go build -o backup .

-FROM alpine:3.14
+FROM alpine:3.15

 WORKDIR /root

-RUN apk add --update ca-certificates
+RUN apk add --no-cache ca-certificates

-COPY --from=builder /app/backup /usr/bin/backup
+COPY --from=builder /app/cmd/backup/backup /usr/bin/backup

 COPY ./entrypoint.sh /root/
 RUN chmod +x entrypoint.sh
--- a/README.md
+++ b/README.md
@@ -1,9 +1,13 @@
+<a href="https://www.offen.dev/">
+    <img src="https://offen.github.io/press-kit/offen-material/gfx-GitHub-Offen-logo.svg" alt="Offen logo" title="Offen" width="150px"/>
+</a>
+
 # docker-volume-backup

 Backup Docker volumes locally or to any S3 compatible storage.

 The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
-It handles __recurring or one-off backups of Docker volumes__ to a __local directory__ or __any S3 compatible storage__ (or both), and __rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
+It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3 or WebDAV compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.

 <!-- MarkdownTOC -->

@@ -12,22 +16,32 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
  - [One-off backups using Docker CLI](#one-off-backups-using-docker-cli)
 - [Configuration reference](#configuration-reference)
 - [How to](#how-to)
-  - [Stopping containers during backup](#stopping-containers-during-backup)
+  - [Stop containers during backup](#stop-containers-during-backup)
  - [Automatically pruning old backups](#automatically-pruning-old-backups)
  - [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs)
+  - [Customize notifications](#customize-notifications)
+  - [Run custom commands before / after backup](#run-custom-commands-before--after-backup)
  - [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg)
  - [Restoring a volume from a backup](#restoring-a-volume-from-a-backup)
  - [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in)
  - [Using with Docker Swarm](#using-with-docker-swarm)
  - [Manually triggering a backup](#manually-triggering-a-backup)
+  - [Update deprecated email configuration](#update-deprecated-email-configuration)
+  - [Replace deprecated `BACKUP_FROM_SNAPSHOT` usage](#replace-deprecated-backup_from_snapshot-usage)
+  - [Using a custom Docker host](#using-a-custom-docker-host)
+  - [Run multiple backup schedules in the same container](#run-multiple-backup-schedules-in-the-same-container)
+  - [Define different retention schedules](#define-different-retention-schedules)
 - [Recipes](#recipes)
  - [Backing up to AWS S3](#backing-up-to-aws-s3)
+  - [Backing up to Filebase](#backing-up-to-filebase)
  - [Backing up to MinIO](#backing-up-to-minio)
+  - [Backing up to WebDAV](#backing-up-to-webdav)
  - [Backing up locally](#backing-up-locally)
  - [Backing up to AWS S3 as well as locally](#backing-up-to-aws-s3-as-well-as-locally)
  - [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
  - [Rotating away backups that are older than 7 days](#rotating-away-backups-that-are-older-than-7-days)
  - [Encrypting your backups using GPG](#encrypting-your-backups-using-gpg)
+  - [Using mysqldump to prepare the backup](#using-mysqldump-to-prepare-the-backup)
  - [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup)
 - [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup)

@@ -72,7 +86,8 @@ services:
      - data:/backup/my-app-backup:ro
      # Mounting the Docker socket allows the script to stop and restart
      # the container during backup. You can omit this if you don't want
-      # to stop the container
+      # to stop the container. In case you need to proxy the socket, you can
+      # also provide a location by setting `DOCKER_HOST` in the container
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # If you mount a local directory or volume to `/archive` a local
      # copy of the backup will be stored there. You can override the
@@ -94,7 +109,7 @@ docker run --rm \
  --env AWS_SECRET_ACCESS_KEY="<xxx>" \
  --env AWS_S3_BUCKET_NAME="<xxx>" \
  --entrypoint backup \
-  offen/docker-volume-backup:latest
+  offen/docker-volume-backup:v2
 ```

 Alternatively, pass a `--env-file` in order to use a full config as described below.
@@ -121,11 +136,26 @@ You can populate below template according to your requirements and use it as you

 # BACKUP_FILENAME="backup-%Y-%m-%dT%H-%M-%S.tar.gz"

+# Setting BACKUP_FILENAME_EXPAND to true allows for environment variable
+# placeholders in BACKUP_FILENAME, BACKUP_LATEST_SYMLINK and in
+# BACKUP_PRUNING_PREFIX that will get expanded at runtime,
+# e.g. `backup-$HOSTNAME-%Y-%m-%dT%H-%M-%S.tar.gz`. Expansion happens before
+# interpolating strftime tokens. It is disabled by default.
+# Please note that you will need to escape the `$` when providing the value
+# in a docker-compose.yml file, i.e. using $$VAR instead of $VAR.
+
+# BACKUP_FILENAME_EXPAND="true"
+
 # When storing local backups, a symlink to the latest backup can be created
 # in case a value is given for this key. This has no effect on remote backups.

 # BACKUP_LATEST_SYMLINK="backup.latest.tar.gz"

+# ************************************************************************
+# The BACKUP_FROM_SNAPSHOT option has been deprecated and will be removed
+# in the next major version. Please use exec-pre and exec-post
+# as documented below instead.
+# ************************************************************************
 # Whether to copy the content of backup folder before creating the tar archive.
 # In the rare scenario where the content of the source backup volume is continously
 # updating, but we do not wish to stop the container while performing the backup,
@@ -133,6 +163,17 @@ You can populate below template according to your requirements and use it as you

 # BACKUP_FROM_SNAPSHOT="false"

+# By default, the `/backup` directory inside the container will be backed up.
+# In case you need to use a custom location, set `BACKUP_SOURCES`.
+
+# BACKUP_SOURCES="/other/location"
+
+# When given, all files in BACKUP_SOURCES whose full path matches the given
+# regular expression will be excluded from the archive. Regular Expressions
+# can be used as from the Go standard library https://pkg.go.dev/regexp
+
+# BACKUP_EXCLUDE_REGEXP="\.log$"
+
 ########### BACKUP STORAGE

 # The name of the remote bucket that should be used for storing backups. If
@@ -140,6 +181,11 @@ You can populate below template according to your requirements and use it as you

 # AWS_S3_BUCKET_NAME="backup-bucket"

+# If you want to store the backup in a non-root location on your bucket
+# you can provide a path. The path must not contain a leading slash.
+
+# AWS_S3_PATH="my/backup/location"
+
 # Define credentials for authenticating against the backup storage and a bucket
 # name. Although all of these keys are `AWS`-prefixed, the setup can be used
 # with any S3 compatible storage.
@@ -155,7 +201,7 @@ You can populate below template according to your requirements and use it as you
 # AWS_IAM_ROLE_ENDPOINT="http://169.254.169.254"

 # This is the FQDN of your storage server, e.g. `storage.example.com`.
-# Do not set this when working against AWS S3 (the default value is 
+# Do not set this when working against AWS S3 (the default value is
 # `s3.amazonaws.com`). If you need to set a specific (non-https) protocol, you
 # will need to use the option below.

@@ -168,12 +214,37 @@ You can populate below template according to your requirements and use it as you
 # AWS_ENDPOINT_PROTO="https"

 # Setting this variable to `true` will disable verification of
-# SSL certificates. You shouldn't use this unless you use self-signed
-# certificates for your remote storage backend. This can only be used
-# when AWS_ENDPOINT_PROTO is set to `https`.
+# SSL certificates for AWS_ENDPOINT. You shouldn't use this unless you use
+# self-signed certificates for your remote storage backend. This can only be
+# used when AWS_ENDPOINT_PROTO is set to `https`.

 # AWS_ENDPOINT_INSECURE="true"

+# You can also backup files to any WebDAV server:
+
+# The URL of the remote WebDAV server
+
+# WEBDAV_URL="https://webdav.example.com"
+
+# The Directory to place the backups to on the WebDAV server.
+# If the path is not present on the server it will be created.
+
+# WEBDAV_PATH="/my/directory/"
+
+# The username for the WebDAV server
+
+# WEBDAV_USERNAME="user"
+
+# The password for the WebDAV server
+
+# WEBDAV_PASSWORD="password"
+
+# Setting this variable to `true` will disable verification of
+# SSL certificates for WEBDAV_URL. You shouldn't use this unless you use
+# self-signed certificates for your remote storage backend.
+
+# WEBDAV_URL_INSECURE="true"
+
 # In addition to storing backups remotely, you can also keep local copies.
 # Pass a container-local path to store your backups if needed. You also need to
 # mount a local folder or Docker volume into that location (`/archive`
@@ -235,11 +306,75 @@ You can populate below template according to your requirements and use it as you

 # BACKUP_STOP_CONTAINER_LABEL="service1"

-########### EMAIL NOTIFICATIONS ON FAILED BACKUP RUNS
+########### EXECUTING COMMANDS IN CONTAINERS PRE/POST BACKUP

-# In case SMTP credentials are provided, notification emails can be sent out on
-# failed backup runs. These emails will contain the start time, the error
-# message and all log output prior to the failure.
+# It is possible to define commands to be run in any container before and after
+# a backup is conducted. The commands themselves are defined in labels like
+# `docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump [options] > dump.sql'.
+# Several options exist for controlling this feature:
+
+# By default, any output of such a command is suppressed. If this value
+# is configured to be "true", command execution output will be forwarded to
+# the backup container's stdout and stderr.
+
+# EXEC_FORWARD_OUTPUT="true"
+
+# Without any further configuration, all commands defined in labels will be
+# run before and after a backup. If you need more fine grained control, you
+# can use this option to set a label that will be used for narrowing down
+# the set of eligible containers. When set, an eligible container will also need
+# to be labeled as `docker-volume-backup.exec-label=database`.
+
+# EXEC_LABEL="database"
+
+########### NOTIFICATIONS
+
+# Notifications (email, Slack, etc.) can be sent out when a backup run finishes.
+# Configuration is provided as a comma-separated list of URLs as consumed
+# by `shoutrrr`: https://containrrr.dev/shoutrrr/v0.5/services/overview/
+# The content of such notifications can be customized. Dedicated documentation
+# on how to do this can be found in the README. When providing multiple URLs or
+# an URL that contains a comma, the values can be URL encoded to avoid ambiguities.
+
+# The below URL demonstrates how to send an email using the provided SMTP
+# configuration and credentials.
+
+# NOTIFICATION_URLS=smtp://username:password@host:587/?fromAddress=sender@example.com&toAddresses=recipient@example.com
+
+# By default, notifications would only be sent out when a backup run fails
+# To receive notifications for every run, set `NOTIFICATION_LEVEL` to `info`
+# instead of the default `error`.
+
+# NOTIFICATION_LEVEL="error"
+
+########### DOCKER HOST
+
+# If you are interfacing with Docker via TCP you can set the Docker host here
+# instead of mounting the Docker socket as a volume. This is unset by default.
+
+# DOCKER_HOST="tcp://docker_socket_proxy:2375"
+
+########### LOCK_TIMEOUT
+
+# In the case of overlapping cron schedules run by the same container,
+# subsequent invocations will wait for previous runs to finish before starting.
+# By default, this will time out and fail in case the lock could not be acquired
+# after 60 minutes. In case you need to adjust this timeout, supply a duration
+# value as per https://pkg.go.dev/time#ParseDuration to `LOCK_TIMEOUT`
+
+# LOCK_TIMEOUT="60m"
+
+########### EMAIL NOTIFICATIONS
+
+# ************************************************************************
+# Providing notification configuration like this has been deprecated
+# and will be removed in the next major version. Please use NOTIFICATION_URLS
+# as documented above instead.
+# ************************************************************************
+
+# In case SMTP credentials are provided, notification emails can be sent out when
+# a backup run finished. These emails will contain the start time, the error
+# message on failure and all prior log output.

 # The recipient(s) of the notification. Supply a comma separated list
 # of adresses if you want to notify multiple recipients. If this is
@@ -260,9 +395,14 @@ You can populate below template according to your requirements and use it as you
 # EMAIL_SMTP_PORT="<port>"
 ```

+In case you encouter double quoted values in your configuration you might be running an [older version of `docker-compose`].
+You can work around this by either updating `docker-compose` or unquoting your configuration values.
+
+[compose-issue]: https://github.com/docker/compose/issues/2854
+
 ## How to

-### Stopping containers during backup
+### Stop containers during backup

 In many cases, it will be desirable to stop the services that are consuming the volume you want to backup in order to ensure data integrity.
 This image can automatically stop and restart containers and services (in case you are running Docker in Swarm mode).
@@ -280,7 +420,7 @@ services:
      - docker-volume-backup.stop-during-backup=service1

  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      BACKUP_STOP_CONTAINER_LABEL: service1
    volumes:
@@ -303,7 +443,7 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
      BACKUP_PRUNING_PREFIX: backup-
@@ -326,16 +466,99 @@ version: '3'

 services:
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      # ... other configuration values go here
-      EMAIL_SMTP_HOST: "smtp.example.com"
-      EMAIL_SMTP_PASSWORD: "password"
-      EMAIL_SMTP_USERNAME: "username"
-      EMAIL_NOTIFICATION_SENDER: "noreply@example.com"
-      EMAIL_NOTIFICATION_RECIPIENT: "notifications@example.com"
+      NOTIFICATION_URLS=smtp://me:secret@smtp.example.com:587/?fromAddress=no-reply@example.com&toAddresses=you@example.com
 ```

+Notification backends other than email are also supported.
+Refer to the documentation of [shoutrrr][shoutrrr-docs] to find out about options and configuration.
+
+[shoutrrr-docs]: https://containrrr.dev/shoutrrr/v0.5/services/overview/
+
+### Customize notifications
+
+The title and body of the notifications can be easily tailored to your needs using [go templates](https://pkg.go.dev/text/template).
+Templates must be mounted inside the container in `/etc/dockervolumebackup/notifications.d/`: any file inside this directory will be parsed.
+
+The files have to define [nested templates](https://pkg.go.dev/text/template#hdr-Nested_template_definitions) in order to override the original values. An example:
+```
+{{ define "title_success" -}}
+✅ Successfully ran backup {{ .Config.BackupStopContainerLabel }}
+{{- end }}
+
+{{ define "body_success" -}}
+▶️ Start time: {{ .Stats.StartTime | formatTime }}
+⏹️ End time: {{ .Stats.EndTime | formatTime }}
+⌛ Took time: {{ .Stats.TookTime }}
+🛑 Stopped containers: {{ .Stats.Containers.Stopped }}/{{ .Stats.Containers.All }} ({{ .Stats.Containers.StopErrors }} errors)
+⚖️ Backup size: {{ .Stats.BackupFile.Size | formatBytesBin }} / {{ .Stats.BackupFile.Size | formatBytesDec }}
+🗑️ Pruned backups: {{ .Stats.Storages.Local.Pruned }}/{{ .Stats.Storages.Local.Total }} ({{ .Stats.Storages.Local.PruneErrors }} errors)
+{{- end }}
+```
+
+Overridable template names are: `title_success`, `body_success`, `title_failure`, `body_failure`.
+
+For a full list of available variables and functions, see [this page](https://github.com/offen/docker-volume-backup/blob/master/docs/NOTIFICATION-TEMPLATES.md).
+
+### Run custom commands before / after backup
+
+In certain scenarios it can be required to run specific commands before and after a backup is taken (e.g. dumping a database).
+When mounting the Docker socket into the `docker-volume-backup` container, you can define pre- and post-commands that will be run in the context of the target container.
+Such commands are defined by specifying the command in a `docker-volume-backup.exec-[pre|post]` label.
+
+Taking a database dump using `mysqldump` would look like this:
+
+```yml
+version: '3'
+
+services:
+  # ... define other services using the `data` volume here
+  database:
+    image: mariadb
+    volumes:
+      - backup_data:/tmp/backups
+    labels:
+      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /backups/dump.sql'
+
+volumes:
+  backup_data:
+```
+
+Due to Docker limitations, you currently cannot use any kind of redirection in these commands unless you pass the command to `/bin/sh -c` or similar.
+I.e. instead of using `echo "ok" > ok.txt` you will need to use `/bin/sh -c 'echo "ok" > ok.txt'`.
+
+If you need fine grained control about which container's commands are run, you can use the `EXEC_LABEL` configuration on your `docker-volume-backup` container:
+
+```yml
+version: '3'
+
+services:
+  database:
+    image: mariadb
+    volumes:
+      - backup_data:/tmp/backups
+    labels:
+      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /tmp/volume/dump.sql'
+      - docker-volume-backup.exec-label=database
+
+  backup:
+    image: offen/docker-volume-backup:v2
+    environment:
+      EXEC_LABEL: database
+    volumes:
+      - data:/backup/dump:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+
+volumes:
+  backup_data:
+```
+
+
+The backup procedure is guaranteed to wait for all `pre` commands to finish.
+However there are no guarantees about the order in which they are run, which could also happen concurrently.
+
 ### Encrypting your backup using GPG

 The image supports encrypting backups using GPG out of the box.
@@ -356,17 +579,37 @@ In case you need to restore a volume from a backup, the most straight forward pr
  ```console
  tar -C /tmp -xvf  backup.tar.gz
  ```
- Using a temporary one-off container, mount the volume (the example assumes it's named `data`) and copy over the backup. Make sure you copy the correct path level (this depends on how you mount your volume into the backup container), you might need to strip some leading elements
+- Using a temporary once-off container, mount the volume (the example assumes it's named `data`) and copy over the backup. Make sure you copy the correct path level (this depends on how you mount your volume into the backup container), you might need to strip some leading elements
  ```console
-  docker run -d --name backup_restore -v data:/backup_restore alpine
-  docker cp /tmp/backup/data-backup backup_restore:/backup_restore
-  docker stop backup_restore
-  docker rm backup_restore
+  docker run -d --name temp_restore_container -v data:/backup_restore alpine
+  docker cp /tmp/backup/data-backup temp_restore_container:/backup_restore
+  docker stop temp_restore_container
+  docker rm temp_restore_container
  ```
- Restart the container(s) that are using the volume 
+- Restart the container(s) that are using the volume

 Depending on your setup and the application(s) you are running, this might involve other steps to be taken still.

+---
+
+If you want to rollback an entire volume to an earlier backup snapshot (recommended for database volumes):
+
+- Trigger a manual backup if necessary (see `Manually triggering a backup`).
+- Stop the container(s) that are using the volume.
+- If volume was initially created using docker-compose, find out exact volume name using:
+  ```console
+  docker volume ls
+  ```
+- Remove existing volume (the example assumes it's named `data`):
+  ```console
+  docker volume rm data
+  ```
+- Create new volume with the same name and restore a snapshot:
+  ```console
+  docker run --rm -it -v data:/backup/my-app-backup -v /path/to/local_backups:/archive:ro alpine tar -xvzf /archive/full_backup_filename.tar.gz
+  ```
+- Restart the container(s) that are using the volume.
+
 ### Set the timezone the container runs in

 By default a container based on this image will run in the UTC timezone.
@@ -378,7 +621,7 @@ version: '3'

 services:
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    volumes:
      - data:/backup/my-app-backup:ro
      - /etc/timezone:/etc/timezone:ro
@@ -401,7 +644,7 @@ When running in Swarm mode, it's also advised to set a hard memory limit on your
 ```yml
 services:
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    deployment:
      resources:
        limits:
@@ -416,6 +659,126 @@ You can manually trigger a backup run outside of the defined cron schedule by ex
 docker exec <container_ref> backup
 ```

+### Update deprecated email configuration
+
+Starting with version 2.6.0, configuring email notifications using `EMAIL_*` keys has been deprecated.
+Instead of providing multiple values using multiple keys, you can now provide a single URL for `NOTIFICATION_URLS`.
+
+Before:
+```ini
+EMAIL_NOTIFICATION_RECIPIENT="you@example.com"
+EMAIL_NOTIFICATION_SENDER="no-reply@example.com"
+EMAIL_SMTP_HOST="posteo.de"
+EMAIL_SMTP_PASSWORD="secret"
+EMAIL_SMTP_USERNAME="me"
+EMAIL_SMTP_PORT="587"
+```
+
+After:
+```ini
+NOTIFICATION_URLS=smtp://me:secret@posteo.de:587/?fromAddress=no-reply@example.com&toAddresses=you@example.com
+```
+
+### Replace deprecated `BACKUP_FROM_SNAPSHOT` usage
+
+Starting with version 2.15.0, the `BACKUP_FROM_SNAPSHOT` feature has been deprecated.
+If you need to prepare your sources before the backup is taken, use `exec-pre`, `exec-post` and an intermediate volume:
+
+```yml
+version: '3'
+
+services:
+  my_app:
+    build: .
+    volumes:
+      - data:/var/my_app
+      - backup:/tmp/backup
+    labels:
+      - docker-volume-backup.exec-pre=cp -r /var/my_app /tmp/backup/my-app
+      - docker-volume-backup.exec-post=rm -rf /tmp/backup/my-app
+
+  backup:
+    image: offen/docker-volume-backup:latest
+    environment:
+      BACKUP_SOURCES: /tmp/backup
+    volumes:
+      - backup:/backup:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+
+volumes:
+  data:
+  backup:
+```
+
+### Using a custom Docker host
+
+If you are interfacing with Docker via TCP, set `DOCKER_HOST` to the correct URL.
+```ini
+DOCKER_HOST=tcp://docker_socket_proxy:2375
+```
+
+In case you are using a socket proxy, it must support `GET` and `POST` requests to the `/containers` endpoint. If you are using Docker Swarm, it must also support the `/services` endpoint. If you are using pre/post backup commands, it must also support the `/exec` endpoint.
+
+### Run multiple backup schedules in the same container
+
+Multiple backup schedules with different configuration can be configured by mounting an arbitrary number of configuration files (using the `.env` format) into `/etc/dockervolumebackup/conf.d`:
+
+```yml
+version: '3'
+
+services:
+  # ... define other services using the `data` volume here
+  backup:
+    image: offen/docker-volume-backup:v2
+    volumes:
+      - data:/backup/my-app-backup:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - ./configuration:/etc/dockervolumebackup/conf.d
+
+volumes:
+  data:
+```
+
+A separate cronjob will be created for each config file.
+If a configuration value is set both in the global environment as well as in the config file, the config file will take precedence.
+The `backup` command expects to run on an exclusive lock, so in case you provide the same or overlapping schedules in your cron expressions, the runs will still be executed serially, one after the other.
+The exact order of schedules that use the same cron expression is not specified.
+In case you need your schedules to overlap, you need to create a dedicated container for each schedule instead.
+When changing the configuration, you currently need to manually restart the container for the changes to take effect.
+
+### Define different retention schedules
+
+If you want to manage backup retention on different schedules, the most straight forward approach is to define a dedicated configuration for retention rule using a different prefix in the `BACKUP_FILENAME` parameter and then run them on different cron schedules.
+
+For example, if you wanted to keep daily backups for 7 days, weekly backups for a month, and retain monthly backups forever, you could create three configuration files and mount them into `/etc/dockervolumebackup.d`:
+
+```ini
+# 01daily.conf
+BACKUP_FILENAME="daily-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
+# run every day at 2am
+BACKUP_CRON_EXPRESSION="0 2 * * *"
+BACKUP_PRUNING_PREFIX="daily-backup-"
+BACKUP_RETENTION_DAYS="7"
+```
+
+```ini
+# 02weekly.conf
+BACKUP_FILENAME="weekly-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
+# run every monday at 3am
+BACKUP_CRON_EXPRESSION="0 3 * * 1"
+BACKUP_PRUNING_PREFIX="weekly-backup-"
+BACKUP_RETENTION_DAYS="31"
+```
+
+```ini
+# 03monthly.conf
+BACKUP_FILENAME="monthly-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
+# run every 1st of a month at 4am
+BACKUP_CRON_EXPRESSION="0 4 1 * *"
+```
+
+Note that while it's possible to define colliding cron schedules for each of these configurations, you might need to adjust the value for `LOCK_TIMEOUT` in case your backups are large and might take longer than an hour.
+
 ## Recipes

 This section lists configuration for some real-world use cases that you can mix and match according to your needs.
@@ -428,9 +791,9 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    volumes:
@@ -441,6 +804,28 @@ volumes:
  data:
 ```

+### Backing up to Filebase
+
+```yml
+version: '3'
+
+services:
+  # ... define other services using the `data` volume here
+  backup:
+    image: offen/docker-volume-backup:v2
+    environment:
+      AWS_ENDPOINT: s3.filebase.com
+      AWS_S3_BUCKET_NAME: filebase-bucket
+      AWS_ACCESS_KEY_ID: FILEBASE-ACCESS-KEY
+      AWS_SECRET_ACCESS_KEY: FILEBASE-SECRET-KEY
+    volumes:
+      - data:/backup/my-app-backup:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+
+volumes:
+  data:
+```
+
 ### Backing up to MinIO

 ```yml
@@ -449,10 +834,10 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      AWS_ENDPOINT: minio.example.com
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: MINIOACCESSKEY
      AWS_SECRET_ACCESS_KEY: MINIOSECRETKEY
    volumes:
@@ -463,6 +848,28 @@ volumes:
  data:
 ```

+### Backing up to WebDAV
+
+```yml
+version: '3'
+
+services:
+  # ... define other services using the `data` volume here
+  backup:
+    image: offen/docker-volume-backup:v2
+    environment:
+      WEBDAV_URL: https://webdav.mydomain.me
+      WEBDAV_PATH: /my/directory/
+      WEBDAV_USERNAME: user
+      WEBDAV_PASSWORD: password
+    volumes:
+      - data:/backup/my-app-backup:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+
+volumes:
+  data:
+```
+
 ### Backing up locally

 ```yml
@@ -471,7 +878,7 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
      BACKUP_LATEST_SYMLINK: backup-latest.tar.gz
@@ -492,9 +899,9 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    volumes:
@@ -514,11 +921,11 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
      # take a backup on every hour
      BACKUP_CRON_EXPRESSION: "0 * * * *"
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    volumes:
@@ -537,9 +944,9 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
@@ -561,9 +968,9 @@ version: '3'
 services:
  # ... define other services using the `data` volume here
  backup:
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment:
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      GPG_PASSPHRASE: somesecretstring
@@ -575,6 +982,32 @@ volumes:
  data:
 ```

+### Using mysqldump to prepare the backup
+
+```yml
+version: '3'
+
+services:
+  database:
+    image: mariadb:latest
+    labels:
+      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -psecret --all-databases > /tmp/dumps/dump.sql'
+    volumes:
+      - app_data:/tmp/dumps
+  backup:
+    image: offen/docker-volume-backup:v2
+    environment:
+      BACKUP_FILENAME: db.tar.gz
+      BACKUP_CRON_EXPRESSION: "0 2 * * *"
+    volumes:
+      - ./local:/archive
+      - data:/backup/data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+volumes:
+  data:
+```
+
 ### Running multiple instances in the same setup

 ```yml
@@ -583,10 +1016,10 @@ version: '3'
 services:
  # ... define other services using the `data_1` and `data_2` volumes here
  backup_1: &backup_service
-    image: offen/docker-volume-backup:latest
+    image: offen/docker-volume-backup:v2
    environment: &backup_environment
      BACKUP_CRON_EXPRESSION: "0 2 * * *"
-      AWS_BUCKET_NAME: backup-bucket
+      AWS_S3_BUCKET_NAME: backup-bucket
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      # Label the container using the `data_1` volume as `docker-volume-backup.stop-during-backup=service1`
@@ -616,12 +1049,12 @@ This image is heavily inspired by `futurice/docker-volume-backup`. We decided to

 - The original image is based on `ubuntu` and requires additional tools, making it heavy.
 This version is roughly 1/25 in compressed size (it's ~12MB).
- The original image uses a shell script, when this version is written in Go, which makes it easier to extend and maintain (more verbose too).
+- The original image uses a shell script, when this version is written in Go.
 - The original image proposed to handle backup rotation through AWS S3 lifecycle policies.
 This image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO.
 Local copies of backups can also be pruned once they reach a certain age.
 - InfluxDB specific functionality from the original image was removed.
 - `arm64` and `arm/v7` architectures are supported.
 - Docker in Swarm mode is supported.
- Notifications on failed backups are supported
- IAM authentication through instance profiles is supported
+- Notifications on finished backups are supported.
+- IAM authentication through instance profiles is supported.
--- a/cmd/backup/archive.go
+++ b/cmd/backup/archive.go
@@ -0,0 +1,133 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+// Portions of this file are taken from package `targz`, Copyright (c) 2014 Fredrik Wallgren
+// Licensed under the MIT License: https://github.com/walle/targz/blob/57fe4206da5abf7dd3901b4af3891ec2f08c7b08/LICENSE
+
+package main
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"fmt"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+)
+
+func createArchive(files []string, inputFilePath, outputFilePath string) error {
+	inputFilePath = stripTrailingSlashes(inputFilePath)
+	inputFilePath, outputFilePath, err := makeAbsolute(inputFilePath, outputFilePath)
+	if err != nil {
+		return fmt.Errorf("createArchive: error transposing given file paths: %w", err)
+	}
+	if err := os.MkdirAll(filepath.Dir(outputFilePath), 0755); err != nil {
+		return fmt.Errorf("createArchive: error creating output file path: %w", err)
+	}
+
+	if err := compress(files, outputFilePath, filepath.Dir(inputFilePath)); err != nil {
+		return fmt.Errorf("createArchive: error creating archive: %w", err)
+	}
+
+	return nil
+}
+
+func stripTrailingSlashes(path string) string {
+	if len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[0 : len(path)-1]
+	}
+
+	return path
+}
+
+func makeAbsolute(inputFilePath, outputFilePath string) (string, string, error) {
+	inputFilePath, err := filepath.Abs(inputFilePath)
+	if err == nil {
+		outputFilePath, err = filepath.Abs(outputFilePath)
+	}
+
+	return inputFilePath, outputFilePath, err
+}
+
+func compress(paths []string, outFilePath, subPath string) error {
+	file, err := os.Create(outFilePath)
+	if err != nil {
+		return fmt.Errorf("compress: error creating out file: %w", err)
+	}
+
+	prefix := path.Dir(outFilePath)
+	gzipWriter := gzip.NewWriter(file)
+	tarWriter := tar.NewWriter(gzipWriter)
+
+	for _, p := range paths {
+		if err := writeTarGz(p, tarWriter, prefix); err != nil {
+			return fmt.Errorf("compress error writing %s to archive: %w", p, err)
+		}
+	}
+
+	err = tarWriter.Close()
+	if err != nil {
+		return fmt.Errorf("compress: error closing tar writer: %w", err)
+	}
+
+	err = gzipWriter.Close()
+	if err != nil {
+		return fmt.Errorf("compress: error closing gzip writer: %w", err)
+	}
+
+	err = file.Close()
+	if err != nil {
+		return fmt.Errorf("compress: error closing file: %w", err)
+	}
+
+	return nil
+}
+
+func writeTarGz(path string, tarWriter *tar.Writer, prefix string) error {
+	fileInfo, err := os.Lstat(path)
+	if err != nil {
+		return fmt.Errorf("writeTarGz: error getting file infor for %s: %w", path, err)
+	}
+
+	if fileInfo.Mode()&os.ModeSocket == os.ModeSocket {
+		return nil
+	}
+
+	var link string
+	if fileInfo.Mode()&os.ModeSymlink == os.ModeSymlink {
+		var err error
+		if link, err = os.Readlink(path); err != nil {
+			return fmt.Errorf("writeTarGz: error resolving symlink %s: %w", path, err)
+		}
+	}
+
+	header, err := tar.FileInfoHeader(fileInfo, link)
+	if err != nil {
+		return fmt.Errorf("writeTarGz: error getting file info header: %w", err)
+	}
+	header.Name = strings.TrimPrefix(path, prefix)
+
+	err = tarWriter.WriteHeader(header)
+	if err != nil {
+		return fmt.Errorf("writeTarGz: error writing file info header: %w", err)
+	}
+
+	if !fileInfo.Mode().IsRegular() {
+		return nil
+	}
+
+	file, err := os.Open(path)
+	if err != nil {
+		return fmt.Errorf("writeTarGz: error opening %s: %w", path, err)
+	}
+	defer file.Close()
+
+	_, err = io.Copy(tarWriter, file)
+	if err != nil {
+		return fmt.Errorf("writeTarGz: error copying %s to tar writer: %w", path, err)
+	}
+
+	return nil
+}
--- a/cmd/backup/config.go
+++ b/cmd/backup/config.go
@@ -0,0 +1,67 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"fmt"
+	"regexp"
+	"time"
+)
+
+// Config holds all configuration values that are expected to be set
+// by users.
+type Config struct {
+	BackupSources              string        `split_words:"true" default:"/backup"`
+	BackupFilename             string        `split_words:"true" default:"backup-%Y-%m-%dT%H-%M-%S.tar.gz"`
+	BackupFilenameExpand       bool          `split_words:"true"`
+	BackupLatestSymlink        string        `split_words:"true"`
+	BackupArchive              string        `split_words:"true" default:"/archive"`
+	BackupRetentionDays        int32         `split_words:"true" default:"-1"`
+	BackupPruningLeeway        time.Duration `split_words:"true" default:"1m"`
+	BackupPruningPrefix        string        `split_words:"true"`
+	BackupStopContainerLabel   string        `split_words:"true" default:"true"`
+	BackupFromSnapshot         bool          `split_words:"true"`
+	BackupExcludeRegexp        RegexpDecoder `split_words:"true"`
+	AwsS3BucketName            string        `split_words:"true"`
+	AwsS3Path                  string        `split_words:"true"`
+	AwsEndpoint                string        `split_words:"true" default:"s3.amazonaws.com"`
+	AwsEndpointProto           string        `split_words:"true" default:"https"`
+	AwsEndpointInsecure        bool          `split_words:"true"`
+	AwsAccessKeyID             string        `envconfig:"AWS_ACCESS_KEY_ID"`
+	AwsSecretAccessKey         string        `split_words:"true"`
+	AwsIamRoleEndpoint         string        `split_words:"true"`
+	GpgPassphrase              string        `split_words:"true"`
+	NotificationURLs           []string      `envconfig:"NOTIFICATION_URLS"`
+	NotificationLevel          string        `split_words:"true" default:"error"`
+	EmailNotificationRecipient string        `split_words:"true"`
+	EmailNotificationSender    string        `split_words:"true" default:"noreply@nohost"`
+	EmailSMTPHost              string        `envconfig:"EMAIL_SMTP_HOST"`
+	EmailSMTPPort              int           `envconfig:"EMAIL_SMTP_PORT" default:"587"`
+	EmailSMTPUsername          string        `envconfig:"EMAIL_SMTP_USERNAME"`
+	EmailSMTPPassword          string        `envconfig:"EMAIL_SMTP_PASSWORD"`
+	WebdavUrl                  string        `split_words:"true"`
+	WebdavUrlInsecure          bool          `split_words:"true"`
+	WebdavPath                 string        `split_words:"true" default:"/"`
+	WebdavUsername             string        `split_words:"true"`
+	WebdavPassword             string        `split_words:"true"`
+	ExecLabel                  string        `split_words:"true"`
+	ExecForwardOutput          bool          `split_words:"true"`
+	LockTimeout                time.Duration `split_words:"true" default:"60m"`
+}
+
+type RegexpDecoder struct {
+	Re *regexp.Regexp
+}
+
+func (r *RegexpDecoder) Decode(v string) error {
+	if v == "" {
+		return nil
+	}
+	re, err := regexp.Compile(v)
+	if err != nil {
+		return fmt.Errorf("config: error compiling given regexp `%s`: %w", v, err)
+	}
+	*r = RegexpDecoder{Re: re}
+	return nil
+}
--- a/cmd/backup/exec.go
+++ b/cmd/backup/exec.go
@@ -0,0 +1,123 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+// Portions of this file are taken and adapted from `moby`, Copyright 2012-2017 Docker, Inc.
+// Licensed under the Apache 2.0 License: https://github.com/moby/moby/blob/8e610b2b55bfd1bfa9436ab110d311f5e8a74dcb/LICENSE
+
+package main
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
+
+	"github.com/cosiner/argv"
+	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/api/types/filters"
+	"github.com/docker/docker/pkg/stdcopy"
+	"golang.org/x/sync/errgroup"
+)
+
+func (s *script) exec(containerRef string, command string) ([]byte, []byte, error) {
+	args, _ := argv.Argv(command, nil, nil)
+	execID, err := s.cli.ContainerExecCreate(context.Background(), containerRef, types.ExecConfig{
+		Cmd:          args[0],
+		AttachStdin:  true,
+		AttachStderr: true,
+	})
+	if err != nil {
+		return nil, nil, fmt.Errorf("exec: error creating container exec: %w", err)
+	}
+
+	resp, err := s.cli.ContainerExecAttach(context.Background(), execID.ID, types.ExecStartCheck{})
+	if err != nil {
+		return nil, nil, fmt.Errorf("exec: error attaching container exec: %w", err)
+	}
+	defer resp.Close()
+
+	var outBuf, errBuf bytes.Buffer
+	outputDone := make(chan error)
+
+	go func() {
+		_, err := stdcopy.StdCopy(&outBuf, &errBuf, resp.Reader)
+		outputDone <- err
+	}()
+
+	select {
+	case err := <-outputDone:
+		if err != nil {
+			return nil, nil, fmt.Errorf("exec: error demultiplexing output: %w", err)
+		}
+		break
+	}
+
+	stdout, err := ioutil.ReadAll(&outBuf)
+	if err != nil {
+		return nil, nil, fmt.Errorf("exec: error reading stdout: %w", err)
+	}
+	stderr, err := ioutil.ReadAll(&errBuf)
+	if err != nil {
+		return nil, nil, fmt.Errorf("exec: error reading stderr: %w", err)
+	}
+
+	res, err := s.cli.ContainerExecInspect(context.Background(), execID.ID)
+	if err != nil {
+		return nil, nil, fmt.Errorf("exec: error inspecting container exec: %w", err)
+	}
+
+	if res.ExitCode > 0 {
+		return stdout, stderr, fmt.Errorf("exec: running command exited %d", res.ExitCode)
+	}
+
+	return stdout, stderr, nil
+}
+
+func (s *script) runLabeledCommands(label string) error {
+	f := []filters.KeyValuePair{
+		{Key: "label", Value: label},
+	}
+	if s.c.ExecLabel != "" {
+		f = append(f, filters.KeyValuePair{
+			Key:   "label",
+			Value: fmt.Sprintf("docker-volume-backup.exec-label=%s", s.c.ExecLabel),
+		})
+	}
+	containersWithCommand, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
+		Quiet:   true,
+		Filters: filters.NewArgs(f...),
+	})
+	if err != nil {
+		return fmt.Errorf("runLabeledCommands: error querying for containers: %w", err)
+	}
+
+	if len(containersWithCommand) == 0 {
+		return nil
+	}
+
+	g := new(errgroup.Group)
+
+	for _, container := range containersWithCommand {
+		c := container
+		g.Go(func() error {
+			cmd, _ := c.Labels[label]
+			s.logger.Infof("Running %s command %s for container %s", label, cmd, strings.TrimPrefix(c.Names[0], "/"))
+			stdout, stderr, err := s.exec(c.ID, cmd)
+			if s.c.ExecForwardOutput {
+				os.Stderr.Write(stderr)
+				os.Stdout.Write(stdout)
+			}
+			if err != nil {
+				return fmt.Errorf("runLabeledCommands: error executing command: %w", err)
+			}
+			return nil
+		})
+	}
+
+	if err := g.Wait(); err != nil {
+		return fmt.Errorf("runLabeledCommands: error from errgroup: %w", err)
+	}
+	return nil
+}
--- a/cmd/backup/hooks.go
+++ b/cmd/backup/hooks.go
@@ -0,0 +1,56 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"fmt"
+	"sort"
+)
+
+// hook contains a queued action that can be trigger them when the script
+// reaches a certain point (e.g. unsuccessful backup)
+type hook struct {
+	level  hookLevel
+	action func(err error) error
+}
+
+type hookLevel int
+
+const (
+	hookLevelPlumbing hookLevel = iota
+	hookLevelError
+	hookLevelInfo
+)
+
+var hookLevels = map[string]hookLevel{
+	"info":  hookLevelInfo,
+	"error": hookLevelError,
+}
+
+// registerHook adds the given action at the given level.
+func (s *script) registerHook(level hookLevel, action func(err error) error) {
+	s.hooks = append(s.hooks, hook{level, action})
+}
+
+// runHooks runs all hooks that have been registered using the
+// given levels in the defined ordering. In case executing a hook returns an
+// error, the following hooks will still be run before the function returns.
+func (s *script) runHooks(err error) error {
+	sort.SliceStable(s.hooks, func(i, j int) bool {
+		return s.hooks[i].level < s.hooks[j].level
+	})
+	var actionErrors []error
+	for _, hook := range s.hooks {
+		if hook.level > s.hookLevel {
+			continue
+		}
+		if actionErr := hook.action(err); actionErr != nil {
+			actionErrors = append(actionErrors, fmt.Errorf("runHooks: error running hook: %w", actionErr))
+		}
+	}
+	if len(actionErrors) != 0 {
+		return join(actionErrors...)
+	}
+	return nil
+}
--- a/cmd/backup/lock.go
+++ b/cmd/backup/lock.go
@@ -0,0 +1,58 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"errors"
+	"fmt"
+	"time"
+
+	"github.com/gofrs/flock"
+)
+
+// lock opens a lockfile at the given location, keeping it locked until the
+// caller invokes the returned release func. In case the lock is currently blocked
+// by another execution, it will repeatedly retry until the lock is available
+// or the given timeout is exceeded.
+func (s *script) lock(lockfile string) (func() error, error) {
+	start := time.Now()
+	defer func() {
+		s.stats.LockedTime = time.Now().Sub(start)
+	}()
+
+	retry := time.NewTicker(5 * time.Second)
+	defer retry.Stop()
+	deadline := time.NewTimer(s.c.LockTimeout)
+	defer deadline.Stop()
+
+	fileLock := flock.New(lockfile)
+
+	for {
+		acquired, err := fileLock.TryLock()
+		if err != nil {
+			return noop, fmt.Errorf("lock: error trying lock: %w", err)
+		}
+		if acquired {
+			if s.encounteredLock {
+				s.logger.Info("Acquired exclusive lock on subsequent attempt, ready to continue.")
+			}
+			return fileLock.Unlock, nil
+		}
+
+		if !s.encounteredLock {
+			s.logger.Infof(
+				"Exclusive lock was not available on first attempt. Will retry until it becomes available or the timeout of %s is exceeded.",
+				s.c.LockTimeout,
+			)
+			s.encounteredLock = true
+		}
+
+		select {
+		case <-retry.C:
+			continue
+		case <-deadline.C:
+			return noop, errors.New("lock: timed out waiting for lockfile to become available")
+		}
+	}
+}
--- a/cmd/backup/main.go
+++ b/cmd/backup/main.go
@@ -1,49 +1,26 @@
-// Copyright 2021 - Offen Authors <hioffen@posteo.de>
+// Copyright 2021-2022 - Offen Authors <hioffen@posteo.de>
 // SPDX-License-Identifier: MPL-2.0

 package main

 import (
-	"bytes"
-	"context"
-	"errors"
-	"fmt"
-	"io"
 	"os"
-	"path"
-	"path/filepath"
-	"strings"
-	"time"
-
-	"github.com/docker/docker/api/types"
-	"github.com/docker/docker/api/types/filters"
-	"github.com/docker/docker/api/types/swarm"
-	"github.com/docker/docker/client"
-	"github.com/go-gomail/gomail"
-	"github.com/gofrs/flock"
-	"github.com/kelseyhightower/envconfig"
-	"github.com/leekchan/timeutil"
-	"github.com/m90/targz"
-	"github.com/minio/minio-go/v7"
-	"github.com/minio/minio-go/v7/pkg/credentials"
-	"github.com/otiai10/copy"
-	"github.com/sirupsen/logrus"
-	"golang.org/x/crypto/openpgp"
 )

 func main() {
-	unlock := lock("/var/lock/dockervolumebackup.lock")
-	defer unlock()
-
 	s, err := newScript()
 	if err != nil {
 		panic(err)
 	}

+	unlock, err := s.lock("/var/lock/dockervolumebackup.lock")
+	defer unlock()
+	s.must(err)
+
 	defer func() {
 		if pArg := recover(); pArg != nil {
 			if err, ok := pArg.(error); ok {
-				if hookErr := s.runHooks(err, hookLevelCleanup, hookLevelFailure); hookErr != nil {
+				if hookErr := s.runHooks(err); hookErr != nil {
 					s.logger.Errorf("An error occurred calling the registered hooks: %s", hookErr)
 				}
 				os.Exit(1)
@@ -51,7 +28,7 @@ func main() {
 			panic(pArg)
 		}

-		if err := s.runHooks(nil, hookLevelCleanup); err != nil {
+		if err := s.runHooks(nil); err != nil {
 			s.logger.Errorf(
 				"Backup procedure ran successfully, but an error ocurred calling the registered hooks: %v",
 				err,
@@ -62,6 +39,13 @@ func main() {
 	}()

 	s.must(func() error {
+		runPostCommands, err := s.runCommands()
+		defer func() {
+			s.must(runPostCommands())
+		}()
+		if err != nil {
+			return err
+		}
 		restartContainers, err := s.stopContainers()
 		// The mechanism for restarting containers is not using hooks as it
 		// should happen as soon as possible (i.e. before uploading backups or
@@ -77,673 +61,5 @@ func main() {

 	s.must(s.encryptBackup())
 	s.must(s.copyBackup())
-	s.must(s.pruneOldBackups())
+	s.must(s.pruneBackups())
 }
-
-// script holds all the stateful information required to orchestrate a
-// single backup run.
-type script struct {
-	cli    *client.Client
-	mc     *minio.Client
-	logger *logrus.Logger
-	hooks  []hook
-
-	start  time.Time
-	file   string
-	output *bytes.Buffer
-
-	c *config
-}
-
-type config struct {
-	BackupSources              string        `split_words:"true" default:"/backup"`
-	BackupFilename             string        `split_words:"true" default:"backup-%Y-%m-%dT%H-%M-%S.tar.gz"`
-	BackupLatestSymlink        string        `split_words:"true"`
-	BackupArchive              string        `split_words:"true" default:"/archive"`
-	BackupRetentionDays        int32         `split_words:"true" default:"-1"`
-	BackupPruningLeeway        time.Duration `split_words:"true" default:"1m"`
-	BackupPruningPrefix        string        `split_words:"true"`
-	BackupStopContainerLabel   string        `split_words:"true" default:"true"`
-	BackupFromSnapshot         bool          `split_words:"true"`
-	AwsS3BucketName            string        `split_words:"true"`
-	AwsEndpoint                string        `split_words:"true" default:"s3.amazonaws.com"`
-	AwsEndpointProto           string        `split_words:"true" default:"https"`
-	AwsEndpointInsecure        bool          `split_words:"true"`
-	AwsAccessKeyID             string        `envconfig:"AWS_ACCESS_KEY_ID"`
-	AwsSecretAccessKey         string        `split_words:"true"`
-	AwsIamRoleEndpoint         string        `split_words:"true"`
-	GpgPassphrase              string        `split_words:"true"`
-	EmailNotificationRecipient string        `split_words:"true"`
-	EmailNotificationSender    string        `split_words:"true" default:"noreply@nohost"`
-	EmailSMTPHost              string        `envconfig:"EMAIL_SMTP_HOST"`
-	EmailSMTPPort              int           `envconfig:"EMAIL_SMTP_PORT" default:"587"`
-	EmailSMTPUsername          string        `envconfig:"EMAIL_SMTP_USERNAME"`
-	EmailSMTPPassword          string        `envconfig:"EMAIL_SMTP_PASSWORD"`
-}
-
-var msgBackupFailed = "backup run failed"
-
-// newScript creates all resources needed for the script to perform actions against
-// remote resources like the Docker engine or remote storage locations. All
-// reading from env vars or other configuration sources is expected to happen
-// in this method.
-func newScript() (*script, error) {
-	stdOut, logBuffer := buffer(os.Stdout)
-	s := &script{
-		c: &config{},
-		logger: &logrus.Logger{
-			Out:       stdOut,
-			Formatter: new(logrus.TextFormatter),
-			Hooks:     make(logrus.LevelHooks),
-			Level:     logrus.InfoLevel,
-		},
-		start:  time.Now(),
-		output: logBuffer,
-	}
-
-	if err := envconfig.Process("", s.c); err != nil {
-		return nil, fmt.Errorf("newScript: failed to process configuration values: %w", err)
-	}
-
-	s.file = path.Join("/tmp", s.c.BackupFilename)
-
-	_, err := os.Stat("/var/run/docker.sock")
-	if !os.IsNotExist(err) {
-		cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
-		if err != nil {
-			return nil, fmt.Errorf("newScript: failed to create docker client")
-		}
-		s.cli = cli
-	}
-
-	if s.c.AwsS3BucketName != "" {
-		var creds *credentials.Credentials
-		if s.c.AwsAccessKeyID != "" && s.c.AwsSecretAccessKey != "" {
-			creds = credentials.NewStaticV4(
-				s.c.AwsAccessKeyID,
-				s.c.AwsSecretAccessKey,
-				"",
-			)
-		} else if s.c.AwsIamRoleEndpoint != "" {
-			creds = credentials.NewIAM(s.c.AwsIamRoleEndpoint)
-		} else {
-			return nil, errors.New("newScript: AWS_S3_BUCKET_NAME is defined, but no credentials were provided")
-		}
-
-		options := minio.Options{
-			Creds:  creds,
-			Secure: s.c.AwsEndpointProto == "https",
-		}
-
-		if s.c.AwsEndpointInsecure {
-			if !options.Secure {
-				return nil, errors.New("newScript: AWS_ENDPOINT_INSECURE = true is only meaningful for https")
-			}
-
-			transport, err := minio.DefaultTransport(true)
-			if err != nil {
-				return nil, fmt.Errorf("newScript: failed to create default minio transport")
-			}
-			transport.TLSClientConfig.InsecureSkipVerify = true
-			options.Transport = transport
-		}
-
-		mc, err := minio.New(s.c.AwsEndpoint, &options)
-		if err != nil {
-			return nil, fmt.Errorf("newScript: error setting up minio client: %w", err)
-		}
-		s.mc = mc
-	}
-
-	if s.c.EmailNotificationRecipient != "" {
-		s.hooks = append(s.hooks, hook{hookLevelFailure, func(err error, start time.Time, logOutput string) error {
-			mailer := gomail.NewDialer(
-				s.c.EmailSMTPHost, s.c.EmailSMTPPort, s.c.EmailSMTPUsername, s.c.EmailSMTPPassword,
-			)
-
-			subject := fmt.Sprintf(
-				"Failure running docker-volume-backup at %s", start.Format(time.RFC3339),
-			)
-			body := fmt.Sprintf(
-				"Running docker-volume-backup failed with error: %s\n\nLog output of the failed run was:\n\n%s\n", err, logOutput,
-			)
-
-			message := gomail.NewMessage()
-			message.SetHeader("From", s.c.EmailNotificationSender)
-			message.SetHeader("To", s.c.EmailNotificationRecipient)
-			message.SetHeader("Subject", subject)
-			message.SetBody("text/plain", body)
-			return mailer.DialAndSend(message)
-		}})
-	}
-
-	return s, nil
-}
-
-var noop = func() error { return nil }
-
-// registerHook adds the given action at the given level.
-func (s *script) registerHook(level hookLevel, action func(err error, start time.Time, logOutput string) error) {
-	s.hooks = append(s.hooks, hook{level, action})
-}
-
-// stopContainers stops all Docker containers that are marked as to being
-// stopped during the backup and returns a function that can be called to
-// restart everything that has been stopped.
-func (s *script) stopContainers() (func() error, error) {
-	if s.cli == nil {
-		return noop, nil
-	}
-
-	allContainers, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
-		Quiet: true,
-	})
-	if err != nil {
-		return noop, fmt.Errorf("stopContainersAndRun: error querying for containers: %w", err)
-	}
-
-	containerLabel := fmt.Sprintf(
-		"docker-volume-backup.stop-during-backup=%s",
-		s.c.BackupStopContainerLabel,
-	)
-	containersToStop, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
-		Quiet: true,
-		Filters: filters.NewArgs(filters.KeyValuePair{
-			Key:   "label",
-			Value: containerLabel,
-		}),
-	})
-
-	if err != nil {
-		return noop, fmt.Errorf("stopContainersAndRun: error querying for containers to stop: %w", err)
-	}
-
-	if len(containersToStop) == 0 {
-		return noop, nil
-	}
-
-	s.logger.Infof(
-		"Stopping %d container(s) labeled `%s` out of %d running container(s).",
-		len(containersToStop),
-		containerLabel,
-		len(allContainers),
-	)
-
-	var stoppedContainers []types.Container
-	var stopErrors []error
-	for _, container := range containersToStop {
-		if err := s.cli.ContainerStop(context.Background(), container.ID, nil); err != nil {
-			stopErrors = append(stopErrors, err)
-		} else {
-			stoppedContainers = append(stoppedContainers, container)
-		}
-	}
-
-	var stopError error
-	if len(stopErrors) != 0 {
-		stopError = fmt.Errorf(
-			"stopContainersAndRun: %d error(s) stopping containers: %w",
-			len(stopErrors),
-			join(stopErrors...),
-		)
-	}
-
-	return func() error {
-		servicesRequiringUpdate := map[string]struct{}{}
-
-		var restartErrors []error
-		for _, container := range stoppedContainers {
-			if swarmServiceName, ok := container.Labels["com.docker.swarm.service.name"]; ok {
-				servicesRequiringUpdate[swarmServiceName] = struct{}{}
-				continue
-			}
-			if err := s.cli.ContainerStart(context.Background(), container.ID, types.ContainerStartOptions{}); err != nil {
-				restartErrors = append(restartErrors, err)
-			}
-		}
-
-		if len(servicesRequiringUpdate) != 0 {
-			services, _ := s.cli.ServiceList(context.Background(), types.ServiceListOptions{})
-			for serviceName := range servicesRequiringUpdate {
-				var serviceMatch swarm.Service
-				for _, service := range services {
-					if service.Spec.Name == serviceName {
-						serviceMatch = service
-						break
-					}
-				}
-				if serviceMatch.ID == "" {
-					return fmt.Errorf("stopContainersAndRun: couldn't find service with name %s", serviceName)
-				}
-				serviceMatch.Spec.TaskTemplate.ForceUpdate = 1
-				if _, err := s.cli.ServiceUpdate(
-					context.Background(), serviceMatch.ID,
-					serviceMatch.Version, serviceMatch.Spec, types.ServiceUpdateOptions{},
-				); err != nil {
-					restartErrors = append(restartErrors, err)
-				}
-			}
-		}
-
-		if len(restartErrors) != 0 {
-			return fmt.Errorf(
-				"stopContainersAndRun: %d error(s) restarting containers and services: %w",
-				len(restartErrors),
-				join(restartErrors...),
-			)
-		}
-		s.logger.Infof(
-			"Restarted %d container(s) and the matching service(s).",
-			len(stoppedContainers),
-		)
-		return nil
-	}, stopError
-}
-
-// takeBackup creates a tar archive of the configured backup location and
-// saves it to disk.
-func (s *script) takeBackup() error {
-	s.file = timeutil.Strftime(&s.start, s.file)
-	backupSources := s.c.BackupSources
-
-	if s.c.BackupFromSnapshot {
-		backupSources = filepath.Join("/tmp", s.c.BackupSources)
-		// copy before compressing guard against a situation where backup folder's content are still growing.
-		s.registerHook(hookLevelCleanup, func(error, time.Time, string) error {
-			if err := remove(backupSources); err != nil {
-				return fmt.Errorf("takeBackup: error removing snapshot: %w", err)
-			}
-			s.logger.Infof("Removed snapshot `%s`.", backupSources)
-			return nil
-		})
-		if err := copy.Copy(s.c.BackupSources, backupSources, copy.Options{
-			PreserveTimes: true,
-			PreserveOwner: true,
-		}); err != nil {
-			return fmt.Errorf("takeBackup: error creating snapshot: %w", err)
-		}
-		s.logger.Infof("Created snapshot of `%s` at `%s`.", s.c.BackupSources, backupSources)
-	}
-
-	tarFile := s.file
-	s.registerHook(hookLevelCleanup, func(error, time.Time, string) error {
-		if err := remove(tarFile); err != nil {
-			return fmt.Errorf("takeBackup: error removing tar file: %w", err)
-		}
-		s.logger.Infof("Removed tar file `%s`.", tarFile)
-		return nil
-	})
-	if err := targz.Compress(backupSources, tarFile); err != nil {
-		return fmt.Errorf("takeBackup: error compressing backup folder: %w", err)
-	}
-
-	s.logger.Infof("Created backup of `%s` at `%s`.", backupSources, tarFile)
-	return nil
-}
-
-// encryptBackup encrypts the backup file using PGP and the configured passphrase.
-// In case no passphrase is given it returns early, leaving the backup file
-// untouched.
-func (s *script) encryptBackup() error {
-	if s.c.GpgPassphrase == "" {
-		return nil
-	}
-
-	gpgFile := fmt.Sprintf("%s.gpg", s.file)
-	s.registerHook(hookLevelCleanup, func(error, time.Time, string) error {
-		if err := remove(gpgFile); err != nil {
-			return fmt.Errorf("encryptBackup: error removing gpg file: %w", err)
-		}
-		s.logger.Infof("Removed GPG file `%s`.", gpgFile)
-		return nil
-	})
-
-	outFile, err := os.Create(gpgFile)
-	defer outFile.Close()
-	if err != nil {
-		return fmt.Errorf("encryptBackup: error opening out file: %w", err)
-	}
-
-	_, name := path.Split(s.file)
-	dst, err := openpgp.SymmetricallyEncrypt(outFile, []byte(s.c.GpgPassphrase), &openpgp.FileHints{
-		IsBinary: true,
-		FileName: name,
-	}, nil)
-	defer dst.Close()
-	if err != nil {
-		return fmt.Errorf("encryptBackup: error encrypting backup file: %w", err)
-	}
-
-	src, err := os.Open(s.file)
-	if err != nil {
-		return fmt.Errorf("encryptBackup: error opening backup file `%s`: %w", s.file, err)
-	}
-
-	if _, err := io.Copy(dst, src); err != nil {
-		return fmt.Errorf("encryptBackup: error writing ciphertext to file: %w", err)
-	}
-
-	s.file = gpgFile
-	s.logger.Infof("Encrypted backup using given passphrase, saving as `%s`.", s.file)
-	return nil
-}
-
-// copyBackup makes sure the backup file is copied to both local and remote locations
-// as per the given configuration.
-func (s *script) copyBackup() error {
-	_, name := path.Split(s.file)
-	if s.mc != nil {
-		if _, err := s.mc.FPutObject(context.Background(), s.c.AwsS3BucketName, name, s.file, minio.PutObjectOptions{
-			ContentType: "application/tar+gzip",
-		}); err != nil {
-			return fmt.Errorf("copyBackup: error uploading backup to remote storage: %w", err)
-		}
-		s.logger.Infof("Uploaded a copy of backup `%s` to bucket `%s`.", s.file, s.c.AwsS3BucketName)
-	}
-
-	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
-		if err := copyFile(s.file, path.Join(s.c.BackupArchive, name)); err != nil {
-			return fmt.Errorf("copyBackup: error copying file to local archive: %w", err)
-		}
-		s.logger.Infof("Stored copy of backup `%s` in local archive `%s`.", s.file, s.c.BackupArchive)
-		if s.c.BackupLatestSymlink != "" {
-			symlink := path.Join(s.c.BackupArchive, s.c.BackupLatestSymlink)
-			if _, err := os.Lstat(symlink); err == nil {
-				os.Remove(symlink)
-			}
-			if err := os.Symlink(name, symlink); err != nil {
-				return fmt.Errorf("copyBackup: error creating latest symlink: %w", err)
-			}
-			s.logger.Infof("Created/Updated symlink `%s` for latest backup.", s.c.BackupLatestSymlink)
-		}
-	}
-	return nil
-}
-
-// pruneOldBackups rotates away backups from local and remote storages using
-// the given configuration. In case the given configuration would delete all
-// backups, it does nothing instead.
-func (s *script) pruneOldBackups() error {
-	if s.c.BackupRetentionDays < 0 {
-		return nil
-	}
-
-	if s.c.BackupPruningLeeway != 0 {
-		s.logger.Infof("Sleeping for %s before pruning backups.", s.c.BackupPruningLeeway)
-		time.Sleep(s.c.BackupPruningLeeway)
-	}
-
-	deadline := time.Now().AddDate(0, 0, -int(s.c.BackupRetentionDays))
-
-	if s.mc != nil {
-		candidates := s.mc.ListObjects(context.Background(), s.c.AwsS3BucketName, minio.ListObjectsOptions{
-			WithMetadata: true,
-			Prefix:       s.c.BackupPruningPrefix,
-		})
-
-		var matches []minio.ObjectInfo
-		var lenCandidates int
-		for candidate := range candidates {
-			lenCandidates++
-			if candidate.Err != nil {
-				return fmt.Errorf(
-					"pruneOldBackups: error looking up candidates from remote storage: %w",
-					candidate.Err,
-				)
-			}
-			if candidate.LastModified.Before(deadline) {
-				matches = append(matches, candidate)
-			}
-		}
-
-		if len(matches) != 0 && len(matches) != lenCandidates {
-			objectsCh := make(chan minio.ObjectInfo)
-			go func() {
-				for _, match := range matches {
-					objectsCh <- match
-				}
-				close(objectsCh)
-			}()
-			errChan := s.mc.RemoveObjects(context.Background(), s.c.AwsS3BucketName, objectsCh, minio.RemoveObjectsOptions{})
-			var removeErrors []error
-			for result := range errChan {
-				if result.Err != nil {
-					removeErrors = append(removeErrors, result.Err)
-				}
-			}
-
-			if len(removeErrors) != 0 {
-				return fmt.Errorf(
-					"pruneOldBackups: %d error(s) removing files from remote storage: %w",
-					len(removeErrors),
-					join(removeErrors...),
-				)
-			}
-			s.logger.Infof(
-				"Pruned %d out of %d remote backup(s) as their age exceeded the configured retention period of %d days.",
-				len(matches),
-				lenCandidates,
-				s.c.BackupRetentionDays,
-			)
-		} else if len(matches) != 0 && len(matches) == lenCandidates {
-			s.logger.Warnf(
-				"The current configuration would delete all %d remote backup copies.",
-				len(matches),
-			)
-			s.logger.Warn("Refusing to do so, please check your configuration.")
-		} else {
-			s.logger.Infof("None of %d remote backup(s) were pruned.", lenCandidates)
-		}
-	}
-
-	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
-		globPattern := path.Join(
-			s.c.BackupArchive,
-			fmt.Sprintf("%s*", s.c.BackupPruningPrefix),
-		)
-		globMatches, err := filepath.Glob(globPattern)
-		if err != nil {
-			return fmt.Errorf(
-				"pruneOldBackups: error looking up matching files using pattern %s: %w",
-				globPattern,
-				err,
-			)
-		}
-
-		var candidates []string
-		for _, candidate := range globMatches {
-			fi, err := os.Lstat(candidate)
-			if err != nil {
-				return fmt.Errorf(
-					"pruneOldBackups: error calling Lstat on file %s: %w",
-					candidate,
-					err,
-				)
-			}
-
-			if fi.Mode()&os.ModeSymlink != os.ModeSymlink {
-				candidates = append(candidates, candidate)
-			}
-		}
-
-		var matches []string
-		for _, candidate := range candidates {
-			fi, err := os.Stat(candidate)
-			if err != nil {
-				return fmt.Errorf(
-					"pruneOldBackups: error calling stat on file %s: %w",
-					candidate,
-					err,
-				)
-			}
-			if fi.ModTime().Before(deadline) {
-				matches = append(matches, candidate)
-			}
-		}
-
-		if len(matches) != 0 && len(matches) != len(candidates) {
-			var removeErrors []error
-			for _, match := range matches {
-				if err := os.Remove(match); err != nil {
-					removeErrors = append(removeErrors, err)
-				}
-			}
-			if len(removeErrors) != 0 {
-				return fmt.Errorf(
-					"pruneOldBackups: %d error(s) deleting local files, starting with: %w",
-					len(removeErrors),
-					join(removeErrors...),
-				)
-			}
-			s.logger.Infof(
-				"Pruned %d out of %d local backup(s) as their age exceeded the configured retention period of %d days.",
-				len(matches),
-				len(candidates),
-				s.c.BackupRetentionDays,
-			)
-		} else if len(matches) != 0 && len(matches) == len(candidates) {
-			s.logger.Warnf(
-				"The current configuration would delete all %d local backup copies.",
-				len(matches),
-			)
-			s.logger.Warn("Refusing to do so, please check your configuration.")
-		} else {
-			s.logger.Infof("None of %d local backup(s) were pruned.", len(candidates))
-		}
-	}
-	return nil
-}
-
-// runHooks runs all hooks that have been registered using the
-// given levels in the defined ordering. In case executing a hook returns an
-// error, the following hooks will still be run before the function returns.
-func (s *script) runHooks(err error, levels ...hookLevel) error {
-	var actionErrors []error
-	for _, level := range levels {
-		for _, hook := range s.hooks {
-			if hook.level != level {
-				continue
-			}
-			if actionErr := hook.action(err, s.start, s.output.String()); actionErr != nil {
-				actionErrors = append(actionErrors, fmt.Errorf("runHooks: error running hook: %w", actionErr))
-			}
-		}
-	}
-	if len(actionErrors) != 0 {
-		return join(actionErrors...)
-	}
-	return nil
-}
-
-// must exits the script run prematurely in case the given error
-// is non-nil.
-func (s *script) must(err error) {
-	if err != nil {
-		s.logger.Errorf("Fatal error running backup: %s", err)
-		panic(err)
-	}
-}
-
-// remove removes the given file or directory from disk.
-func remove(location string) error {
-	fi, err := os.Lstat(location)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return nil
-		}
-		return fmt.Errorf("remove: error checking for existence of `%s`: %w", location, err)
-	}
-	if fi.IsDir() {
-		err = os.RemoveAll(location)
-	} else {
-		err = os.Remove(location)
-	}
-	if err != nil {
-		return fmt.Errorf("remove: error removing `%s`: %w", location, err)
-	}
-	return nil
-}
-
-// lock opens a lockfile at the given location, keeping it locked until the
-// caller invokes the returned release func. When invoked while the file is
-// still locked the function panics.
-func lock(lockfile string) func() error {
-	fileLock := flock.New(lockfile)
-	acquired, err := fileLock.TryLock()
-	if err != nil {
-		panic(err)
-	}
-	if !acquired {
-		panic("unable to acquire file lock")
-	}
-	return fileLock.Unlock
-}
-
-// copy creates a copy of the file located at `dst` at `src`.
-func copyFile(src, dst string) error {
-	in, err := os.Open(src)
-	if err != nil {
-		return err
-	}
-	defer in.Close()
-
-	out, err := os.Create(dst)
-	if err != nil {
-		return err
-	}
-
-	_, err = io.Copy(out, in)
-	if err != nil {
-		out.Close()
-		return err
-	}
-	return out.Close()
-}
-
-// join takes a list of errors and joins them into a single error
-func join(errs ...error) error {
-	if len(errs) == 1 {
-		return errs[0]
-	}
-	var msgs []string
-	for _, err := range errs {
-		if err == nil {
-			continue
-		}
-		msgs = append(msgs, err.Error())
-	}
-	return errors.New("[" + strings.Join(msgs, ", ") + "]")
-}
-
-// buffer takes an io.Writer and returns a wrapped version of the
-// writer that writes to both the original target as well as the returned buffer
-func buffer(w io.Writer) (io.Writer, *bytes.Buffer) {
-	buffering := &bufferingWriter{buf: bytes.Buffer{}, writer: w}
-	return buffering, &buffering.buf
-}
-
-type bufferingWriter struct {
-	buf    bytes.Buffer
-	writer io.Writer
-}
-
-func (b *bufferingWriter) Write(p []byte) (n int, err error) {
-	if n, err := b.buf.Write(p); err != nil {
-		return n, fmt.Errorf("bufferingWriter: error writing to buffer: %w", err)
-	}
-	return b.writer.Write(p)
-}
-
-// hook contains a queued action that can be trigger them when the script
-// reaches a certain point (e.g. unsuccessful backup)
-type hook struct {
-	level  hookLevel
-	action func(err error, start time.Time, logOutput string) error
-}
-
-type hookLevel int
-
-const (
-	hookLevelFailure hookLevel = iota
-	hookLevelCleanup
-)
--- a/cmd/backup/notifications.go
+++ b/cmd/backup/notifications.go
@@ -0,0 +1,105 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"bytes"
+	_ "embed"
+	"fmt"
+	"text/template"
+	"time"
+
+	sTypes "github.com/containrrr/shoutrrr/pkg/types"
+)
+
+//go:embed notifications.tmpl
+var defaultNotifications string
+
+// NotificationData data to be passed to the notification templates
+type NotificationData struct {
+	Error  error
+	Config *Config
+	Stats  *Stats
+}
+
+// notify sends a notification using the given title and body templates.
+// Automatically creates notification data, adding the given error
+func (s *script) notify(titleTemplate string, bodyTemplate string, err error) error {
+	params := NotificationData{
+		Error:  err,
+		Stats:  s.stats,
+		Config: s.c,
+	}
+
+	titleBuf := &bytes.Buffer{}
+	if err := s.template.ExecuteTemplate(titleBuf, titleTemplate, params); err != nil {
+		return fmt.Errorf("notifyFailure: error executing %s template: %w", titleTemplate, err)
+	}
+
+	bodyBuf := &bytes.Buffer{}
+	if err := s.template.ExecuteTemplate(bodyBuf, bodyTemplate, params); err != nil {
+		return fmt.Errorf("notifyFailure: error executing %s template: %w", bodyTemplate, err)
+	}
+
+	if err := s.sendNotification(titleBuf.String(), bodyBuf.String()); err != nil {
+		return fmt.Errorf("notifyFailure: error notifying: %w", err)
+	}
+	return nil
+}
+
+// notifyFailure sends a notification about a failed backup run
+func (s *script) notifyFailure(err error) error {
+	return s.notify("title_failure", "body_failure", err)
+}
+
+// notifyFailure sends a notification about a successful backup run
+func (s *script) notifySuccess() error {
+	return s.notify("title_success", "body_success", nil)
+}
+
+// sendNotification sends a notification to all configured third party services
+func (s *script) sendNotification(title, body string) error {
+	var errs []error
+	for _, result := range s.sender.Send(body, &sTypes.Params{"title": title}) {
+		if result != nil {
+			errs = append(errs, result)
+		}
+	}
+	if len(errs) != 0 {
+		return fmt.Errorf("sendNotification: error sending message: %w", join(errs...))
+	}
+	return nil
+}
+
+var templateHelpers = template.FuncMap{
+	"formatTime": func(t time.Time) string {
+		return t.Format(time.RFC3339)
+	},
+	"formatBytesDec": func(bytes uint64) string {
+		return formatBytes(bytes, true)
+	},
+	"formatBytesBin": func(bytes uint64) string {
+		return formatBytes(bytes, false)
+	},
+}
+
+// formatBytes converts an amount of bytes in a human-readable representation
+// the decimal parameter specifies if using powers of 1000 (decimal) or powers of 1024 (binary)
+func formatBytes(b uint64, decimal bool) string {
+	unit := uint64(1024)
+	format := "%.1f %ciB"
+	if decimal {
+		unit = uint64(1000)
+		format = "%.1f %cB"
+	}
+	if b < unit {
+		return fmt.Sprintf("%d B", b)
+	}
+	div, exp := unit, 0
+	for n := b / unit; n >= unit; n /= unit {
+		div *= unit
+		exp++
+	}
+	return fmt.Sprintf(format, float64(b)/float64(div), "kMGTPE"[exp])
+}
--- a/cmd/backup/notifications.tmpl
+++ b/cmd/backup/notifications.tmpl
@@ -0,0 +1,26 @@
+{{ define "title_failure" -}}
+Failure running docker-volume-backup at {{ .Stats.StartTime | formatTime }}
+{{- end }}
+
+
+{{ define "body_failure" -}}
+Running docker-volume-backup failed with error: {{ .Error }}
+
+Log output of the failed run was:
+
+{{ .Stats.LogOutput }}
+{{- end }}
+
+
+{{ define "title_success" -}}
+Success running docker-volume-backup at {{ .Stats.StartTime | formatTime }}
+{{- end }}
+
+
+{{ define "body_success" -}}
+Running docker-volume-backup succeeded.
+
+Log output was:
+
+{{ .Stats.LogOutput }}
+{{- end }}
--- a/cmd/backup/script.go
+++ b/cmd/backup/script.go
@@ -0,0 +1,725 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"net/http"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+	"text/template"
+	"time"
+
+	"github.com/containrrr/shoutrrr"
+	"github.com/containrrr/shoutrrr/pkg/router"
+	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/api/types/filters"
+	"github.com/docker/docker/api/types/swarm"
+	"github.com/docker/docker/client"
+	"github.com/kelseyhightower/envconfig"
+	"github.com/leekchan/timeutil"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+	"github.com/otiai10/copy"
+	"github.com/sirupsen/logrus"
+	"github.com/studio-b12/gowebdav"
+	"golang.org/x/crypto/openpgp"
+)
+
+// script holds all the stateful information required to orchestrate a
+// single backup run.
+type script struct {
+	cli          *client.Client
+	minioClient  *minio.Client
+	webdavClient *gowebdav.Client
+	logger       *logrus.Logger
+	sender       *router.ServiceRouter
+	template     *template.Template
+	hooks        []hook
+	hookLevel    hookLevel
+
+	file  string
+	stats *Stats
+
+	encounteredLock bool
+
+	c *Config
+}
+
+// newScript creates all resources needed for the script to perform actions against
+// remote resources like the Docker engine or remote storage locations. All
+// reading from env vars or other configuration sources is expected to happen
+// in this method.
+func newScript() (*script, error) {
+	stdOut, logBuffer := buffer(os.Stdout)
+	s := &script{
+		c: &Config{},
+		logger: &logrus.Logger{
+			Out:       stdOut,
+			Formatter: new(logrus.TextFormatter),
+			Hooks:     make(logrus.LevelHooks),
+			Level:     logrus.InfoLevel,
+		},
+		stats: &Stats{
+			StartTime: time.Now(),
+			LogOutput: logBuffer,
+			Storages:  StoragesStats{},
+		},
+	}
+
+	s.registerHook(hookLevelPlumbing, func(error) error {
+		s.stats.EndTime = time.Now()
+		s.stats.TookTime = s.stats.EndTime.Sub(s.stats.StartTime)
+		return nil
+	})
+
+	if err := envconfig.Process("", s.c); err != nil {
+		return nil, fmt.Errorf("newScript: failed to process configuration values: %w", err)
+	}
+
+	s.file = path.Join("/tmp", s.c.BackupFilename)
+	if s.c.BackupFilenameExpand {
+		s.file = os.ExpandEnv(s.file)
+		s.c.BackupLatestSymlink = os.ExpandEnv(s.c.BackupLatestSymlink)
+		s.c.BackupPruningPrefix = os.ExpandEnv(s.c.BackupPruningPrefix)
+	}
+	s.file = timeutil.Strftime(&s.stats.StartTime, s.file)
+
+	_, err := os.Stat("/var/run/docker.sock")
+	_, dockerHostSet := os.LookupEnv("DOCKER_HOST")
+	if !os.IsNotExist(err) || dockerHostSet {
+		cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
+		if err != nil {
+			return nil, fmt.Errorf("newScript: failed to create docker client")
+		}
+		s.cli = cli
+	}
+
+	if s.c.AwsS3BucketName != "" {
+		var creds *credentials.Credentials
+		if s.c.AwsAccessKeyID != "" && s.c.AwsSecretAccessKey != "" {
+			creds = credentials.NewStaticV4(
+				s.c.AwsAccessKeyID,
+				s.c.AwsSecretAccessKey,
+				"",
+			)
+		} else if s.c.AwsIamRoleEndpoint != "" {
+			creds = credentials.NewIAM(s.c.AwsIamRoleEndpoint)
+		} else {
+			return nil, errors.New("newScript: AWS_S3_BUCKET_NAME is defined, but no credentials were provided")
+		}
+
+		options := minio.Options{
+			Creds:  creds,
+			Secure: s.c.AwsEndpointProto == "https",
+		}
+
+		if s.c.AwsEndpointInsecure {
+			if !options.Secure {
+				return nil, errors.New("newScript: AWS_ENDPOINT_INSECURE = true is only meaningful for https")
+			}
+
+			transport, err := minio.DefaultTransport(true)
+			if err != nil {
+				return nil, fmt.Errorf("newScript: failed to create default minio transport")
+			}
+			transport.TLSClientConfig.InsecureSkipVerify = true
+			options.Transport = transport
+		}
+
+		mc, err := minio.New(s.c.AwsEndpoint, &options)
+		if err != nil {
+			return nil, fmt.Errorf("newScript: error setting up minio client: %w", err)
+		}
+		s.minioClient = mc
+	}
+
+	if s.c.WebdavUrl != "" {
+		if s.c.WebdavUsername == "" || s.c.WebdavPassword == "" {
+			return nil, errors.New("newScript: WEBDAV_URL is defined, but no credentials were provided")
+		} else {
+			webdavClient := gowebdav.NewClient(s.c.WebdavUrl, s.c.WebdavUsername, s.c.WebdavPassword)
+			s.webdavClient = webdavClient
+			if s.c.WebdavUrlInsecure {
+				defaultTransport, ok := http.DefaultTransport.(*http.Transport)
+				if !ok {
+					return nil, errors.New("newScript: unexpected error when asserting type for http.DefaultTransport")
+				}
+				webdavTransport := defaultTransport.Clone()
+				webdavTransport.TLSClientConfig.InsecureSkipVerify = s.c.WebdavUrlInsecure
+				s.webdavClient.SetTransport(webdavTransport)
+			}
+		}
+	}
+
+	if s.c.EmailNotificationRecipient != "" {
+		emailURL := fmt.Sprintf(
+			"smtp://%s:%s@%s:%d/?from=%s&to=%s",
+			s.c.EmailSMTPUsername,
+			s.c.EmailSMTPPassword,
+			s.c.EmailSMTPHost,
+			s.c.EmailSMTPPort,
+			s.c.EmailNotificationSender,
+			s.c.EmailNotificationRecipient,
+		)
+		s.c.NotificationURLs = append(s.c.NotificationURLs, emailURL)
+		s.logger.Warn(
+			"Using EMAIL_* keys for providing notification configuration has been deprecated and will be removed in the next major version.",
+		)
+		s.logger.Warn(
+			"Please use NOTIFICATION_URLS instead. Refer to the README for an upgrade guide.",
+		)
+	}
+
+	hookLevel, ok := hookLevels[s.c.NotificationLevel]
+	if !ok {
+		return nil, fmt.Errorf("newScript: unknown NOTIFICATION_LEVEL %s", s.c.NotificationLevel)
+	}
+	s.hookLevel = hookLevel
+
+	if len(s.c.NotificationURLs) > 0 {
+		sender, senderErr := shoutrrr.CreateSender(s.c.NotificationURLs...)
+		if senderErr != nil {
+			return nil, fmt.Errorf("newScript: error creating sender: %w", senderErr)
+		}
+		s.sender = sender
+
+		tmpl := template.New("")
+		tmpl.Funcs(templateHelpers)
+		tmpl, err = tmpl.Parse(defaultNotifications)
+		if err != nil {
+			return nil, fmt.Errorf("newScript: unable to parse default notifications templates: %w", err)
+		}
+
+		if fi, err := os.Stat("/etc/dockervolumebackup/notifications.d"); err == nil && fi.IsDir() {
+			tmpl, err = tmpl.ParseGlob("/etc/dockervolumebackup/notifications.d/*.*")
+			if err != nil {
+				return nil, fmt.Errorf("newScript: unable to parse user defined notifications templates: %w", err)
+			}
+		}
+		s.template = tmpl
+
+		// To prevent duplicate notifications, ensure the regsistered callbacks
+		// run mutually exclusive.
+		s.registerHook(hookLevelError, func(err error) error {
+			if err == nil {
+				return nil
+			}
+			return s.notifyFailure(err)
+		})
+		s.registerHook(hookLevelInfo, func(err error) error {
+			if err != nil {
+				return nil
+			}
+			return s.notifySuccess()
+		})
+	}
+
+	return s, nil
+}
+
+func (s *script) runCommands() (func() error, error) {
+	if s.cli == nil {
+		return noop, nil
+	}
+
+	if err := s.runLabeledCommands("docker-volume-backup.exec-pre"); err != nil {
+		return noop, fmt.Errorf("runCommands: error running pre commands: %w", err)
+	}
+	return func() error {
+		if err := s.runLabeledCommands("docker-volume-backup.exec-post"); err != nil {
+			return fmt.Errorf("runCommands: error running post commands: %w", err)
+		}
+		return nil
+	}, nil
+}
+
+// stopContainers stops all Docker containers that are marked as to being
+// stopped during the backup and returns a function that can be called to
+// restart everything that has been stopped.
+func (s *script) stopContainers() (func() error, error) {
+	if s.cli == nil {
+		return noop, nil
+	}
+
+	allContainers, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
+		Quiet: true,
+	})
+	if err != nil {
+		return noop, fmt.Errorf("stopContainersAndRun: error querying for containers: %w", err)
+	}
+
+	containerLabel := fmt.Sprintf(
+		"docker-volume-backup.stop-during-backup=%s",
+		s.c.BackupStopContainerLabel,
+	)
+	containersToStop, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
+		Quiet: true,
+		Filters: filters.NewArgs(filters.KeyValuePair{
+			Key:   "label",
+			Value: containerLabel,
+		}),
+	})
+
+	if err != nil {
+		return noop, fmt.Errorf("stopContainersAndRun: error querying for containers to stop: %w", err)
+	}
+
+	if len(containersToStop) == 0 {
+		return noop, nil
+	}
+
+	s.logger.Infof(
+		"Stopping %d container(s) labeled `%s` out of %d running container(s).",
+		len(containersToStop),
+		containerLabel,
+		len(allContainers),
+	)
+
+	var stoppedContainers []types.Container
+	var stopErrors []error
+	for _, container := range containersToStop {
+		if err := s.cli.ContainerStop(context.Background(), container.ID, nil); err != nil {
+			stopErrors = append(stopErrors, err)
+		} else {
+			stoppedContainers = append(stoppedContainers, container)
+		}
+	}
+
+	var stopError error
+	if len(stopErrors) != 0 {
+		stopError = fmt.Errorf(
+			"stopContainersAndRun: %d error(s) stopping containers: %w",
+			len(stopErrors),
+			join(stopErrors...),
+		)
+	}
+
+	s.stats.Containers = ContainersStats{
+		All:     uint(len(allContainers)),
+		ToStop:  uint(len(containersToStop)),
+		Stopped: uint(len(stoppedContainers)),
+	}
+
+	return func() error {
+		servicesRequiringUpdate := map[string]struct{}{}
+
+		var restartErrors []error
+		for _, container := range stoppedContainers {
+			if swarmServiceName, ok := container.Labels["com.docker.swarm.service.name"]; ok {
+				servicesRequiringUpdate[swarmServiceName] = struct{}{}
+				continue
+			}
+			if err := s.cli.ContainerStart(context.Background(), container.ID, types.ContainerStartOptions{}); err != nil {
+				restartErrors = append(restartErrors, err)
+			}
+		}
+
+		if len(servicesRequiringUpdate) != 0 {
+			services, _ := s.cli.ServiceList(context.Background(), types.ServiceListOptions{})
+			for serviceName := range servicesRequiringUpdate {
+				var serviceMatch swarm.Service
+				for _, service := range services {
+					if service.Spec.Name == serviceName {
+						serviceMatch = service
+						break
+					}
+				}
+				if serviceMatch.ID == "" {
+					return fmt.Errorf("stopContainersAndRun: couldn't find service with name %s", serviceName)
+				}
+				serviceMatch.Spec.TaskTemplate.ForceUpdate = 1
+				if _, err := s.cli.ServiceUpdate(
+					context.Background(), serviceMatch.ID,
+					serviceMatch.Version, serviceMatch.Spec, types.ServiceUpdateOptions{},
+				); err != nil {
+					restartErrors = append(restartErrors, err)
+				}
+			}
+		}
+
+		if len(restartErrors) != 0 {
+			return fmt.Errorf(
+				"stopContainersAndRun: %d error(s) restarting containers and services: %w",
+				len(restartErrors),
+				join(restartErrors...),
+			)
+		}
+		s.logger.Infof(
+			"Restarted %d container(s) and the matching service(s).",
+			len(stoppedContainers),
+		)
+		return nil
+	}, stopError
+}
+
+// takeBackup creates a tar archive of the configured backup location and
+// saves it to disk.
+func (s *script) takeBackup() error {
+	backupSources := s.c.BackupSources
+
+	if s.c.BackupFromSnapshot {
+		s.logger.Warn(
+			"Using BACKUP_FROM_SNAPSHOT has been deprecated and will be removed in the next major version.",
+		)
+		s.logger.Warn(
+			"Please use `exec-pre` and `exec-post` commands to prepare your backup sources. Refer to the README for an upgrade guide.",
+		)
+		backupSources = filepath.Join("/tmp", s.c.BackupSources)
+		// copy before compressing guard against a situation where backup folder's content are still growing.
+		s.registerHook(hookLevelPlumbing, func(error) error {
+			if err := remove(backupSources); err != nil {
+				return fmt.Errorf("takeBackup: error removing snapshot: %w", err)
+			}
+			s.logger.Infof("Removed snapshot `%s`.", backupSources)
+			return nil
+		})
+		if err := copy.Copy(s.c.BackupSources, backupSources, copy.Options{
+			PreserveTimes: true,
+			PreserveOwner: true,
+		}); err != nil {
+			return fmt.Errorf("takeBackup: error creating snapshot: %w", err)
+		}
+		s.logger.Infof("Created snapshot of `%s` at `%s`.", s.c.BackupSources, backupSources)
+	}
+
+	tarFile := s.file
+	s.registerHook(hookLevelPlumbing, func(error) error {
+		if err := remove(tarFile); err != nil {
+			return fmt.Errorf("takeBackup: error removing tar file: %w", err)
+		}
+		s.logger.Infof("Removed tar file `%s`.", tarFile)
+		return nil
+	})
+
+	backupPath, err := filepath.Abs(stripTrailingSlashes(backupSources))
+	if err != nil {
+		return fmt.Errorf("takeBackup: error getting absolute path: %w", err)
+	}
+
+	var filesEligibleForBackup []string
+	if err := filepath.WalkDir(backupPath, func(path string, di fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+
+		if s.c.BackupExcludeRegexp.Re != nil && s.c.BackupExcludeRegexp.Re.MatchString(path) {
+			return nil
+		}
+		filesEligibleForBackup = append(filesEligibleForBackup, path)
+		return nil
+	}); err != nil {
+		return fmt.Errorf("compress: error walking filesystem tree: %w", err)
+	}
+
+	if err := createArchive(filesEligibleForBackup, backupSources, tarFile); err != nil {
+		return fmt.Errorf("takeBackup: error compressing backup folder: %w", err)
+	}
+
+	s.logger.Infof("Created backup of `%s` at `%s`.", backupSources, tarFile)
+	return nil
+}
+
+// encryptBackup encrypts the backup file using PGP and the configured passphrase.
+// In case no passphrase is given it returns early, leaving the backup file
+// untouched.
+func (s *script) encryptBackup() error {
+	if s.c.GpgPassphrase == "" {
+		return nil
+	}
+
+	gpgFile := fmt.Sprintf("%s.gpg", s.file)
+	s.registerHook(hookLevelPlumbing, func(error) error {
+		if err := remove(gpgFile); err != nil {
+			return fmt.Errorf("encryptBackup: error removing gpg file: %w", err)
+		}
+		s.logger.Infof("Removed GPG file `%s`.", gpgFile)
+		return nil
+	})
+
+	outFile, err := os.Create(gpgFile)
+	defer outFile.Close()
+	if err != nil {
+		return fmt.Errorf("encryptBackup: error opening out file: %w", err)
+	}
+
+	_, name := path.Split(s.file)
+	dst, err := openpgp.SymmetricallyEncrypt(outFile, []byte(s.c.GpgPassphrase), &openpgp.FileHints{
+		IsBinary: true,
+		FileName: name,
+	}, nil)
+	defer dst.Close()
+	if err != nil {
+		return fmt.Errorf("encryptBackup: error encrypting backup file: %w", err)
+	}
+
+	src, err := os.Open(s.file)
+	if err != nil {
+		return fmt.Errorf("encryptBackup: error opening backup file `%s`: %w", s.file, err)
+	}
+
+	if _, err := io.Copy(dst, src); err != nil {
+		return fmt.Errorf("encryptBackup: error writing ciphertext to file: %w", err)
+	}
+
+	s.file = gpgFile
+	s.logger.Infof("Encrypted backup using given passphrase, saving as `%s`.", s.file)
+	return nil
+}
+
+// copyBackup makes sure the backup file is copied to both local and remote locations
+// as per the given configuration.
+func (s *script) copyBackup() error {
+	_, name := path.Split(s.file)
+	if stat, err := os.Stat(s.file); err != nil {
+		return fmt.Errorf("copyBackup: unable to stat backup file: %w", err)
+	} else {
+		size := stat.Size()
+		s.stats.BackupFile = BackupFileStats{
+			Size:     uint64(size),
+			Name:     name,
+			FullPath: s.file,
+		}
+	}
+
+	if s.minioClient != nil {
+		if _, err := s.minioClient.FPutObject(context.Background(), s.c.AwsS3BucketName, filepath.Join(s.c.AwsS3Path, name), s.file, minio.PutObjectOptions{
+			ContentType: "application/tar+gzip",
+		}); err != nil {
+			return fmt.Errorf("copyBackup: error uploading backup to remote storage: %w", err)
+		}
+		s.logger.Infof("Uploaded a copy of backup `%s` to bucket `%s`.", s.file, s.c.AwsS3BucketName)
+	}
+
+	if s.webdavClient != nil {
+		bytes, err := os.ReadFile(s.file)
+		if err != nil {
+			return fmt.Errorf("copyBackup: error reading the file to be uploaded: %w", err)
+		}
+		if err := s.webdavClient.MkdirAll(s.c.WebdavPath, 0644); err != nil {
+			return fmt.Errorf("copyBackup: error creating directory '%s' on WebDAV server: %w", s.c.WebdavPath, err)
+		}
+		if err := s.webdavClient.Write(filepath.Join(s.c.WebdavPath, name), bytes, 0644); err != nil {
+			return fmt.Errorf("copyBackup: error uploading the file to WebDAV server: %w", err)
+		}
+		s.logger.Infof("Uploaded a copy of backup `%s` to WebDAV-URL '%s' at path '%s'.", s.file, s.c.WebdavUrl, s.c.WebdavPath)
+	}
+
+	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
+		if err := copyFile(s.file, path.Join(s.c.BackupArchive, name)); err != nil {
+			return fmt.Errorf("copyBackup: error copying file to local archive: %w", err)
+		}
+		s.logger.Infof("Stored copy of backup `%s` in local archive `%s`.", s.file, s.c.BackupArchive)
+		if s.c.BackupLatestSymlink != "" {
+			symlink := path.Join(s.c.BackupArchive, s.c.BackupLatestSymlink)
+			if _, err := os.Lstat(symlink); err == nil {
+				os.Remove(symlink)
+			}
+			if err := os.Symlink(name, symlink); err != nil {
+				return fmt.Errorf("copyBackup: error creating latest symlink: %w", err)
+			}
+			s.logger.Infof("Created/Updated symlink `%s` for latest backup.", s.c.BackupLatestSymlink)
+		}
+	}
+	return nil
+}
+
+// pruneBackups rotates away backups from local and remote storages using
+// the given configuration. In case the given configuration would delete all
+// backups, it does nothing instead and logs a warning.
+func (s *script) pruneBackups() error {
+	if s.c.BackupRetentionDays < 0 {
+		return nil
+	}
+
+	deadline := time.Now().AddDate(0, 0, -int(s.c.BackupRetentionDays)).Add(s.c.BackupPruningLeeway)
+
+	// doPrune holds general control flow that applies to any kind of storage.
+	// Callers can pass in a thunk that performs the actual deletion of files.
+	var doPrune = func(lenMatches, lenCandidates int, description string, doRemoveFiles func() error) error {
+		if lenMatches != 0 && lenMatches != lenCandidates {
+			if err := doRemoveFiles(); err != nil {
+				return err
+			}
+			s.logger.Infof(
+				"Pruned %d out of %d %s as their age exceeded the configured retention period of %d days.",
+				lenMatches,
+				lenCandidates,
+				description,
+				s.c.BackupRetentionDays,
+			)
+		} else if lenMatches != 0 && lenMatches == lenCandidates {
+			s.logger.Warnf("The current configuration would delete all %d existing %s.", lenMatches, description)
+			s.logger.Warn("Refusing to do so, please check your configuration.")
+		} else {
+			s.logger.Infof("None of %d existing %s were pruned.", lenCandidates, description)
+		}
+		return nil
+	}
+
+	if s.minioClient != nil {
+		candidates := s.minioClient.ListObjects(context.Background(), s.c.AwsS3BucketName, minio.ListObjectsOptions{
+			WithMetadata: true,
+			Prefix:       filepath.Join(s.c.AwsS3Path, s.c.BackupPruningPrefix),
+			Recursive:    true,
+		})
+
+		var matches []minio.ObjectInfo
+		var lenCandidates int
+		for candidate := range candidates {
+			lenCandidates++
+			if candidate.Err != nil {
+				return fmt.Errorf(
+					"pruneBackups: error looking up candidates from remote storage: %w",
+					candidate.Err,
+				)
+			}
+			if candidate.LastModified.Before(deadline) {
+				matches = append(matches, candidate)
+			}
+		}
+
+		s.stats.Storages.S3 = StorageStats{
+			Total:  uint(lenCandidates),
+			Pruned: uint(len(matches)),
+		}
+
+		doPrune(len(matches), lenCandidates, "remote backup(s)", func() error {
+			objectsCh := make(chan minio.ObjectInfo)
+			go func() {
+				for _, match := range matches {
+					objectsCh <- match
+				}
+				close(objectsCh)
+			}()
+			errChan := s.minioClient.RemoveObjects(context.Background(), s.c.AwsS3BucketName, objectsCh, minio.RemoveObjectsOptions{})
+			var removeErrors []error
+			for result := range errChan {
+				if result.Err != nil {
+					removeErrors = append(removeErrors, result.Err)
+				}
+			}
+			if len(removeErrors) != 0 {
+				return join(removeErrors...)
+			}
+			return nil
+		})
+	}
+
+	if s.webdavClient != nil {
+		candidates, err := s.webdavClient.ReadDir(s.c.WebdavPath)
+		if err != nil {
+			return fmt.Errorf("pruneBackups: error looking up candidates from remote storage: %w", err)
+		}
+		var matches []fs.FileInfo
+		var lenCandidates int
+		for _, candidate := range candidates {
+			if !strings.HasPrefix(candidate.Name(), s.c.BackupPruningPrefix) {
+				continue
+			}
+			lenCandidates++
+			if candidate.ModTime().Before(deadline) {
+				matches = append(matches, candidate)
+			}
+		}
+
+		s.stats.Storages.WebDAV = StorageStats{
+			Total:  uint(lenCandidates),
+			Pruned: uint(len(matches)),
+		}
+
+		doPrune(len(matches), lenCandidates, "WebDAV backup(s)", func() error {
+			for _, match := range matches {
+				if err := s.webdavClient.Remove(filepath.Join(s.c.WebdavPath, match.Name())); err != nil {
+					return fmt.Errorf("pruneBackups: error removing file from WebDAV storage: %w", err)
+				}
+			}
+			return nil
+		})
+	}
+
+	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
+		globPattern := path.Join(
+			s.c.BackupArchive,
+			fmt.Sprintf("%s*", s.c.BackupPruningPrefix),
+		)
+		globMatches, err := filepath.Glob(globPattern)
+		if err != nil {
+			return fmt.Errorf(
+				"pruneBackups: error looking up matching files using pattern %s: %w",
+				globPattern,
+				err,
+			)
+		}
+
+		var candidates []string
+		for _, candidate := range globMatches {
+			fi, err := os.Lstat(candidate)
+			if err != nil {
+				return fmt.Errorf(
+					"pruneBackups: error calling Lstat on file %s: %w",
+					candidate,
+					err,
+				)
+			}
+
+			if fi.Mode()&os.ModeSymlink != os.ModeSymlink {
+				candidates = append(candidates, candidate)
+			}
+		}
+
+		var matches []string
+		for _, candidate := range candidates {
+			fi, err := os.Stat(candidate)
+			if err != nil {
+				return fmt.Errorf(
+					"pruneBackups: error calling stat on file %s: %w",
+					candidate,
+					err,
+				)
+			}
+			if fi.ModTime().Before(deadline) {
+				matches = append(matches, candidate)
+			}
+		}
+
+		s.stats.Storages.Local = StorageStats{
+			Total:  uint(len(candidates)),
+			Pruned: uint(len(matches)),
+		}
+
+		doPrune(len(matches), len(candidates), "local backup(s)", func() error {
+			var removeErrors []error
+			for _, match := range matches {
+				if err := os.Remove(match); err != nil {
+					removeErrors = append(removeErrors, err)
+				}
+			}
+			if len(removeErrors) != 0 {
+				return fmt.Errorf(
+					"pruneBackups: %d error(s) deleting local files, starting with: %w",
+					len(removeErrors),
+					join(removeErrors...),
+				)
+			}
+			return nil
+		})
+	}
+	return nil
+}
+
+// must exits the script run prematurely in case the given error
+// is non-nil.
+func (s *script) must(err error) {
+	if err != nil {
+		s.logger.Errorf("Fatal error running backup: %s", err)
+		panic(err)
+	}
+}
--- a/cmd/backup/stats.go
+++ b/cmd/backup/stats.go
@@ -0,0 +1,50 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"bytes"
+	"time"
+)
+
+// ContainersStats stats about the docker containers
+type ContainersStats struct {
+	All        uint
+	ToStop     uint
+	Stopped    uint
+	StopErrors uint
+}
+
+// BackupFileStats stats about the created backup file
+type BackupFileStats struct {
+	Name     string
+	FullPath string
+	Size     uint64
+}
+
+// StorageStats stats about the status of an archival directory
+type StorageStats struct {
+	Total       uint
+	Pruned      uint
+	PruneErrors uint
+}
+
+// StoragesStats stats about each possible archival location (Local, WebDAV, S3)
+type StoragesStats struct {
+	Local  StorageStats
+	WebDAV StorageStats
+	S3     StorageStats
+}
+
+// Stats global stats regarding script execution
+type Stats struct {
+	StartTime  time.Time
+	EndTime    time.Time
+	TookTime   time.Duration
+	LockedTime time.Duration
+	LogOutput  *bytes.Buffer
+	Containers ContainersStats
+	BackupFile BackupFileStats
+	Storages   StoragesStats
+}
--- a/cmd/backup/util.go
+++ b/cmd/backup/util.go
@@ -0,0 +1,90 @@
+// Copyright 2022 - Offen Authors <hioffen@posteo.de>
+// SPDX-License-Identifier: MPL-2.0
+
+package main
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"strings"
+)
+
+var noop = func() error { return nil }
+
+// copy creates a copy of the file located at `dst` at `src`.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		return err
+	}
+	defer in.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+
+	_, err = io.Copy(out, in)
+	if err != nil {
+		out.Close()
+		return err
+	}
+	return out.Close()
+}
+
+// join takes a list of errors and joins them into a single error
+func join(errs ...error) error {
+	if len(errs) == 1 {
+		return errs[0]
+	}
+	var msgs []string
+	for _, err := range errs {
+		if err == nil {
+			continue
+		}
+		msgs = append(msgs, err.Error())
+	}
+	return errors.New("[" + strings.Join(msgs, ", ") + "]")
+}
+
+// remove removes the given file or directory from disk.
+func remove(location string) error {
+	fi, err := os.Lstat(location)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return fmt.Errorf("remove: error checking for existence of `%s`: %w", location, err)
+	}
+	if fi.IsDir() {
+		err = os.RemoveAll(location)
+	} else {
+		err = os.Remove(location)
+	}
+	if err != nil {
+		return fmt.Errorf("remove: error removing `%s`: %w", location, err)
+	}
+	return nil
+}
+
+// buffer takes an io.Writer and returns a wrapped version of the
+// writer that writes to both the original target as well as the returned buffer
+func buffer(w io.Writer) (io.Writer, *bytes.Buffer) {
+	buffering := &bufferingWriter{buf: bytes.Buffer{}, writer: w}
+	return buffering, &buffering.buf
+}
+
+type bufferingWriter struct {
+	buf    bytes.Buffer
+	writer io.Writer
+}
+
+func (b *bufferingWriter) Write(p []byte) (n int, err error) {
+	if n, err := b.buf.Write(p); err != nil {
+		return n, fmt.Errorf("bufferingWriter: error writing to buffer: %w", err)
+	}
+	return b.writer.Write(p)
+}
--- a/docs/NOTIFICATION-TEMPLATES.md
+++ b/docs/NOTIFICATION-TEMPLATES.md
@@ -0,0 +1,39 @@
+# Notification templates reference
+
+In order to customize title and body of notifications you'll have to write a [go template](https://pkg.go.dev/text/template) and mount it inside the `/etc/dockervolumebackup/notifications.d/` directory.
+
+Configuration, data about the backup run and helper functions will be passed to this template, this page documents them fully.
+
+## Data
+Here is a list of all data passed to the template:
+
+* `Config`: this object holds the configuration that has been passed to the script. The field names are the name of the recognized environment variables converted in PascalCase. (e.g. `BACKUP_STOP_CONTAINER_LABEL` becomes `BackupStopContainerLabel`)
+* `Error`: the error that made the backup fail. Only available in the `title_failure` and `body_failure` templates
+* `Stats`: objects that holds stats regarding script execution. In case of an unsuccessful run, some information may not be available.
+  * `StartTime`: time when the script started execution
+  * `EndTime`: time when the backup has completed successfully (after pruning)
+  * `TookTime`: amount of time it took for the backup to run. (equal to `EndTime - StartTime`)
+  * `LockedTime`: amount of time it took for the backup to acquire the exclusive lock
+  * `LogOutput`: full log of the application
+  * `Containers`: object containing stats about the docker containers
+    * `All`: total number of containers
+    * `ToStop`: number of containers matched by the stop rule
+    * `Stopped`: number of containers successfully stopped
+    * `StopErrors`: number of containers that were unable to be stopped (equal to `ToStop - Stopped`)
+  * `BackupFile`: object containing information about the backup file
+    * `Name`: name of the backup file (e.g. `backup-2022-02-11T01-00-00.tar.gz`)
+    * `FullPath`: full path of the backup file (e.g. `/archive/backup-2022-02-11T01-00-00.tar.gz`)
+    * `Size`: size in bytes of the backup file
+  * `Storages`: object that holds stats about each storage
+    * `Local`, `S3` or `WebDAV`:
+      * `Total`: total number of backup files
+      * `Pruned`: number of backup files that were deleted due to pruning rule
+      * `PruneErrors`: number of backup files that were unable to be pruned
+
+## Functions
+
+Some formatting functions are also available:
+
+* `formatTime`: formats a time object using [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339) format (e.g. `2022-02-11T01:00:00Z`)
+* `formatBytesBin`: formats an amount of bytes using powers of 1024 (e.g. `7055258` bytes will be `6.7 MiB`) 
+* `formatBytesDec`: formats an amount of bytes using powers of 1000 (e.g. `7055258` bytes will be `7.1 MB`)
--- a/entrypoint.sh
+++ b/entrypoint.sh
@@ -5,10 +5,21 @@

 set -e

-BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"
+if [ ! -d "/etc/dockervolumebackup/conf.d" ]; then
+  BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"

-echo "Installing cron.d entry with expression $BACKUP_CRON_EXPRESSION."
-echo "$BACKUP_CRON_EXPRESSION backup 2>&1" | crontab -
+  echo "Installing cron.d entry with expression $BACKUP_CRON_EXPRESSION."
+  echo "$BACKUP_CRON_EXPRESSION backup 2>&1" | crontab -
+else
+  echo "/etc/dockervolumebackup/conf.d was found, using configuration files from this directory."
+
+  for file in /etc/dockervolumebackup/conf.d/*; do
+    source $file
+    BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"
+    echo "Appending cron.d entry with expression $BACKUP_CRON_EXPRESSION and configuration file $file"
+    (crontab -l; echo "$BACKUP_CRON_EXPRESSION /bin/sh -c 'set -a; source $file; set +a && backup' 2>&1") | crontab -
+  done
+fi

 echo "Starting cron in foreground."
 crond -f -l 8
--- a/go.mod
+++ b/go.mod
@@ -1,18 +1,20 @@
 module github.com/offen/docker-volume-backup

-go 1.17
+go 1.18

 require (
+	github.com/containrrr/shoutrrr v0.5.2
+	github.com/cosiner/argv v0.1.0
 	github.com/docker/docker v20.10.11+incompatible
-	github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df
 	github.com/gofrs/flock v0.8.1
 	github.com/kelseyhightower/envconfig v1.4.0
 	github.com/leekchan/timeutil v0.0.0-20150802142658-28917288c48d
-	github.com/m90/targz v0.0.0-20210904082215-2e9a4529a615
 	github.com/minio/minio-go/v7 v7.0.16
 	github.com/otiai10/copy v1.7.0
 	github.com/sirupsen/logrus v1.8.1
+	github.com/studio-b12/gowebdav v0.0.0-20220128162035-c7b1ff8a5e62
 	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
+	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
 )

 require (
@@ -22,18 +24,25 @@ require (
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/docker/go-units v0.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.0 // indirect
+	github.com/fatih/color v1.10.0 // indirect
+	github.com/fsnotify/fsnotify v1.4.9 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/protobuf v1.5.0 // indirect
 	github.com/google/uuid v1.3.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/klauspost/compress v1.13.6 // indirect
 	github.com/klauspost/cpuid/v2 v2.0.9 // indirect
+	github.com/mattn/go-colorable v0.1.8 // indirect
+	github.com/mattn/go-isatty v0.0.12 // indirect
 	github.com/minio/md5-simd v1.1.2 // indirect
 	github.com/minio/sha256-simd v1.0.0 // indirect
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/morikuni/aec v1.0.0 // indirect
+	github.com/nxadm/tail v1.4.6 // indirect
+	github.com/onsi/ginkgo v1.14.2 // indirect
+	github.com/onsi/gomega v1.10.3 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/opencontainers/image-spec v1.0.1 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
@@ -41,10 +50,11 @@ require (
 	golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 // indirect
 	golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1 // indirect
 	golang.org/x/text v0.3.6 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a // indirect
 	google.golang.org/grpc v1.33.2 // indirect
 	google.golang.org/protobuf v1.26.0 // indirect
-	gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
-	gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df // indirect
 	gopkg.in/ini.v1 v1.65.0 // indirect
+	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
 )
--- a/go.sum
+++ b/go.sum
@@ -61,6 +61,7 @@ github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAE
 github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
 github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d/go.mod h1:HI8ITrYtUY+O+ZhtlqUnD8+KwNPOyugEhfP9fdUIaEQ=
+github.com/agnivade/wasmbrowsertest v0.3.1/go.mod h1:zQt6ZTdl338xxRaMW395qccVE2eQm0SjC/SDz0mPWQI=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -89,6 +90,12 @@ github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghf
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/checkpoint-restore/go-criu/v4 v4.1.0/go.mod h1:xUQBLp4RLc5zJtWY++yjOoMoB5lihDt7fai+75m+rGw=
 github.com/checkpoint-restore/go-criu/v5 v5.0.0/go.mod h1:cfwC0EG7HMUenopBsUf9d89JlCLQIfgVcNsNN0t6T2M=
+github.com/chromedp/cdproto v0.0.0-20190614062957-d6d2f92b486d/go.mod h1:S8mB5wY3vV+vRIzf39xDXsw3XKYewW9X6rW2aEmkrSw=
+github.com/chromedp/cdproto v0.0.0-20190621002710-8cbd498dd7a0/go.mod h1:S8mB5wY3vV+vRIzf39xDXsw3XKYewW9X6rW2aEmkrSw=
+github.com/chromedp/cdproto v0.0.0-20190812224334-39ef923dcb8d/go.mod h1:0YChpVzuLJC5CPr+x3xkHN6Z8KOSXjNbL7qV8Wc4GW0=
+github.com/chromedp/cdproto v0.0.0-20190926234355-1b4886c6fad6/go.mod h1:0YChpVzuLJC5CPr+x3xkHN6Z8KOSXjNbL7qV8Wc4GW0=
+github.com/chromedp/chromedp v0.3.1-0.20190619195644-fd957a4d2901/go.mod h1:mJdvfrVn594N9tfiPecUidF6W5jPRKHymqHfzbobPsM=
+github.com/chromedp/chromedp v0.4.0/go.mod h1:DC3QUn4mJ24dwjcaGQLoZrhm4X/uPHZ6spDbS2uFhm4=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
@@ -183,8 +190,11 @@ github.com/containernetworking/plugins v0.9.1/go.mod h1:xP/idU2ldlzN6m4p5LmGiwRD
 github.com/containers/ocicrypt v1.0.1/go.mod h1:MeJDzk1RJHv89LjsH0Sp5KTY3ZYkjXO/C+bKAeWFIrc=
 github.com/containers/ocicrypt v1.1.0/go.mod h1:b8AOe0YR67uU8OqfVNcznfFpAzu3rdgUV4GP9qXPfu4=
 github.com/containers/ocicrypt v1.1.1/go.mod h1:Dm55fwWm1YZAjYRaJ94z2mfZikIyIN4B0oB3dj3jFxY=
+github.com/containrrr/shoutrrr v0.5.2 h1:97P+wZDpN2gzBKLt4PDP1ZUTLz+k8AJVs40WOu9NNw8=
+github.com/containrrr/shoutrrr v0.5.2/go.mod h1:XSU8tOIZ1JG8m6OuPozfGLpj6Ed+S8ZrRJaEodQhbzw=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-iptables v0.4.5/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU=
 github.com/coreos/go-iptables v0.5.0/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU=
 github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
@@ -198,6 +208,8 @@ github.com/coreos/go-systemd/v22 v22.1.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+
 github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
 github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
+github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg=
+github.com/cosiner/argv v0.1.0/go.mod h1:EusR6TucWKX+zFgtdUsKT2Cvg45K5rtpCcWz4hK06d8=
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
@@ -234,6 +246,7 @@ github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3
 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
 github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
 github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/edsrzf/mmap-go v1.0.0/go.mod h1:YO35OhQPt3KJa3ryjFM5Bs14WD66h8eGKpfaBNrHW5M=
 github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
 github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
@@ -242,21 +255,27 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
 github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
+github.com/fatih/color v1.6.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fatih/color v1.10.0 h1:s36xzo75JdqLaaWoiEHk767eHiwo0598uUxyfiPkDsg=
+github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
 github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fullsailor/pkcs7 v0.0.0-20190404230743-d7302db945fa/go.mod h1:KnogPXtdwXqoenmZCw6S+25EAm2MkxbG0deNDu4cbSA=
 github.com/garyburd/redigo v0.0.0-20150301180006-535138d7bcd7/go.mod h1:NR3MbYisc3/PwhQ00EMzDiPmrwpPxAn5GI05/YaO1SY=
 github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
+github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
-github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df h1:Bao6dhmbTA1KFVxmJ6nBoMuOJit2yjEgLJpIMYpop0E=
-github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df/go.mod h1:GJr+FCSXshIwgHBtLglIg9M2l2kQSi6QjVAngtzI08Y=
 github.com/go-ini/ini v1.25.4/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
+github.com/go-interpreter/wagon v0.5.1-0.20190713202023-55a163980b6c/go.mod h1:5+b/MBYkclRZngKF5s6qrgWxSLgE9F5dFdO1hAueZLc=
+github.com/go-interpreter/wagon v0.6.0/go.mod h1:5+b/MBYkclRZngKF5s6qrgWxSLgE9F5dFdO1hAueZLc=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
@@ -270,7 +289,14 @@ github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL9
 github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
 github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
 github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
+github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/go-playground/validator/v10 v10.2.0/go.mod h1:uOYAAleCW8F/7oMFd6aG0GOhaH6EGOAJShg8Id5JGkI=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
+github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee/go.mod h1:L0fX3K22YWvt/FAX9NnzrNzcI4wNYi9Yku4O0LKYflo=
+github.com/gobwas/pool v0.2.0/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
+github.com/gobwas/ws v1.0.2/go.mod h1:szmBTxLgaFppYjEmNtny/v3w89xOydFnnZMcgRRu/EM=
 github.com/godbus/dbus v0.0.0-20151105175453-c7fdd8b5cd55/go.mod h1:/YcGZj5zSblfDWMMoOzV4fas9FZnQYTkDnsGvmh2Grw=
 github.com/godbus/dbus v0.0.0-20180201030542-885f9cc04c9c/go.mod h1:/YcGZj5zSblfDWMMoOzV4fas9FZnQYTkDnsGvmh2Grw=
 github.com/godbus/dbus v0.0.0-20190422162347-ade71ed3457e/go.mod h1:bBOAhwG1umN6/6ZUMtDFBMQR8jRg9O75tm9K00oMsK4=
@@ -331,6 +357,7 @@ github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
+github.com/google/pprof v0.0.0-20190908185732-236ed259b199/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
@@ -338,6 +365,7 @@ github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm4
 github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.1.5/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -350,6 +378,7 @@ github.com/gorilla/mux v1.7.2 h1:zoNxOV7WjqXptQOVngLmcSQgXmgk4NMz1HibBchjl/I=
 github.com/gorilla/mux v1.7.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
 github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
+github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
@@ -372,11 +401,14 @@ github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH
 github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/j-keck/arping v0.0.0-20160618110441-2cf9dc699c56/go.mod h1:ymszkNOg6tORTn+6F6j+Jc8TOr5osrynvN6ivFWZ2GA=
+github.com/jarcoal/httpmock v1.0.4 h1:jp+dy/+nonJE4g4xbVtl9QdrUNbn6/3hDT5R4nDIZnA=
+github.com/jarcoal/httpmock v1.0.4/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik=
 github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jmespath/go-jmespath v0.0.0-20160803190731-bd40a432e4c7/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
@@ -390,18 +422,20 @@ github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvW
 github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/klauspost/compress v1.10.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
+github.com/klauspost/compress v1.11.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
 github.com/klauspost/compress v1.13.6 h1:P76CopJELS0TiO2mebmnzgWaajssP/EszplttgQxcgc=
 github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
 github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
-github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
 github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
 github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/knq/sysutil v0.0.0-20181215143952-f05b59f0f307/go.mod h1:BjPj+aVjl9FW/cCGiF3nGh5v+9Gd3VCgBQbod/GlMaQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@@ -416,15 +450,25 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/leekchan/timeutil v0.0.0-20150802142658-28917288c48d h1:2puqoOQwi3Ai1oznMOsFIbifm6kIfJaLLyYzWD4IzTs=
 github.com/leekchan/timeutil v0.0.0-20150802142658-28917288c48d/go.mod h1:hO90vCP2x3exaSH58BIAowSKvV+0OsY21TtzuFGHON4=
-github.com/m90/targz v0.0.0-20210904082215-2e9a4529a615 h1:rn0LO2tQEgCDOct8qnbcslTUpAIWdVlWcGkjoumhf2U=
-github.com/m90/targz v0.0.0-20210904082215-2e9a4529a615/go.mod h1:YZK3bSO/oVlk9G+v00BxgzxW2Us4p/R4ysHOBjk0fJI=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/mailru/easyjson v0.0.0-20190403194419-1ea4449da983/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20190620125010-da37f6c1e481/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
 github.com/marstr/guid v1.1.0/go.mod h1:74gB1z2wpxxInTG6yaqA7KrtM0NZ+RbrcqDvYHefzho=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
+github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
+github.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ0s8=
+github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
+github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
 github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
+github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
+github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
+github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
+github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
 github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
 github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
@@ -442,6 +486,7 @@ github.com/mistifyio/go-zfs v2.1.2-0.20190413222219-f784269be439+incompatible/go
 github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/mitchellh/mapstructure v1.2.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/mitchellh/osext v0.0.0-20151018003038-5e2d6d41470f/go.mod h1:OkQIRizQZAeMln+1tSwduZz7+Af5oFlKirV/MSYes2A=
 github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
 github.com/moby/sys/mountinfo v0.4.0/go.mod h1:rEr8tzG/lsIZHBtN/JjGG+LMYx9eXgW2JI+6q0qou+A=
@@ -465,19 +510,27 @@ github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRW
 github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
 github.com/ncw/swift v1.0.47/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
 github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
+github.com/nxadm/tail v1.4.6 h1:11TGpSHY7Esh/i/qnq02Jo5oVrI1Gue8Slbq0ujPZFQ=
+github.com/nxadm/tail v1.4.6/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
 github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
 github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
 github.com/onsi/ginkgo v0.0.0-20151202141238-7f8ab55aaf3b/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.10.3/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
+github.com/onsi/ginkgo v1.14.2 h1:8mVmC9kjFFmA8H4pKMUhcblgifdkOIXPvbhN1T36q1M=
+github.com/onsi/ginkgo v1.14.2/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
 github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
+github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
+github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
+github.com/onsi/gomega v1.10.3 h1:gph6h/qe9GSUw1NhH1gp+qb+h8rXD8Cy60Z32Qw3ELA=
 github.com/onsi/gomega v1.10.3/go.mod h1:V9xEwhxec5O8UDM77eCW8vLymOMltsqPVYWrpDsH8xc=
 github.com/opencontainers/go-digest v0.0.0-20170106003457-a6d0ee40d420/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
 github.com/opencontainers/go-digest v0.0.0-20180430190053-c9281466c8b2/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
@@ -512,6 +565,7 @@ github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT9
 github.com/otiai10/mint v1.3.3 h1:7JgpsBaN0uMkyju4tbYHu0mnM55hNKVYLsXmwr15NQI=
 github.com/otiai10/mint v1.3.3/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
 github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -562,6 +616,7 @@ github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdh
 github.com/seccomp/libseccomp-golang v0.9.1/go.mod h1:GbW5+tmTXfcxTToHLXlScSlAvWlF4P2Ca7zGrPiEpWo=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
 github.com/sirupsen/logrus v1.0.4-0.20170822132746-89742aefa4b2/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
+github.com/sirupsen/logrus v1.0.5/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
 github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
@@ -578,16 +633,20 @@ github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasO
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
 github.com/spf13/cobra v0.0.2-0.20171109065643-2da4a54c5cee/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
 github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
+github.com/spf13/cobra v0.0.7/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
 github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
 github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.1-0.20171106142849-4c012f6dcd95/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
+github.com/spf13/viper v1.6.3/go.mod h1:jUMtyi0/lB5yZH/FjyGAoH7IMNrIhlBf6pXZmbMDvzw=
 github.com/stefanberger/go-pkcs11uri v0.0.0-20201008174630-78d3cae3a980/go.mod h1:AO3tvPzVZ/ayst6UlUKUv6rcPQInYe3IknH3jYhAKu8=
 github.com/stretchr/objx v0.0.0-20180129172003-8a3f7159479f/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -599,13 +658,21 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/studio-b12/gowebdav v0.0.0-20211109083228-3f8721cd4b6f h1:L2NE7BXnSlSLoNYZ0lCwZDjdnYjCNYC71k9ClZUTFTs=
+github.com/studio-b12/gowebdav v0.0.0-20211109083228-3f8721cd4b6f/go.mod h1:bHA7t77X/QFExdeAnDzK6vKM34kEZAcE1OX4MfiwjkE=
+github.com/studio-b12/gowebdav v0.0.0-20220128162035-c7b1ff8a5e62 h1:b2nJXyPCa9HY7giGM+kYcnQ71m14JnGdQabMPmyt++8=
+github.com/studio-b12/gowebdav v0.0.0-20220128162035-c7b1ff8a5e62/go.mod h1:bHA7t77X/QFExdeAnDzK6vKM34kEZAcE1OX4MfiwjkE=
+github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
 github.com/syndtr/gocapability v0.0.0-20170704070218-db04d3cc01c8/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
 github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
 github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
 github.com/tchap/go-patricia v2.2.6+incompatible/go.mod h1:bmLyhP68RS6kStMGxByiQ23RP/odRBOTVjwp2cDyi6I=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
+github.com/twitchyliquid64/golang-asm v0.0.0-20190126203739-365674df15fc/go.mod h1:NoCfSFWosfqMqmmD7hApkirIK9ozpHjxRnRxs1l413A=
 github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
+github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
+github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
 github.com/urfave/cli v0.0.0-20171014202726-7bc6a0acffa5/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
 github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
 github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
@@ -628,6 +695,7 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec
 github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43/go.mod h1:aX5oPXxHm3bOH+xeAttToC8pqch2ScQN/JoXYupl6xs=
 github.com/yvasiyarov/gorelic v0.0.0-20141212073537-a9bba5b9ab50/go.mod h1:NUSPSUX/bi6SeDMUh6brw0nXpxHnc96TguQh0+r/ssA=
 github.com/yvasiyarov/newrelic_platform_go v0.0.0-20140908184405-b21fdbd4370f/go.mod h1:GlGEuHIJweS1mbCqG+7vt2nvWLzLLnRHbXz5JKd/Qbg=
+go.coder.com/go-tools v0.0.0-20190317003359-0c6a35b74a16/go.mod h1:iKV5yK9t+J5nG9O3uF6KYdPEz3dyfMyB15MN1rbQ8Qw=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
@@ -642,6 +710,7 @@ go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 golang.org/x/crypto v0.0.0-20171113213409-9f005a07e0d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20180426230345-b49d69b5da94/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181009213950-7c1a557ab941/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
@@ -691,6 +760,7 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181102091132-c10e9556a7bc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -715,6 +785,7 @@ golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
+golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201006153459-a7d1128ccaa0/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@@ -736,12 +807,16 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190306220234-b354f8bf4d9e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -753,13 +828,17 @@ golang.org/x/sys v0.0.0-20190602015325-4c4f7f33c9ed/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190606203320-7fc4e5ec1444/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190618155005-516e3c20635f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190801041406-cbf593c0f2f3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190812073006-9eafafc0a87e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190927073244-c990c680b611/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -771,6 +850,7 @@ golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200120151820-655fe14d7479/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200124204421-9fbb57f87de9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -780,6 +860,7 @@ golang.org/x/sys v0.0.0-20200217220822-9197077df867/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -794,6 +875,7 @@ golang.org/x/sys v0.0.0-20201117170446-d9b008d0a637/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201202213521-69691e467435/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210113181707-4bcb84eeeb78/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210324051608-47abb6519492/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -834,6 +916,7 @@ golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgw
 golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20190920225731-5eefd052ad72/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -927,8 +1010,6 @@ google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/l
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
-gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc h1:2gGKlE2+asNV9m7xrywl36YYNnBG5ZQ0r/BOOxqPpmk=
-gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc/go.mod h1:m7x9LTH6d71AHyAX77c9yqWCCa3UKHcVEj9y7hAtKDk=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20141024133853-64131543e789/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -938,9 +1019,9 @@ gopkg.in/cheggaaa/pb.v1 v1.0.25/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qS
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2/go.mod h1:Xk6kEKp8OKb+X14hQBKWaSkCsqBpgog8nAV2xsGOxlo=
-gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df h1:n7WqCuqOuCbNr617RXOY0AWRXxgwEyPp2z+p0+hgMuE=
-gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df/go.mod h1:LRQQ+SO6ZHR7tOkpBDuZnXENFzX8qRjMDMyPD6BRkCw=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/ini.v1 v1.55.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/ini.v1 v1.65.0 h1:B2//IEITFk89S+Nl2tozBeqUvFEpUAY6daarSlrx8jU=
 gopkg.in/ini.v1 v1.65.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
@@ -949,6 +1030,7 @@ gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
 gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
 gopkg.in/square/go-jose.v2 v2.3.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
 gopkg.in/square/go-jose.v2 v2.5.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -957,11 +1039,15 @@ gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
 gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gosrc.io/xmpp v0.5.1/go.mod h1:L3NFMqYOxyLz3JGmgFyWf7r9htE91zVGiK40oW4RwdY=
+gotest.tools v2.1.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
 gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
 gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
+gotest.tools/gotestsum v0.3.5/go.mod h1:Mnf3e5FUzXbkCfynWBGOwLssY7gTQgCHObK9tMpAriY=
 gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
 gotest.tools/v3 v3.0.3 h1:4AuOwCGf4lLR9u3YOe2awrHygurzhO/HeQ6laiA6Sx0=
 gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8=
@@ -996,6 +1082,9 @@ k8s.io/klog/v2 v2.4.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y=
 k8s.io/kube-openapi v0.0.0-20201113171705-d219536bb9fd/go.mod h1:WOJ3KddDSol4tAGcJo0Tvi+dK12EcqSLqcWsryKMpfM=
 k8s.io/kubernetes v1.13.0/go.mod h1:ocZa8+6APFNC2tX1DZASIbocyYT5jHzqFVsY5aoB7Jk=
 k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
+mvdan.cc/sh v2.6.4+incompatible/go.mod h1:IeeQbZq+x2SUGBensq/jge5lLQbS3XT2ktyp3wrt4x8=
+nhooyr.io/websocket v1.6.5/go.mod h1:F259lAzPRAH0htX2y3ehpJe09ih1aSHN7udWki1defY=
+nhooyr.io/websocket v1.8.6/go.mod h1:B70DZP8IakI65RVQ51MsWP/8jndNma26DVA/nFSCgW0=
 rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
 rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
 rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
--- a/test/cli/run.sh
+++ b/test/cli/run.sh
@@ -7,6 +7,9 @@ cd $(dirname $0)
 docker network create test_network
 docker volume create backup_data
 docker volume create app_data
+# This volume is created to test whether empty directories are handled
+# correctly. It is not supposed to hold any data.
+docker volume create empty_data

 docker run -d \
  --name minio \
@@ -31,6 +34,7 @@ sleep 10
 docker run --rm \
  --network test_network \
  -v app_data:/backup/app_data \
+  -v empty_data:/backup/empty_data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --env AWS_ACCESS_KEY_ID=test \
  --env AWS_SECRET_ACCESS_KEY=GMusLtUmILge2by+z890kQ \
@@ -40,14 +44,16 @@ docker run --rm \
  --env BACKUP_FILENAME=test.tar.gz \
  --env "BACKUP_FROM_SNAPSHOT=true" \
  --entrypoint backup \
-  offen/docker-volume-backup:$TEST_VERSION
+  offen/docker-volume-backup:${TEST_VERSION:-canary}

 docker run --rm -it \
  -v backup_data:/data alpine \
-  ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db'
+  ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db && test -d /backup/empty_data'

-echo "[TEST:PASS] Found relevant files in untared backup."
+echo "[TEST:PASS] Found relevant files in untared remote backup."

+# This test does not stop containers during backup. This is happening on
+# purpose in order to cover this setup as well.
 if [ "$(docker ps -q | wc -l)" != "2" ]; then
  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
  docker ps
--- a/test/commands/.gitignore
+++ b/test/commands/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/commands/docker-compose.yml
+++ b/test/commands/docker-compose.yml
@@ -0,0 +1,36 @@
+version: '3.8'
+
+services:
+  database:
+    image: mariadb:10.7
+    deploy:
+      restart_policy:
+        condition: on-failure
+    environment:
+      MARIADB_ROOT_PASSWORD: test
+      MARIADB_DATABASE: backup
+    labels:
+      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -ptest --all-databases > /tmp/volume/dump.sql'
+      - docker-volume-backup.exec-post=/bin/sh -c 'echo "post" > /tmp/volume/post.txt'
+      - docker-volume-backup.exec-label=test
+    volumes:
+      - app_data:/tmp/volume
+
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    deploy:
+      restart_policy:
+        condition: on-failure
+    environment:
+      BACKUP_FILENAME: test.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      EXEC_LABEL: test
+      EXEC_FORWARD_OUTPUT: "true"
+    volumes:
+      - archive:/archive
+      - app_data:/backup/data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+volumes:
+  app_data:
+  archive:
--- a/test/commands/run.sh
+++ b/test/commands/run.sh
@@ -0,0 +1,62 @@
+#!/bin/sh
+
+set -e
+
+cd $(dirname $0)
+
+
+docker-compose up -d
+sleep 30 # mariadb likes to take a bit before responding
+
+docker-compose exec backup backup
+sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' commands_archive) ./local
+
+tar -xvf ./local/test.tar.gz
+if [ ! -f ./backup/data/dump.sql ]; then
+  echo "[TEST:FAIL] Could not find file written by pre command."
+  exit 1
+fi
+echo "[TEST:PASS] Found expected file."
+
+if [ -f ./backup/data/post.txt ]; then
+  echo "[TEST:FAIL] File created in post command was present in backup."
+  exit 1
+fi
+echo "[TEST:PASS] Did not find unexpected file."
+
+docker-compose down --volumes
+sudo rm -rf ./local
+
+
+echo "[TEST:INFO] Running commands test in swarm mode next."
+
+docker swarm init
+
+docker stack deploy --compose-file=docker-compose.yml test_stack
+
+while [ -z $(docker ps -q -f name=backup) ]; do
+  echo "[TEST:INFO] Backup container not ready yet. Retrying."
+  sleep 1
+done
+
+sleep 20
+
+docker exec $(docker ps -q -f name=backup) backup
+
+sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' test_stack_archive) ./local
+
+tar -xvf ./local/test.tar.gz
+if [ ! -f ./backup/data/dump.sql ]; then
+  echo "[TEST:FAIL] Could not find file written by pre command."
+  exit 1
+fi
+echo "[TEST:PASS] Found expected file."
+
+if [ -f ./backup/data/post.txt ]; then
+  echo "[TEST:FAIL] File created in post command was present in backup."
+  exit 1
+fi
+echo "[TEST:PASS] Did not find unexpected file."
+
+docker stack rm test_stack
+docker swarm leave --force
--- a/test/compose/docker-compose.yml
+++ b/test/compose/docker-compose.yml
@@ -10,12 +10,23 @@ services:
      MINIO_SECRET_KEY: GMusLtUmILge2by+z890kQ
    entrypoint: /bin/ash -c 'mkdir -p /data/backup && minio server /data'
    volumes:
-      - backup_data:/data
+      - minio_backup_data:/data

-  backup: &default_backup_service
-    image: offen/docker-volume-backup:${TEST_VERSION}
+  webdav:
+    image: bytemark/webdav:2.4
+    environment:
+      AUTH_TYPE: Digest
+      USERNAME: test
+      PASSWORD: test
+    volumes:
+      - webdav_backup_data:/var/lib/dav
+
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    hostname: hostnametoken
    depends_on:
      - minio
+      - webdav
    restart: always
    environment:
      AWS_ACCESS_KEY_ID: test
@@ -23,13 +34,19 @@ services:
      AWS_ENDPOINT: minio:9000
      AWS_ENDPOINT_PROTO: http
      AWS_S3_BUCKET_NAME: backup
-      BACKUP_FILENAME: test.tar.gz
-      BACKUP_LATEST_SYMLINK: test.latest.tar.gz.gpg
+      BACKUP_FILENAME_EXPAND: 'true'
+      BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
+      BACKUP_LATEST_SYMLINK: test-$$HOSTNAME.latest.tar.gz.gpg
      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
      BACKUP_PRUNING_LEEWAY: 5s
      BACKUP_PRUNING_PREFIX: test
      GPG_PASSPHRASE: 1234secret
+      WEBDAV_URL: http://webdav/
+      WEBDAV_URL_INSECURE: 'true'
+      WEBDAV_PATH: /my/new/path/
+      WEBDAV_USERNAME: test
+      WEBDAV_PASSWORD: test
    volumes:
      - ./local:/archive
      - app_data:/backup/app_data:ro
@@ -43,5 +60,6 @@ services:
      - app_data:/var/opt/offen

 volumes:
-  backup_data:
+  minio_backup_data:
+  webdav_backup_data:
  app_data:
--- a/test/compose/run.sh
+++ b/test/compose/run.sh
@@ -9,31 +9,39 @@ mkdir -p local
 docker-compose up -d
 sleep 5

+# A symlink for a known file in the volume is created so the test can check
+# whether symlinks are preserved on backup.
 docker-compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
 docker-compose exec backup backup

-docker run --rm -it \
-  -v compose_backup_data:/data alpine \
-  ash -c 'apk add gnupg && echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /data/backup/test.tar.gz.gpg > /tmp/test.tar.gz && tar -xf /tmp/test.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
-
-echo "[TEST:PASS] Found relevant files in untared remote backup."
-
-test -L ./local/test.latest.tar.gz.gpg
-echo 1234secret | gpg -d --yes --passphrase-fd 0 ./local/test.tar.gz.gpg > ./local/decrypted.tar.gz
-tar -xf ./local/decrypted.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db
-rm ./local/decrypted.tar.gz
-test -L /tmp/backup/app_data/db.link
-
-echo "[TEST:PASS] Found relevant files in untared local backup."
-
-if [ "$(docker-compose ps -q | wc -l)" != "3" ]; then
+sleep 5
+if [ "$(docker-compose ps -q | wc -l)" != "4" ]; then
  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
  docker-compose ps
  exit 1
 fi
-
 echo "[TEST:PASS] All containers running post backup."

+
+docker run --rm -it \
+  -v compose_minio_backup_data:/minio_data \
+  -v compose_webdav_backup_data:/webdav_data alpine \
+  ash -c 'apk add gnupg && \
+          echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /minio_data/backup/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xvf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db && \
+          echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /webdav_data/data/my/new/path/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xvf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
+
+echo "[TEST:PASS] Found relevant files in decrypted and untared remote backups."
+
+echo 1234secret | gpg -d --pinentry-mode loopback --yes --passphrase-fd 0 ./local/test-hostnametoken.tar.gz.gpg > ./local/decrypted.tar.gz
+tar -xf ./local/decrypted.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db
+rm ./local/decrypted.tar.gz
+test -L /tmp/backup/app_data/db.link
+
+echo "[TEST:PASS] Found relevant files in decrypted and untared local backup."
+
+test -L ./local/test-hostnametoken.latest.tar.gz.gpg
+echo "[TEST:PASS] Found symlink to latest version in local backup."
+
 # The second part of this test checks if backups get deleted when the retention
 # is set to 0 days (which it should not as it would mean all backups get deleted)
 # TODO: find out if we can test actual deletion without having to wait for a day
@@ -43,16 +51,18 @@ sleep 5
 docker-compose exec backup backup

 docker run --rm -it \
-  -v compose_backup_data:/data alpine \
-  ash -c '[ $(find /data/backup/ -type f | wc -l) = "1" ]'
+  -v compose_minio_backup_data:/minio_data \
+  -v compose_webdav_backup_data:/webdav_data alpine \
+  ash -c '[ $(find /minio_data/backup/ -type f | wc -l) = "1" ] && \
+          [ $(find /webdav_data/data/my/new/path/ -type f | wc -l) = "1" ]'

 echo "[TEST:PASS] Remote backups have not been deleted."

 if [ "$(find ./local -type f | wc -l)" != "1" ]; then
  echo "[TEST:FAIL] Backups should not have been deleted, instead seen:"
  find ./local -type f
+  exit 1
 fi
-
 echo "[TEST:PASS] Local backups have not been deleted."

 docker-compose down --volumes
--- a/test/confd/.gitignore
+++ b/test/confd/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/confd/01backup.env
+++ b/test/confd/01backup.env
@@ -0,0 +1,2 @@
+BACKUP_FILENAME="conf.tar.gz"
+BACKUP_CRON_EXPRESSION="*/1 * * * *"
--- a/test/confd/02backup.env
+++ b/test/confd/02backup.env
@@ -0,0 +1,2 @@
+BACKUP_FILENAME="other.tar.gz"
+BACKUP_CRON_EXPRESSION="*/1 * * * *"
--- a/test/confd/03never.env
+++ b/test/confd/03never.env
@@ -0,0 +1,2 @@
+BACKUP_FILENAME="never.tar.gz"
+BACKUP_CRON_EXPRESSION="0 0 5 31 2 ?"
--- a/test/confd/docker-compose.yml
+++ b/test/confd/docker-compose.yml
@@ -0,0 +1,23 @@
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    restart: always
+    volumes:
+      - ./local:/archive
+      - app_data:/backup/app_data:ro
+      - ./01backup.env:/etc/dockervolumebackup/conf.d/01backup.env
+      - ./02backup.env:/etc/dockervolumebackup/conf.d/02backup.env
+      - ./03never.env:/etc/dockervolumebackup/conf.d/03never.env
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  app_data:
--- a/test/confd/run.sh
+++ b/test/confd/run.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+set -e
+
+cd $(dirname $0)
+
+mkdir -p local
+
+docker-compose up -d
+
+# sleep until a backup is guaranteed to have happened on the 1 minute schedule
+sleep 100
+
+docker-compose down --volumes
+
+if [ ! -f ./local/conf.tar.gz ]; then
+  echo "[TEST:FAIL] Config from file was not used."
+  exit 1
+fi
+echo "[TEST:PASS] Config from file was used."
+
+if [ ! -f ./local/other.tar.gz ]; then
+  echo "[TEST:FAIL] Run on same schedule did not succeed."
+  exit 1
+fi
+echo "[TEST:PASS] Run on same schedule succeeded."
+
+if [ -f ./local/never.tar.gz ]; then
+  echo "[TEST:FAIL] Unexpected file was found."
+  exit 1
+fi
+echo "[TEST:PASS] Unexpected cron did not run."
--- a/test/ignore/.gitignore
+++ b/test/ignore/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/ignore/docker-compose.yml
+++ b/test/ignore/docker-compose.yml
@@ -0,0 +1,15 @@
+version: '3.8'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    deploy:
+      restart_policy:
+        condition: on-failure
+    environment:
+      BACKUP_FILENAME: test.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_EXCLUDE_REGEXP: '\.(me|you)$$'
+    volumes:
+      - ./local:/archive
+      - ./sources:/backup/data:ro
--- a/test/ignore/run.sh
+++ b/test/ignore/run.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+set -e
+
+cd $(dirname $0)
+mkdir -p local
+
+docker-compose up -d
+sleep 5
+docker-compose exec backup backup
+
+docker-compose down --volumes
+
+out=$(mktemp -d)
+sudo tar --same-owner -xvf ./local/test.tar.gz -C "$out"
+
+if [ ! -f "$out/backup/data/me.txt" ]; then
+  echo "[TEST:FAIL] Expected file was not found."
+  exit 1
+fi
+echo "[TEST:PASS] Expected file was found."
+
+if [ -f "$out/backup/data/skip.me" ]; then
+  echo "[TEST:FAIL] Ignored file was found."
+  exit 1
+fi
+echo "[TEST:PASS] Ignored file was not found."
--- a/test/ignore/sources/me.txt
+++ b/test/ignore/sources/me.txt
--- a/test/ignore/sources/skip.me
+++ b/test/ignore/sources/skip.me
--- a/test/notifications/.gitignore
+++ b/test/notifications/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/notifications/docker-compose.yml
+++ b/test/notifications/docker-compose.yml
@@ -0,0 +1,36 @@
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    restart: always
+    environment:
+      BACKUP_FILENAME: test.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_PRUNING_PREFIX: test
+      NOTIFICATION_LEVEL: info
+      NOTIFICATION_URLS: ${NOTIFICATION_URLS}
+    volumes:
+      - ./local:/archive
+      - app_data:/backup/app_data:ro
+      - ./notifications.tmpl:/etc/dockervolumebackup/notifications.d/notifications.tmpl
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+  gotify:
+    image: gotify/server
+    ports:
+      - 8080:80
+    environment:
+      - GOTIFY_DEFAULTUSER_PASS=custom
+    volumes:
+      - gotify_data:/app/data
+
+volumes:
+  app_data:
+  gotify_data:
--- a/test/notifications/notifications.tmpl
+++ b/test/notifications/notifications.tmpl
@@ -0,0 +1,7 @@
+{{ define "title_success" -}}
+Successful test run, yay!
+{{- end }}
+
+{{ define "body_success" -}}
+Backing up {{ .Stats.BackupFile.FullPath }} succeeded.
+{{- end }}
--- a/test/notifications/run.sh
+++ b/test/notifications/run.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+
+set -e
+
+cd $(dirname $0)
+
+mkdir -p local
+
+docker-compose up -d
+sleep 5
+
+GOTIFY_TOKEN=$(curl -sSLX POST -H 'Content-Type: application/json' -d '{"name":"test"}' http://admin:custom@localhost:8080/application | jq -r '.token')
+echo "[TEST:INFO] Set up Gotify application using token $GOTIFY_TOKEN"
+
+docker-compose exec backup backup
+
+NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
+if [ "$NUM_MESSAGES" != 0 ]; then
+  echo "[TEST:FAIL] Expected no notifications to be sent when not configured"
+  exit 1
+fi
+echo "[TEST:PASS] No notifications were sent when not configured."
+
+docker-compose down
+
+NOTIFICATION_URLS="gotify://gotify/${GOTIFY_TOKEN}?disableTLS=true" docker-compose up -d
+
+docker-compose exec backup backup
+
+NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
+if [ "$NUM_MESSAGES" != 1 ]; then
+  echo "[TEST:FAIL] Expected one notifications to be sent when configured"
+  exit 1
+fi
+echo "[TEST:PASS] Correct number of notifications were sent when configured."
+
+MESSAGE_TITLE=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages[0].title')
+MESSAGE_BODY=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages[0].message')
+
+if [ "$MESSAGE_TITLE" != "Successful test run, yay!" ]; then
+  echo "[TEST:FAIL] Unexpected notification title $MESSAGE_TITLE"
+  exit 1
+fi
+echo "[TEST:PASS] Custom notification title was used."
+
+if [ "$MESSAGE_BODY" != "Backing up /tmp/test.tar.gz succeeded." ]; then
+  echo "[TEST:FAIL] Unexpected notification body $MESSAGE_BODY"
+  exit 1
+fi
+echo "[TEST:PASS] Custom notification body was used."
+
+docker-compose down --volumes
--- a/test/ownership/.gitignore
+++ b/test/ownership/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/ownership/docker-compose.yml
+++ b/test/ownership/docker-compose.yml
@@ -0,0 +1,27 @@
+version: '3'
+
+services:
+  db:
+    image: postgres:14-alpine
+    restart: unless-stopped
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+    environment:
+      - POSTGRES_PASSWORD=1FHJMSwt0yhIN1zS7I4DilGUhThBKq0x
+      - POSTGRES_USER=test
+      - POSTGRES_DB=test
+
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION}
+    restart: always
+    environment:
+      BACKUP_FILENAME: backup.tar.gz
+    volumes:
+      - postgres_data:/backup/postgres:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - ./local:/archive
+
+volumes:
+  postgres_data:
--- a/test/ownership/run.sh
+++ b/test/ownership/run.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+# This test refers to https://github.com/offen/docker-volume-backup/issues/71
+
+set -e
+
+cd $(dirname $0)
+
+mkdir -p local
+
+docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+sudo tar --same-owner -xvf ./local/backup.tar.gz -C /tmp
+
+sudo find /tmp/backup/postgres > /dev/null
+echo "[TEST:PASS] Backup contains files at expected location"
+
+for file in $(sudo find /tmp/backup/postgres); do
+  if [ "$(sudo stat -c '%u:%g' $file)" != "70:70" ]; then
+    echo "[TEST:FAIL] Unexpected file ownership for $file: $(sudo stat -c '%u:%g' $file)"
+    exit 1
+  fi
+done
+echo "[TEST:PASS] All files and directories in backup preserved their ownership."
+
+docker-compose down --volumes
--- a/test/swarm/docker-compose.yml
+++ b/test/swarm/docker-compose.yml
@@ -18,8 +18,8 @@ services:
    volumes:
      - backup_data:/data

-  backup: &default_backup_service
-    image: offen/docker-volume-backup:${TEST_VERSION}
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
    depends_on:
      - minio
    deploy:
@@ -43,13 +43,15 @@ services:
    image: offen/offen:latest
    labels:
      - docker-volume-backup.stop-during-backup=true
+    healthcheck:
+      disable: true
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure

  pg:
-    image: postgres:12.2-alpine
+    image: postgres:14-alpine
    environment:
      POSTGRES_PASSWORD: example
    labels:
--- a/test/swarm/run.sh
+++ b/test/swarm/run.sh
@@ -23,14 +23,13 @@ docker run --rm -it \

 echo "[TEST:PASS] Found relevant files in untared backup."

+sleep 5
 if [ "$(docker ps -q | wc -l)" != "5" ]; then
  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
  docker ps -a
  exit 1
 fi
-
 echo "[TEST:PASS] All containers running post backup."

 docker stack rm test_stack
-
 docker swarm leave --force
Author	SHA1	Message	Date
Frederik Ring	b52b271bac	Allow for the exclusion of files from backups (#100 ) * Hoist walking of files so it can be used for features other than archive creation * Add option to ignore files from backup using glob patterns * Use Regexp instead of glob for exclusion * Ignore artifacts * Add teardown to test * Allow single Re for filtering only * Add documentation * Use MatchString on re, add bad input to message in case of error	2022-05-08 11:20:38 +02:00
Frederik Ring	cac5777e79	Add documentation on using multiple configs for complex retention schemes	2022-04-20 18:01:41 +02:00
Frederik Ring	94a1edc4ad	Allow disabling of certificate verification for WebDAV (#98 )	2022-04-20 14:16:59 +02:00
Frederik Ring	a654097e59	Update webdav client library (#97 )	2022-04-20 10:56:26 +02:00
Frederik Ring	1b1fc4856c	List objects recursively when selecting candidates from S3 (#92 )	2022-04-15 11:05:52 +02:00
Frederik Ring	e81c34b8fc	Consider S3 Path when selecting candidates for pruning (#91 )	2022-04-13 17:09:37 +02:00
Simon Dünhöft	9c23767fce	Fixed wrong env name for S3 bucket in README (#89 ) The README was using `AWS_BUCKET_NAME` instead of `AWS_S3_BUCKET_NAME` in the recipes. This resulted in no data being uploaded to S3.	2022-04-12 19:38:15 +02:00
Frederik Ring	51af8c3c77	Deprecate BACKUP_FROM_SNAPSHOT (#81 )	2022-03-25 18:28:58 +01:00
Frederik Ring	1ea0b51b23	Tag releases with major version too (#82 )	2022-03-25 18:27:00 +01:00
Frederik Ring	da8c63f755	Support identical cron schedule (#87 ) * Retry on lock being unavailable * Refactor locking to return plain error * Collect LockedTime in stats * Add test case * Add documentation for LOCK_TIMEOUT * Log in case lock needs to be awaited * Release resources created for awaiting lock	2022-03-25 18:26:34 +01:00
Frederik Ring	9bc8db0f7c	Build using Go 1.18 (#86 )	2022-03-17 11:22:41 +01:00
Frederik Ring	508bc07b4f	Disable healthcheck in swarm test (#85 )	2022-03-17 11:13:07 +01:00
Frederik Ring	b8f71b04a1	Use errgroup for running commands in parallel (#83 )	2022-03-10 11:09:39 +01:00
Frederik Ring	5f3832d621	Consider prefix rules when pruning WebDAV storages (#79 )	2022-03-05 13:33:15 +01:00
Frederik Ring	4b1127b8c4	Document BACKUP_SOURCES	2022-03-04 19:57:32 +01:00
Frederik Ring	ae50a3ac4f	Add attribution to code taken from moby repository	2022-03-04 16:40:34 +01:00
Frederik Ring	bad22eee93	Fix syntax highlighting in container	2022-03-04 14:08:31 +01:00
Frederik Ring	c9ebb9e14e	Allow multiple schedules in the same container (#78 ) * Allow mounting of config directory for multiple schedules * Add docs for conf.d feature * Fix behavior on multiple files * Define default case first in entrypoint script	2022-03-04 13:51:26 +01:00
Frederik Ring	6e1b8553e6	Remove superfluous --update flag from cert install	2022-02-26 16:45:29 +01:00
Frederik Ring	5ec2b2c3ff	Install ca-certs with --no-cache to reduce image size	2022-02-25 08:54:07 +01:00
Rajan Patel	3bbeba5b83	update custom docker host documentation for pre/post commands (#77 )	2022-02-24 05:31:36 +01:00
Frederik Ring	9155b4d130	Add missing print directive, fix go.mod	2022-02-23 10:12:57 +01:00
Kazi	2a17e84ab6	snapshot-style restore example (#76 ) * snapshot-style restore example * manual backup recommendation	2022-02-23 07:58:09 +01:00
Rajan Patel	00f2359461	Add DOCKER_HOST documentation (#74 ) * add DOCKER_HOST documentation * add which endpoints are required for DOCKER_HOST * Update README.md Co-authored-by: Frederik Ring <frederik.ring@gmail.com>	2022-02-22 08:00:26 +01:00
Frederik Ring	0504a92a1f	Add option to run pre/post commands for any container (#73 ) * Add option to run pre commands on arbitrary container * Correctly handle quoted args in commands * Provide defaults for test version arg * Allow filtering of target containers * Add documentation on exec commands * Use mysqldump in exec test * Add mysqldump section to recipes * Also run commands test in swarm mode * Use name instead of id * Add syntax highlighting * Add missing license headers	2022-02-22 07:53:33 +01:00
Frederik Ring	3ded77448c	Do not skip directories when creating tar archive (#72 ) * Update targz library to include potential ownership fix * Move archive logic to main repo * Remove assertions for debugging * Use relative path in assertion * Strip local part from archive location * Log when extracting in tests * Fix trimming of prfix * Add license info to archive.go file * Undo change in test assertion * Add test checking for preserved file ownership * use same postgres version in tests * Wrap errors when archiving, handle deletion at script layer	2022-02-22 07:49:24 +01:00
Frederik Ring	58b42b9036	Supporting proxied Docker APIs through DOCKER_HOST (#70 )	2022-02-18 09:08:21 +01:00
Frederik Ring	180438f1fc	Update ubuntu image used for running integration tests (#67 )	2022-02-15 21:05:03 +01:00
Mauro Molin	30265c14ba	Fixed TookTime (#66 )	2022-02-14 17:32:05 +01:00
Frederik Ring	a57e93d01e	Split source into multiple files, deduplicate pruning logic, do not parse templates when notifications are not used (#63 ) * Split code into multiple files * Deduplicate logic for pruning backups against different storages * Only parse templates when notifications are enabled * Use better description	2022-02-13 10:52:19 +01:00
Frederik Ring	3e17d1b123	Ensure end time is recorded for unsuccessful runs too (#62 ) * Ensure end time is also recorded for unsuccessful runs * Clean up integration tests	2022-02-13 09:41:36 +01:00
Frederik Ring	0e248010a8	Add note about how notifications can be customized to config reference	2022-02-12 20:34:51 +01:00
Frederik Ring	e6af6efd8a	Add test setup for notification feature (#61 ) * Add test case for notification feature * Fix template data * bash is needed to interpret test * Do not use bashisms in test * Only print FullPath * Fix assertion	2022-02-12 20:28:38 +01:00
Frederik Ring	34d04211eb	Update README TOC	2022-02-11 20:06:23 +01:00
Mauro Molin	8dfdd14527	Added custom notification messages using text/template (#60 ) * Added custom notification messages using text/template * Change notification template path and removed automatic newline trim * Added stats and changed structure of template params * Stat file hotfix * Embedded and fixed default notification templates Fix * Changed Output to LogOutput * Changed stats integer to unsigned * Bytes formatting in template func fix * Changed Archives to Storages * Removed unecessary sleep for pruning leeway * Set EndTime after pruning is completed * Added custom notifications documentation * Added 5s sleep in swarm test * Fixed documentation * Dockerfile copies all files in cmd/backup	2022-02-11 20:05:16 +01:00
Frederik Ring	3bb99a7117	Update package targz	2022-02-08 15:12:46 +01:00
Fridgemagnet	ddc34be55d	Updated README.md "Restoring..." section example (#56 ) Edited the "Restoring a volume from a backup" section example to be able to better differentiate between the names of the temporary restore container and the name of the mounted restore volume. New example container name: temp_restore_container The restore volume name remains the same: backup_restore Also changed "one-off container" to "once-off container".	2022-02-04 11:52:59 +01:00
Joshua Noble	cb9b4bfcff	Add support for Filebase (#54 )	2022-02-02 17:22:42 +01:00
Frederik Ring	62bd2f4a5a	Update base docker image to alpine 3.15 (#53 )	2022-01-27 14:40:56 +01:00
Frederik Ring	6fe629ce87	Allow path to be set for bucket storage (#52 )	2022-01-25 21:16:16 +01:00
Frederik Ring	1db896f7cf	Tweak README, improve client naming, tidy go.mod file	2022-01-22 13:35:13 +01:00
Kaerbr	6ded00aa06	Support Nextcloud / WebDav (#48 ) * add studio-b12/gowebdav to be able to upload to webdav server * make sure all env variables are present for webdav upload * implement file upload to WebDav server directory defaults to the base directory * docs: add the new feature to the documentation * if no WebDav env variable are given throw no error * docs: use more elegant english :D Co-authored-by: Frederik Ring <frederik.ring@gmail.com> * docs: use official spelling of "WebDAV" * perf: golang likes to return early instead of having an else block * use WEBDAV_PATH instead of WEBDAV_DIRECTORY * use split_words for more convenience like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support * simplify * feat: add pruning of files in WebDAV remote Based on / Inspired by the minio/S3 implementation of pruning remote files. * remove logging from the development * test: first try implementing tests Sandly I have to use the remote pipeline -- local wont work for me. * test: adapt used volume names * test: specify image only once! * test: minio AND webdav data should be present * test: backups on WebDAV remote are laying in the root directory * test: the webdav server stores date in /var/lib/dav * trying with data subfolder * test: 1 container was added so the number raised from 3 to 4 * webdav subfolder is "data" not "backup" * fix: password AND username must be defined not password OR username * improve logging * feat: if the given path on the server isnt preset it will be created * test: add creation of new folder for webdav to tests Co-authored-by: Frederik Ring <frederik.ring@gmail.com>	2022-01-22 13:29:21 +01:00
Hendrik Niefeld	6b79f1914b	Update README.md	2022-01-06 16:09:33 +01:00
Hendrik Niefeld	40ff2e00c9	Update README.md	2022-01-06 16:07:00 +01:00
Frederik Ring	760cc9cebc	Add issue template	2021-12-29 13:10:43 +01:00
Frederik Ring	1f9582df51	Fix handling of empty directories (#44 ) * Add test checking whether empty directories are included in backups * Update targz library to include fix	2021-12-29 10:10:12 +01:00
Frederik Ring	32575c831e	Also expand env vars in pruning prefix if configured	2021-12-23 09:22:56 +01:00
Frederik Ring	c062710ce8	Allow for env substitution in backup filename (#39 )	2021-12-22 14:39:46 +01:00
Frederik Ring	3a7dfe8e60	Add note about double quoting issue in older compose versions	2021-12-18 13:24:14 +01:00
Frederik Ring	9ec33510e7	Extend docs on notifications	2021-12-18 10:31:12 +01:00
Frederik Ring	4207146fb6	Refactor calling of hooks on exit	2021-12-18 10:31:12 +01:00
Frederik Ring	1f727f698f	Run hooks in order of severity	2021-12-18 10:31:12 +01:00
Frederik Ring	88c90a206c	Use int comparison for checking hooks	2021-12-18 10:31:12 +01:00
Frederik Ring	8bad0656b3	Enable notifications on multiple levels	2021-12-18 10:31:12 +01:00
Frederik Ring	08d78a0bd6	allow sending notifications to multiple channels	2021-12-18 10:31:12 +01:00