Fine grained labels (#115 )

* Refactor label command mechanism to be more flexible * Run all steps wrapped in labeled commands * Rename methods to be in line with lifecycle * Deprecate exec-pre and exec-post labels * Add documentation * Use type alias for lifecycle phases * Fix bad imports * Fix command lookup for deprecated labels * Use more generic naming for lifecycle phase * Fail on erroneous post command * Update documentation
Add further docs on backup selection when using conf.d
2025-12-05 17:18:02 +01:00 · 2022-07-10 10:36:56 +02:00 · 2022-07-08 14:05:45 +02:00 · 2022-07-07 22:22:21 +02:00 · 2022-07-06 13:18:12 +02:00 · 2022-07-06 13:16:43 +02:00
37 changed files with 920 additions and 884 deletions
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -5,6 +5,7 @@ jobs:
    machine:
      image: ubuntu-2004:202201-02
    working_directory: ~/docker-volume-backup
+    resource_class: large
    steps:
      - checkout
      - run:
@@ -29,6 +30,7 @@ jobs:
          DOCKER_BUILDKIT: '1'
          DOCKER_CLI_EXPERIMENTAL: enabled
    working_directory: ~/docker-volume-backup
+    resource_class: large
    steps:
      - checkout
      - setup_remote_docker:
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@
 Backup Docker volumes locally or to any S3 compatible storage.

 The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
-It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3 or WebDAV compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
+It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.

 <!-- MarkdownTOC -->

@@ -20,7 +20,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
  - [Automatically pruning old backups](#automatically-pruning-old-backups)
  - [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs)
  - [Customize notifications](#customize-notifications)
-  - [Run custom commands before / after backup](#run-custom-commands-before--after-backup)
+  - [Run custom commands during the backup lifecycle](#run-custom-commands-during-the-backup-lifecycle)
  - [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg)
  - [Restoring a volume from a backup](#restoring-a-volume-from-a-backup)
  - [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in)
@@ -28,14 +28,17 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
  - [Manually triggering a backup](#manually-triggering-a-backup)
  - [Update deprecated email configuration](#update-deprecated-email-configuration)
  - [Replace deprecated `BACKUP_FROM_SNAPSHOT` usage](#replace-deprecated-backup_from_snapshot-usage)
+  - [Replace deprecated `exec-pre` and `exec-post` labels](#replace-deprecated-exec-pre-and-exec-post-labels)
  - [Using a custom Docker host](#using-a-custom-docker-host)
  - [Run multiple backup schedules in the same container](#run-multiple-backup-schedules-in-the-same-container)
  - [Define different retention schedules](#define-different-retention-schedules)
+  - [Use special characters in notification URLs](#use-special-characters-in-notification-urls)
 - [Recipes](#recipes)
  - [Backing up to AWS S3](#backing-up-to-aws-s3)
  - [Backing up to Filebase](#backing-up-to-filebase)
  - [Backing up to MinIO](#backing-up-to-minio)
  - [Backing up to WebDAV](#backing-up-to-webdav)
+  - [Backing up to SSH](#backing-up-to-ssh)
  - [Backing up locally](#backing-up-locally)
  - [Backing up to AWS S3 as well as locally](#backing-up-to-aws-s3-as-well-as-locally)
  - [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
@@ -220,6 +223,11 @@ You can populate below template according to your requirements and use it as you

 # AWS_ENDPOINT_INSECURE="true"

+# Setting this variable will change the S3 storage class header.
+# Defaults to "STANDARD", you can set this value according to your needs.
+
+# AWS_STORAGE_CLASS="GLACIER"
+
 # You can also backup files to any WebDAV server:

 # The URL of the remote WebDAV server
@@ -245,6 +253,40 @@ You can populate below template according to your requirements and use it as you

 # WEBDAV_URL_INSECURE="true"

+# You can also backup files to any SSH server:
+
+# The URL of the remote SSH server
+
+# SSH_HOST_NAME="server.local"
+
+# The port of the remote SSH server
+# Optional variable default value is `22`
+
+# SSH_PORT=2222
+
+# The Directory to place the backups to on the SSH server.
+
+# SSH_REMOTE_PATH="/my/directory/"
+
+# The username for the SSH server
+
+# SSH_USER="user"
+
+# The password for the SSH server
+
+# SSH_PASSWORD="password"
+
+# The private key path in container for SSH server
+# Default value: /root/.ssh/id_rsa
+# If file is mounted to /root/.ssh/id_rsa path it will be used. Non-RSA keys will
+# also work.
+
+# SSH_IDENTITY_FILE="/root/.ssh/id_rsa"
+
+# The passphrase for the identity file
+
+# SSH_IDENTITY_PASSPHRASE="pass"
+
 # In addition to storing backups remotely, you can also keep local copies.
 # Pass a container-local path to store your backups if needed. You also need to
 # mount a local folder or Docker volume into that location (`/archive`
@@ -310,7 +352,7 @@ You can populate below template according to your requirements and use it as you

 # It is possible to define commands to be run in any container before and after
 # a backup is conducted. The commands themselves are defined in labels like
-# `docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump [options] > dump.sql'.
+# `docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump [options] > dump.sql'.
 # Several options exist for controlling this feature:

 # By default, any output of such a command is suppressed. If this value
@@ -395,7 +437,7 @@ You can populate below template according to your requirements and use it as you
 # EMAIL_SMTP_PORT="<port>"
 ```

-In case you encouter double quoted values in your configuration you might be running an [older version of `docker-compose`].
+In case you encouter double quoted values in your configuration you might be running an [older version of `docker-compose`][compose-issue].
 You can work around this by either updating `docker-compose` or unquoting your configuration values.

 [compose-issue]: https://github.com/docker/compose/issues/2854
@@ -502,11 +544,16 @@ Overridable template names are: `title_success`, `body_success`, `title_failure`

 For a full list of available variables and functions, see [this page](https://github.com/offen/docker-volume-backup/blob/master/docs/NOTIFICATION-TEMPLATES.md).

-### Run custom commands before / after backup
+### Run custom commands during the backup lifecycle

 In certain scenarios it can be required to run specific commands before and after a backup is taken (e.g. dumping a database).
-When mounting the Docker socket into the `docker-volume-backup` container, you can define pre- and post-commands that will be run in the context of the target container.
-Such commands are defined by specifying the command in a `docker-volume-backup.exec-[pre|post]` label.
+When mounting the Docker socket into the `docker-volume-backup` container, you can define pre- and post-commands that will be run in the context of the target container (it is also possible to run commands inside the `docker-volume-backup` container itself using this feature).
+Such commands are defined by specifying the command in a `docker-volume-backup.[step]-[pre|post]` label where `step` can be any of the following phases of a backup lifecyle:
+
+- `archive` (the tar archive is created)
+- `process` (the tar archive is processed, e.g. encrypted - optional)
+- `copy` (the tar archive is copied to all configured storages)
+- `prune` (existing backups are pruned based on the defined ruleset - optional)

 Taking a database dump using `mysqldump` would look like this:

@@ -520,7 +567,7 @@ services:
    volumes:
      - backup_data:/tmp/backups
    labels:
-      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /backups/dump.sql'
+      - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump --all-databases > /backups/dump.sql'

 volumes:
  backup_data:
@@ -540,7 +587,7 @@ services:
    volumes:
      - backup_data:/tmp/backups
    labels:
-      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /tmp/volume/dump.sql'
+      - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump --all-databases > /tmp/volume/dump.sql'
      - docker-volume-backup.exec-label=database

  backup:
@@ -556,7 +603,7 @@ volumes:
 ```


-The backup procedure is guaranteed to wait for all `pre` commands to finish.
+The backup procedure is guaranteed to wait for all `pre` or `post` commands to finish before proceeding.
 However there are no guarantees about the order in which they are run, which could also happen concurrently.

 ### Encrypting your backup using GPG
@@ -682,7 +729,7 @@ NOTIFICATION_URLS=smtp://me:secret@posteo.de:587/?fromAddress=no-reply@example.c
 ### Replace deprecated `BACKUP_FROM_SNAPSHOT` usage

 Starting with version 2.15.0, the `BACKUP_FROM_SNAPSHOT` feature has been deprecated.
-If you need to prepare your sources before the backup is taken, use `exec-pre`, `exec-post` and an intermediate volume:
+If you need to prepare your sources before the backup is taken, use `archive-pre`, `archive-post` and an intermediate volume:

 ```yml
 version: '3'
@@ -694,8 +741,8 @@ services:
      - data:/var/my_app
      - backup:/tmp/backup
    labels:
-      - docker-volume-backup.exec-pre=cp -r /var/my_app /tmp/backup/my-app
-      - docker-volume-backup.exec-post=rm -rf /tmp/backup/my-app
+      - docker-volume-backup.archive-pre=cp -r /var/my_app /tmp/backup/my-app
+      - docker-volume-backup.archive-post=rm -rf /tmp/backup/my-app

  backup:
    image: offen/docker-volume-backup:latest
@@ -710,6 +757,23 @@ volumes:
  backup:
 ```

+### Replace deprecated `exec-pre` and `exec-post` labels
+
+Version 2.19.0 introduced the option to run labeled commands at multiple points in time during the backup lifecycle.
+In order to be able to use more obvious terminology in the new labels, the existing `exec-pre` and `exec-post` labels have been deprecated.
+If you want to emulate the existing behavior, all you need to do is change `exec-pre` to `archive-pre` and `exec-post` to `archive-post`:
+
+```diff
+    labels:
+-     - docker-volume-backup.exec-pre=cp -r /var/my_app /tmp/backup/my-app
+     - docker-volume-backup.archive-pre=cp -r /var/my_app /tmp/backup/my-app
+-     - docker-volume-backup.exec-post=rm -rf /tmp/backup/my-app
+     - docker-volume-backup.archive-post=rm -rf /tmp/backup/my-app
+```
+
+The `EXEC_LABEL` setting and the `docker-volume-backup.exec-label` label stay as is.
+Check the additional documentation on running commands during the backup lifecycle to find out about further possibilities.
+
 ### Using a custom Docker host

 If you are interfacing with Docker via TCP, set `DOCKER_HOST` to the correct URL.
@@ -746,6 +810,25 @@ The exact order of schedules that use the same cron expression is not specified.
 In case you need your schedules to overlap, you need to create a dedicated container for each schedule instead.
 When changing the configuration, you currently need to manually restart the container for the changes to take effect.

+Set `BACKUP_SOURCES` for each config file to control which subset of volume mounts gets backed up:
+
+```yml
+# With a volume configuration like this:
+volumes:
+  - /var/run/docker.sock:/var/run/docker.sock:ro
+  - ./configuration:/etc/dockervolumebackup/conf.d
+  - app1_data:/backup/app1_data:ro
+  - app2_data:/backup/app2_data:ro
+```
+
+```ini
+# In the 1st config file:
+BACKUP_SOURCES=/backup/app1_data
+
+# In the 2nd config file:
+BACKUP_SOURCES=/backup/app2_data
+```
+
 ### Define different retention schedules

 If you want to manage backup retention on different schedules, the most straight forward approach is to define a dedicated configuration for retention rule using a different prefix in the `BACKUP_FILENAME` parameter and then run them on different cron schedules.
@@ -779,6 +862,22 @@ BACKUP_CRON_EXPRESSION="0 4 1 * *"

 Note that while it's possible to define colliding cron schedules for each of these configurations, you might need to adjust the value for `LOCK_TIMEOUT` in case your backups are large and might take longer than an hour.

+### Use special characters in notification URLs
+
+The value given to `NOTIFICATION_URLS` is a comma separated list of URLs.
+If such a URL contains special characters (e.g. commas) it needs to be URL encoded.
+To get an encoded version of your URL, you can use the CLI tool provided by `shoutrrr` (which is the library used for sending notifications):
+
+```
+docker run --rm -ti containrrr/shoutrrr generate [service]
+```
+
+where service is any of the [supported services][shoutrrr-docs], e.g. for SMTP:
+
+```
+docker run --rm -ti containrrr/shoutrrr generate smtp
+```
+
 ## Recipes

 This section lists configuration for some real-world use cases that you can mix and match according to your needs.
@@ -870,6 +969,29 @@ volumes:
  data:
 ```

+### Backing up to SSH
+
+```yml
+version: '3'
+
+services:
+  # ... define other services using the `data` volume here
+  backup:
+    image: offen/docker-volume-backup:v2
+    environment:
+      SSH_HOST_NAME: server.local
+      SSH_PORT: 2222
+      SSH_USER: user
+      SSH_REMOTE_PATH: /data
+    volumes:
+      - data:/backup/my-app-backup:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /path/to/private_key:/root/.ssh/id
+
+volumes:
+  data:
+```
+
 ### Backing up locally

 ```yml
--- a/cmd/backup/config.go
+++ b/cmd/backup/config.go
@@ -28,6 +28,7 @@ type Config struct {
 	AwsEndpoint                string        `split_words:"true" default:"s3.amazonaws.com"`
 	AwsEndpointProto           string        `split_words:"true" default:"https"`
 	AwsEndpointInsecure        bool          `split_words:"true"`
+	AwsStorageClass            string        `split_words:"true"`
 	AwsAccessKeyID             string        `envconfig:"AWS_ACCESS_KEY_ID"`
 	AwsSecretAccessKey         string        `split_words:"true"`
 	AwsIamRoleEndpoint         string        `split_words:"true"`
@@ -45,6 +46,13 @@ type Config struct {
 	WebdavPath                 string        `split_words:"true" default:"/"`
 	WebdavUsername             string        `split_words:"true"`
 	WebdavPassword             string        `split_words:"true"`
+	SSHHostName                string        `split_words:"true"`
+	SSHPort                    string        `split_words:"true" default:"22"`
+	SSHUser                    string        `split_words:"true"`
+	SSHPassword                string        `split_words:"true"`
+	SSHIdentityFile            string        `split_words:"true" default:"/root/.ssh/id_rsa"`
+	SSHIdentityPassphrase      string        `split_words:"true"`
+	SSHRemotePath              string        `split_words:"true"`
 	ExecLabel                  string        `split_words:"true"`
 	ExecForwardOutput          bool          `split_words:"true"`
 	LockTimeout                time.Duration `split_words:"true" default:"60m"`
--- a/cmd/backup/exec.go
+++ b/cmd/backup/exec.go
@@ -93,16 +93,68 @@ func (s *script) runLabeledCommands(label string) error {
 		return fmt.Errorf("runLabeledCommands: error querying for containers: %w", err)
 	}

+	var hasDeprecatedContainers bool
+	if label == "docker-volume-backup.archive-pre" {
+		f[0] = filters.KeyValuePair{
+			Key:   "label",
+			Value: "docker-volume-backup.exec-pre",
+		}
+		deprecatedContainers, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
+			Quiet:   true,
+			Filters: filters.NewArgs(f...),
+		})
+		if err != nil {
+			return fmt.Errorf("runLabeledCommands: error querying for containers: %w", err)
+		}
+		if len(deprecatedContainers) != 0 {
+			hasDeprecatedContainers = true
+			containersWithCommand = append(containersWithCommand, deprecatedContainers...)
+		}
+	}
+
+	if label == "docker-volume-backup.archive-post" {
+		f[0] = filters.KeyValuePair{
+			Key:   "label",
+			Value: "docker-volume-backup.exec-post",
+		}
+		deprecatedContainers, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
+			Quiet:   true,
+			Filters: filters.NewArgs(f...),
+		})
+		if err != nil {
+			return fmt.Errorf("runLabeledCommands: error querying for containers: %w", err)
+		}
+		if len(deprecatedContainers) != 0 {
+			hasDeprecatedContainers = true
+			containersWithCommand = append(containersWithCommand, deprecatedContainers...)
+		}
+	}
+
 	if len(containersWithCommand) == 0 {
 		return nil
 	}

+	if hasDeprecatedContainers {
+		s.logger.Warn(
+			"Using `docker-volume-backup.exec-pre` and `docker-volume-backup.exec-post` labels has been deprecated and will be removed in the next major version.",
+		)
+		s.logger.Warn(
+			"Please use other `-pre` and `-post` labels instead. Refer to the README for an upgrade guide.",
+		)
+	}
+
 	g := new(errgroup.Group)

 	for _, container := range containersWithCommand {
 		c := container
 		g.Go(func() error {
-			cmd, _ := c.Labels[label]
+			cmd, ok := c.Labels[label]
+			if !ok && label == "docker-volume-backup.archive-pre" {
+				cmd, _ = c.Labels["docker-volume-backup.exec-pre"]
+			} else if !ok && label == "docker-volume-backup.archive-post" {
+				cmd, _ = c.Labels["docker-volume-backup.exec-post"]
+			}
+
 			s.logger.Infof("Running %s command %s for container %s", label, cmd, strings.TrimPrefix(c.Names[0], "/"))
 			stdout, stderr, err := s.exec(c.ID, cmd)
 			if s.c.ExecForwardOutput {
@@ -121,3 +173,27 @@ func (s *script) runLabeledCommands(label string) error {
 	}
 	return nil
 }
+
+type lifecyclePhase string
+
+const (
+	lifecyclePhaseArchive lifecyclePhase = "archive"
+	lifecyclePhaseProcess lifecyclePhase = "process"
+	lifecyclePhaseCopy    lifecyclePhase = "copy"
+	lifecyclePhasePrune   lifecyclePhase = "prune"
+)
+
+func (s *script) withLabeledCommands(step lifecyclePhase, cb func() error) func() error {
+	if s.cli == nil {
+		return cb
+	}
+	return func() error {
+		if err := s.runLabeledCommands(fmt.Sprintf("docker-volume-backup.%s-pre", step)); err != nil {
+			return fmt.Errorf("withLabeledCommands: %s: error running pre commands: %w", step, err)
+		}
+		defer func() {
+			s.must(s.runLabeledCommands(fmt.Sprintf("docker-volume-backup.%s-post", step)))
+		}()
+		return cb()
+	}
+}
--- a/cmd/backup/main.go
+++ b/cmd/backup/main.go
@@ -38,14 +38,7 @@ func main() {
 		s.logger.Info("Finished running backup tasks.")
 	}()

-	s.must(func() error {
-		runPostCommands, err := s.runCommands()
-		defer func() {
-			s.must(runPostCommands())
-		}()
-		if err != nil {
-			return err
-		}
+	s.must(s.withLabeledCommands(lifecyclePhaseArchive, func() error {
 		restartContainers, err := s.stopContainers()
 		// The mechanism for restarting containers is not using hooks as it
 		// should happen as soon as possible (i.e. before uploading backups or
@@ -56,10 +49,10 @@ func main() {
 		if err != nil {
 			return err
 		}
-		return s.takeBackup()
-	}())
+		return s.createArchive()
+	})())

-	s.must(s.encryptBackup())
-	s.must(s.copyBackup())
-	s.must(s.pruneBackups())
+	s.must(s.withLabeledCommands(lifecyclePhaseProcess, s.encryptArchive)())
+	s.must(s.withLabeledCommands(lifecyclePhaseCopy, s.copyArchive)())
+	s.must(s.withLabeledCommands(lifecyclePhasePrune, s.pruneBackups)())
 }
--- a/cmd/backup/notifications.go
+++ b/cmd/backup/notifications.go
@@ -7,6 +7,7 @@ import (
 	"bytes"
 	_ "embed"
 	"fmt"
+	"os"
 	"text/template"
 	"time"

@@ -82,6 +83,7 @@ var templateHelpers = template.FuncMap{
 	"formatBytesBin": func(bytes uint64) string {
 		return formatBytes(bytes, false)
 	},
+	"env": os.Getenv,
 }

 // formatBytes converts an amount of bytes in a human-readable representation
--- a/cmd/backup/script.go
+++ b/cmd/backup/script.go
@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"io"
 	"io/fs"
+	"io/ioutil"
 	"net/http"
 	"os"
 	"path"
@@ -28,9 +29,11 @@ import (
 	"github.com/minio/minio-go/v7"
 	"github.com/minio/minio-go/v7/pkg/credentials"
 	"github.com/otiai10/copy"
+	"github.com/pkg/sftp"
 	"github.com/sirupsen/logrus"
 	"github.com/studio-b12/gowebdav"
 	"golang.org/x/crypto/openpgp"
+	"golang.org/x/crypto/ssh"
 )

 // script holds all the stateful information required to orchestrate a
@@ -39,6 +42,8 @@ type script struct {
 	cli          *client.Client
 	minioClient  *minio.Client
 	webdavClient *gowebdav.Client
+	sshClient    *ssh.Client
+	sftpClient   *sftp.Client
 	logger       *logrus.Logger
 	sender       *router.ServiceRouter
 	template     *template.Template
@@ -159,6 +164,57 @@ func newScript() (*script, error) {
 		}
 	}

+	if s.c.SSHHostName != "" {
+		var authMethods []ssh.AuthMethod
+
+		if s.c.SSHPassword != "" {
+			authMethods = append(authMethods, ssh.Password(s.c.SSHPassword))
+		}
+
+		if _, err := os.Stat(s.c.SSHIdentityFile); err == nil {
+			key, err := ioutil.ReadFile(s.c.SSHIdentityFile)
+			if err != nil {
+				return nil, errors.New("newScript: error reading the private key")
+			}
+
+			var signer ssh.Signer
+			if s.c.SSHIdentityPassphrase != "" {
+				signer, err = ssh.ParsePrivateKeyWithPassphrase(key, []byte(s.c.SSHIdentityPassphrase))
+				if err != nil {
+					return nil, errors.New("newScript: error parsing the encrypted private key")
+				}
+				authMethods = append(authMethods, ssh.PublicKeys(signer))
+			} else {
+				signer, err = ssh.ParsePrivateKey(key)
+				if err != nil {
+					return nil, errors.New("newScript: error parsing the private key")
+				}
+				authMethods = append(authMethods, ssh.PublicKeys(signer))
+			}
+		}
+
+		sshClientConfig := &ssh.ClientConfig{
+			User:            s.c.SSHUser,
+			Auth:            authMethods,
+			HostKeyCallback: ssh.InsecureIgnoreHostKey(),
+		}
+		sshClient, err := ssh.Dial("tcp", fmt.Sprintf("%s:%s", s.c.SSHHostName, s.c.SSHPort), sshClientConfig)
+		s.sshClient = sshClient
+		if err != nil {
+			return nil, fmt.Errorf("newScript: error creating ssh client: %w", err)
+		}
+		_, _, err = s.sshClient.SendRequest("keepalive", false, nil)
+		if err != nil {
+			return nil, err
+		}
+
+		sftpClient, err := sftp.NewClient(sshClient)
+		s.sftpClient = sftpClient
+		if err != nil {
+			return nil, fmt.Errorf("newScript: error creating sftp client: %w", err)
+		}
+	}
+
 	if s.c.EmailNotificationRecipient != "" {
 		emailURL := fmt.Sprintf(
 			"smtp://%s:%s@%s:%d/?from=%s&to=%s",
@@ -225,22 +281,6 @@ func newScript() (*script, error) {
 	return s, nil
 }

-func (s *script) runCommands() (func() error, error) {
-	if s.cli == nil {
-		return noop, nil
-	}
-
-	if err := s.runLabeledCommands("docker-volume-backup.exec-pre"); err != nil {
-		return noop, fmt.Errorf("runCommands: error running pre commands: %w", err)
-	}
-	return func() error {
-		if err := s.runLabeledCommands("docker-volume-backup.exec-post"); err != nil {
-			return fmt.Errorf("runCommands: error running post commands: %w", err)
-		}
-		return nil
-	}, nil
-}
-
 // stopContainers stops all Docker containers that are marked as to being
 // stopped during the backup and returns a function that can be called to
 // restart everything that has been stopped.
@@ -360,9 +400,9 @@ func (s *script) stopContainers() (func() error, error) {
 	}, stopError
 }

-// takeBackup creates a tar archive of the configured backup location and
+// createArchive creates a tar archive of the configured backup location and
 // saves it to disk.
-func (s *script) takeBackup() error {
+func (s *script) createArchive() error {
 	backupSources := s.c.BackupSources

 	if s.c.BackupFromSnapshot {
@@ -370,7 +410,7 @@ func (s *script) takeBackup() error {
 			"Using BACKUP_FROM_SNAPSHOT has been deprecated and will be removed in the next major version.",
 		)
 		s.logger.Warn(
-			"Please use `exec-pre` and `exec-post` commands to prepare your backup sources. Refer to the README for an upgrade guide.",
+			"Please use `archive-pre` and `archive-post` commands to prepare your backup sources. Refer to the README for an upgrade guide.",
 		)
 		backupSources = filepath.Join("/tmp", s.c.BackupSources)
 		// copy before compressing guard against a situation where backup folder's content are still growing.
@@ -427,10 +467,10 @@ func (s *script) takeBackup() error {
 	return nil
 }

-// encryptBackup encrypts the backup file using PGP and the configured passphrase.
+// encryptArchive encrypts the backup file using PGP and the configured passphrase.
 // In case no passphrase is given it returns early, leaving the backup file
 // untouched.
-func (s *script) encryptBackup() error {
+func (s *script) encryptArchive() error {
 	if s.c.GpgPassphrase == "" {
 		return nil
 	}
@@ -474,9 +514,9 @@ func (s *script) encryptBackup() error {
 	return nil
 }

-// copyBackup makes sure the backup file is copied to both local and remote locations
+// copyArchive makes sure the backup file is copied to both local and remote locations
 // as per the given configuration.
-func (s *script) copyBackup() error {
+func (s *script) copyArchive() error {
 	_, name := path.Split(s.file)
 	if stat, err := os.Stat(s.file); err != nil {
 		return fmt.Errorf("copyBackup: unable to stat backup file: %w", err)
@@ -491,7 +531,8 @@ func (s *script) copyBackup() error {

 	if s.minioClient != nil {
 		if _, err := s.minioClient.FPutObject(context.Background(), s.c.AwsS3BucketName, filepath.Join(s.c.AwsS3Path, name), s.file, minio.PutObjectOptions{
-			ContentType: "application/tar+gzip",
+			ContentType:  "application/tar+gzip",
+			StorageClass: s.c.AwsStorageClass,
 		}); err != nil {
 			return fmt.Errorf("copyBackup: error uploading backup to remote storage: %w", err)
 		}
@@ -512,6 +553,52 @@ func (s *script) copyBackup() error {
 		s.logger.Infof("Uploaded a copy of backup `%s` to WebDAV-URL '%s' at path '%s'.", s.file, s.c.WebdavUrl, s.c.WebdavPath)
 	}

+	if s.sshClient != nil {
+		source, err := os.Open(s.file)
+		if err != nil {
+			return fmt.Errorf("copyBackup: error reading the file to be uploaded: %w", err)
+		}
+		defer source.Close()
+
+		destination, err := s.sftpClient.Create(filepath.Join(s.c.SSHRemotePath, name))
+		if err != nil {
+			return fmt.Errorf("copyBackup: error creating file on SSH storage: %w", err)
+		}
+		defer destination.Close()
+
+		chunk := make([]byte, 1000000)
+		for {
+			num, err := source.Read(chunk)
+			if err == io.EOF {
+				tot, err := destination.Write(chunk[:num])
+				if err != nil {
+					return fmt.Errorf("copyBackup: error uploading the file to SSH storage: %w", err)
+				}
+
+				if tot != len(chunk[:num]) {
+					return fmt.Errorf("sshClient: failed to write stream")
+				}
+
+				break
+			}
+
+			if err != nil {
+				return fmt.Errorf("copyBackup: error uploading the file to SSH storage: %w", err)
+			}
+
+			tot, err := destination.Write(chunk[:num])
+			if err != nil {
+				return fmt.Errorf("copyBackup: error uploading the file to SSH storage: %w", err)
+			}
+
+			if tot != len(chunk[:num]) {
+				return fmt.Errorf("sshClient: failed to write stream")
+			}
+		}
+
+		s.logger.Infof("Uploaded a copy of backup `%s` to SSH storage '%s' at path '%s'.", s.file, s.c.SSHHostName, s.c.SSHRemotePath)
+	}
+
 	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
 		if err := copyFile(s.file, path.Join(s.c.BackupArchive, name)); err != nil {
 			return fmt.Errorf("copyBackup: error copying file to local archive: %w", err)
@@ -645,6 +732,37 @@ func (s *script) pruneBackups() error {
 		})
 	}

+	if s.sshClient != nil {
+		candidates, err := s.sftpClient.ReadDir(s.c.SSHRemotePath)
+		if err != nil {
+			return fmt.Errorf("pruneBackups: error reading directory from SSH storage: %w", err)
+		}
+
+		var matches []string
+		for _, candidate := range candidates {
+			if !strings.HasPrefix(candidate.Name(), s.c.BackupPruningPrefix) {
+				continue
+			}
+			if candidate.ModTime().Before(deadline) {
+				matches = append(matches, candidate.Name())
+			}
+		}
+
+		s.stats.Storages.SSH = StorageStats{
+			Total:  uint(len(candidates)),
+			Pruned: uint(len(matches)),
+		}
+
+		doPrune(len(matches), len(candidates), "SSH backup(s)", func() error {
+			for _, match := range matches {
+				if err := s.sftpClient.Remove(filepath.Join(s.c.SSHRemotePath, match)); err != nil {
+					return fmt.Errorf("pruneBackups: error removing file from SSH storage: %w", err)
+				}
+			}
+			return nil
+		})
+	}
+
 	if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
 		globPattern := path.Join(
 			s.c.BackupArchive,
--- a/cmd/backup/stats.go
+++ b/cmd/backup/stats.go
@@ -30,10 +30,11 @@ type StorageStats struct {
 	PruneErrors uint
 }

-// StoragesStats stats about each possible archival location (Local, WebDAV, S3)
+// StoragesStats stats about each possible archival location (Local, WebDAV, SSH, S3)
 type StoragesStats struct {
 	Local  StorageStats
 	WebDAV StorageStats
+	SSH    StorageStats
 	S3     StorageStats
 }

--- a/docs/NOTIFICATION-TEMPLATES.md
+++ b/docs/NOTIFICATION-TEMPLATES.md
@@ -25,15 +25,16 @@ Here is a list of all data passed to the template:
    * `FullPath`: full path of the backup file (e.g. `/archive/backup-2022-02-11T01-00-00.tar.gz`)
    * `Size`: size in bytes of the backup file
  * `Storages`: object that holds stats about each storage
-    * `Local`, `S3` or `WebDAV`:
+    * `Local`, `S3`, `WebDAV` or `SSH`:
      * `Total`: total number of backup files
      * `Pruned`: number of backup files that were deleted due to pruning rule
      * `PruneErrors`: number of backup files that were unable to be pruned

 ## Functions

-Some formatting functions are also available:
+Some formatting and helper functions are also available:

 * `formatTime`: formats a time object using [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339) format (e.g. `2022-02-11T01:00:00Z`)
 * `formatBytesBin`: formats an amount of bytes using powers of 1024 (e.g. `7055258` bytes will be `6.7 MiB`) 
 * `formatBytesDec`: formats an amount of bytes using powers of 1000 (e.g. `7055258` bytes will be `7.1 MB`)
+* `env`: returns the value of the environment variable of the given key if set
--- a/entrypoint.sh
+++ b/entrypoint.sh
@@ -22,4 +22,4 @@ else
 fi

 echo "Starting cron in foreground."
-crond -f -l 8
+crond -f -d 8
--- a/go.mod
+++ b/go.mod
@@ -11,15 +11,16 @@ require (
 	github.com/leekchan/timeutil v0.0.0-20150802142658-28917288c48d
 	github.com/minio/minio-go/v7 v7.0.16
 	github.com/otiai10/copy v1.7.0
+	github.com/pkg/sftp v1.13.5
 	github.com/sirupsen/logrus v1.8.1
 	github.com/studio-b12/gowebdav v0.0.0-20220128162035-c7b1ff8a5e62
-	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
-	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
+	golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3
+	golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
 )

 require (
-	github.com/Microsoft/go-winio v0.4.17 // indirect
-	github.com/containerd/containerd v1.5.5 // indirect
+	github.com/Microsoft/go-winio v0.5.2 // indirect
+	github.com/containerd/containerd v1.6.6 // indirect
 	github.com/docker/distribution v2.7.1+incompatible // indirect
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/docker/go-units v0.4.0 // indirect
@@ -27,33 +28,39 @@ require (
 	github.com/fatih/color v1.10.0 // indirect
 	github.com/fsnotify/fsnotify v1.4.9 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
-	github.com/golang/protobuf v1.5.0 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/google/uuid v1.3.0 // indirect
+	github.com/gorilla/mux v1.7.3 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/klauspost/compress v1.13.6 // indirect
+	github.com/klauspost/compress v1.15.6 // indirect
 	github.com/klauspost/cpuid/v2 v2.0.9 // indirect
+	github.com/kr/fs v0.1.0 // indirect
+	github.com/kr/text v0.2.0 // indirect
 	github.com/mattn/go-colorable v0.1.8 // indirect
 	github.com/mattn/go-isatty v0.0.12 // indirect
 	github.com/minio/md5-simd v1.1.2 // indirect
 	github.com/minio/sha256-simd v1.0.0 // indirect
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
+	github.com/moby/term v0.0.0-20200312100748-672ec06f55cd // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/morikuni/aec v1.0.0 // indirect
+	github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
 	github.com/nxadm/tail v1.4.6 // indirect
 	github.com/onsi/ginkgo v1.14.2 // indirect
 	github.com/onsi/gomega v1.10.3 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
-	github.com/opencontainers/image-spec v1.0.1 // indirect
+	github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/rs/xid v1.3.0 // indirect
-	golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 // indirect
-	golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1 // indirect
-	golang.org/x/text v0.3.6 // indirect
+	golang.org/x/net v0.0.0-20220607020251-c690dde0001d // indirect
+	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
+	golang.org/x/text v0.3.7 // indirect
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
-	google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a // indirect
-	google.golang.org/grpc v1.33.2 // indirect
-	google.golang.org/protobuf v1.26.0 // indirect
+	google.golang.org/genproto v0.0.0-20220602131408-e326c6e8e9c8 // indirect
+	google.golang.org/grpc v1.47.0 // indirect
+	google.golang.org/protobuf v1.28.0 // indirect
+	gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f // indirect
 	gopkg.in/ini.v1 v1.65.0 // indirect
 	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
--- a/go.sum
+++ b/go.sum
--- a/test/cli/run.sh
+++ b/test/cli/run.sh
@@ -3,6 +3,8 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))

 docker network create test_network
 docker volume create backup_data
@@ -50,17 +52,11 @@ docker run --rm -it \
  -v backup_data:/data alpine \
  ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db && test -d /backup/empty_data'

-echo "[TEST:PASS] Found relevant files in untared remote backup."
+pass "Found relevant files in untared remote backup."

 # This test does not stop containers during backup. This is happening on
 # purpose in order to cover this setup as well.
-if [ "$(docker ps -q | wc -l)" != "2" ]; then
-  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
-  docker ps
-  exit 1
-fi
-
-echo "[TEST:PASS] All containers running post backup."
+expect_running_containers "2"

 docker rm $(docker stop minio offen)
 docker volume rm backup_data app_data
--- a/test/commands/docker-compose.yml
+++ b/test/commands/docker-compose.yml
@@ -10,8 +10,9 @@ services:
      MARIADB_ROOT_PASSWORD: test
      MARIADB_DATABASE: backup
    labels:
+      # this is testing the deprecated label on purpose
      - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -ptest --all-databases > /tmp/volume/dump.sql'
-      - docker-volume-backup.exec-post=/bin/sh -c 'echo "post" > /tmp/volume/post.txt'
+      - docker-volume-backup.copy-post=/bin/sh -c 'echo "post" > /tmp/volume/post.txt'
      - docker-volume-backup.exec-label=test
    volumes:
      - app_data:/tmp/volume
--- a/test/commands/run.sh
+++ b/test/commands/run.sh
@@ -3,7 +3,8 @@
 set -e

 cd $(dirname $0)
-
+. ../util.sh
+current_test=$(basename $(pwd))

 docker-compose up -d
 sleep 30 # mariadb likes to take a bit before responding
@@ -13,29 +14,27 @@ sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' commands_archive

 tar -xvf ./local/test.tar.gz
 if [ ! -f ./backup/data/dump.sql ]; then
-  echo "[TEST:FAIL] Could not find file written by pre command."
-  exit 1
+  fail "Could not find file written by pre command."
 fi
-echo "[TEST:PASS] Found expected file."
+pass "Found expected file."

 if [ -f ./backup/data/post.txt ]; then
-  echo "[TEST:FAIL] File created in post command was present in backup."
-  exit 1
+  fail "File created in post command was present in backup."
 fi
-echo "[TEST:PASS] Did not find unexpected file."
+pass "Did not find unexpected file."

 docker-compose down --volumes
 sudo rm -rf ./local


-echo "[TEST:INFO] Running commands test in swarm mode next."
+info "Running commands test in swarm mode next."

 docker swarm init

 docker stack deploy --compose-file=docker-compose.yml test_stack

 while [ -z $(docker ps -q -f name=backup) ]; do
-  echo "[TEST:INFO] Backup container not ready yet. Retrying."
+  info "Backup container not ready yet. Retrying."
  sleep 1
 done

@@ -47,16 +46,14 @@ sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' test_stack_archi

 tar -xvf ./local/test.tar.gz
 if [ ! -f ./backup/data/dump.sql ]; then
-  echo "[TEST:FAIL] Could not find file written by pre command."
-  exit 1
+  fail "Could not find file written by pre command."
 fi
-echo "[TEST:PASS] Found expected file."
+pass "Found expected file."

 if [ -f ./backup/data/post.txt ]; then
-  echo "[TEST:FAIL] File created in post command was present in backup."
-  exit 1
+  fail "File created in post command was present in backup."
 fi
-echo "[TEST:PASS] Did not find unexpected file."
+pass "Did not find unexpected file."

 docker stack rm test_stack
 docker swarm leave --force
--- a/test/compose/run.sh
+++ b/test/compose/run.sh
@@ -1,68 +0,0 @@
-#!/bin/sh
-
-set -e
-
-cd $(dirname $0)
-
-mkdir -p local
-
-docker-compose up -d
-sleep 5
-
-# A symlink for a known file in the volume is created so the test can check
-# whether symlinks are preserved on backup.
-docker-compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
-docker-compose exec backup backup
-
-sleep 5
-if [ "$(docker-compose ps -q | wc -l)" != "4" ]; then
-  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
-  docker-compose ps
-  exit 1
-fi
-echo "[TEST:PASS] All containers running post backup."
-
-
-docker run --rm -it \
-  -v compose_minio_backup_data:/minio_data \
-  -v compose_webdav_backup_data:/webdav_data alpine \
-  ash -c 'apk add gnupg && \
-          echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /minio_data/backup/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xvf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db && \
-          echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /webdav_data/data/my/new/path/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xvf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
-
-echo "[TEST:PASS] Found relevant files in decrypted and untared remote backups."
-
-echo 1234secret | gpg -d --pinentry-mode loopback --yes --passphrase-fd 0 ./local/test-hostnametoken.tar.gz.gpg > ./local/decrypted.tar.gz
-tar -xf ./local/decrypted.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db
-rm ./local/decrypted.tar.gz
-test -L /tmp/backup/app_data/db.link
-
-echo "[TEST:PASS] Found relevant files in decrypted and untared local backup."
-
-test -L ./local/test-hostnametoken.latest.tar.gz.gpg
-echo "[TEST:PASS] Found symlink to latest version in local backup."
-
-# The second part of this test checks if backups get deleted when the retention
-# is set to 0 days (which it should not as it would mean all backups get deleted)
-# TODO: find out if we can test actual deletion without having to wait for a day
-BACKUP_RETENTION_DAYS="0" docker-compose up -d
-sleep 5
-
-docker-compose exec backup backup
-
-docker run --rm -it \
-  -v compose_minio_backup_data:/minio_data \
-  -v compose_webdav_backup_data:/webdav_data alpine \
-  ash -c '[ $(find /minio_data/backup/ -type f | wc -l) = "1" ] && \
-          [ $(find /webdav_data/data/my/new/path/ -type f | wc -l) = "1" ]'
-
-echo "[TEST:PASS] Remote backups have not been deleted."
-
-if [ "$(find ./local -type f | wc -l)" != "1" ]; then
-  echo "[TEST:FAIL] Backups should not have been deleted, instead seen:"
-  find ./local -type f
-  exit 1
-fi
-echo "[TEST:PASS] Local backups have not been deleted."
-
-docker-compose down --volumes
--- a/test/confd/run.sh
+++ b/test/confd/run.sh
@@ -3,6 +3,8 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))

 mkdir -p local

@@ -14,19 +16,16 @@ sleep 100
 docker-compose down --volumes

 if [ ! -f ./local/conf.tar.gz ]; then
-  echo "[TEST:FAIL] Config from file was not used."
-  exit 1
+  fail "Config from file was not used."
 fi
-echo "[TEST:PASS] Config from file was used."
+pass "Config from file was used."

 if [ ! -f ./local/other.tar.gz ]; then
-  echo "[TEST:FAIL] Run on same schedule did not succeed."
-  exit 1
+  fail "Run on same schedule did not succeed."
 fi
-echo "[TEST:PASS] Run on same schedule succeeded."
+pass "Run on same schedule succeeded."

 if [ -f ./local/never.tar.gz ]; then
-  echo "[TEST:FAIL] Unexpected file was found."
-  exit 1
+  fail "Unexpected file was found."
 fi
-echo "[TEST:PASS] Unexpected cron did not run."
+pass "Unexpected cron did not run."
--- a/test/compose/.gitignore
+++ b/test/compose/.gitignore
--- a/test/gpg/docker-compose.yml
+++ b/test/gpg/docker-compose.yml
@@ -0,0 +1,26 @@
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    restart: always
+    environment:
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_FILENAME: test.tar.gz
+      BACKUP_LATEST_SYMLINK: test-latest.tar.gz.gpg
+      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
+      GPG_PASSPHRASE: 1234secret
+    volumes:
+      - ./local:/archive
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  app_data:
--- a/test/gpg/run.sh
+++ b/test/gpg/run.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+mkdir -p local
+
+docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+expect_running_containers "2"
+
+tmp_dir=$(mktemp -d)
+
+echo 1234secret | gpg -d --pinentry-mode loopback --yes --passphrase-fd 0 ./local/test.tar.gz.gpg > ./local/decrypted.tar.gz
+tar -xf ./local/decrypted.tar.gz -C $tmp_dir
+ls -lah $tmp_dir
+if [ ! -f $tmp_dir/backup/app_data/offen.db ]; then
+  fail "Could not find expected file in untared archive."
+fi
+rm ./local/decrypted.tar.gz
+
+pass "Found relevant files in decrypted and untared local backup."
+
+if [ ! -L ./local/test-latest.tar.gz.gpg ]; then
+  fail "Could not find local symlink to latest encrypted backup."
+fi
+
+docker-compose down --volumes
--- a/test/ignore/run.sh
+++ b/test/ignore/run.sh
@@ -3,6 +3,9 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))
+
 mkdir -p local

 docker-compose up -d
@@ -15,13 +18,11 @@ out=$(mktemp -d)
 sudo tar --same-owner -xvf ./local/test.tar.gz -C "$out"

 if [ ! -f "$out/backup/data/me.txt" ]; then
-  echo "[TEST:FAIL] Expected file was not found."
-  exit 1
+  fail "Expected file was not found."
 fi
-echo "[TEST:PASS] Expected file was found."
+pass "Expected file was found."

 if [ -f "$out/backup/data/skip.me" ]; then
-  echo "[TEST:FAIL] Ignored file was found."
-  exit 1
+  fail "Ignored file was found."
 fi
-echo "[TEST:PASS] Ignored file was not found."
+pass "Ignored file was not found."
--- a/test/local/.gitignore
+++ b/test/local/.gitignore
@@ -0,0 +1 @@
+local
--- a/test/local/docker-compose.yml
+++ b/test/local/docker-compose.yml
@@ -0,0 +1,29 @@
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    hostname: hostnametoken
+    restart: always
+    environment:
+      BACKUP_FILENAME_EXPAND: 'true'
+      BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
+      BACKUP_LATEST_SYMLINK: test-$$HOSTNAME.latest.tar.gz.gpg
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
+      BACKUP_PRUNING_LEEWAY: 5s
+      BACKUP_PRUNING_PREFIX: test
+    volumes:
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./local:/archive
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  app_data:
--- a/test/local/run.sh
+++ b/test/local/run.sh
@@ -0,0 +1,55 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+mkdir -p local
+
+docker-compose up -d
+sleep 5
+
+# A symlink for a known file in the volume is created so the test can check
+# whether symlinks are preserved on backup.
+docker-compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
+docker-compose exec backup backup
+
+sleep 5
+
+expect_running_containers "2"
+
+tmp_dir=$(mktemp -d)
+tar -xvf ./local/test-hostnametoken.tar.gz -C $tmp_dir
+if [ ! -f "$tmp_dir/backup/app_data/offen.db" ]; then
+  fail "Could not find expected file in untared archive."
+fi
+rm -f ./local/test-hostnametoken.tar.gz
+
+if [ ! -L "$tmp_dir/backup/app_data/db.link" ]; then
+  fail "Could not find expected symlink in untared archive."
+fi
+
+pass "Found relevant files in decrypted and untared local backup."
+
+if [ ! -L ./local/test-hostnametoken.latest.tar.gz.gpg ]; then
+  fail "Could not find symlink to latest version."
+fi
+
+pass "Found symlink to latest version in local backup."
+
+# The second part of this test checks if backups get deleted when the retention
+# is set to 0 days (which it should not as it would mean all backups get deleted)
+# TODO: find out if we can test actual deletion without having to wait for a day
+BACKUP_RETENTION_DAYS="0" docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+if [ "$(find ./local -type f | wc -l)" != "1" ]; then
+  fail "Backups should not have been deleted, instead seen: "$(find ./local -type f)""
+fi
+pass "Local backups have not been deleted."
+
+docker-compose down --volumes
--- a/test/notifications/docker-compose.yml
+++ b/test/notifications/docker-compose.yml
@@ -10,6 +10,7 @@ services:
      BACKUP_PRUNING_PREFIX: test
      NOTIFICATION_LEVEL: info
      NOTIFICATION_URLS: ${NOTIFICATION_URLS}
+      EXTRA_VALUE: extra-value
    volumes:
      - ./local:/archive
      - app_data:/backup/app_data:ro
--- a/test/notifications/notifications.tmpl
+++ b/test/notifications/notifications.tmpl
@@ -1,5 +1,5 @@
 {{ define "title_success" -}}
-Successful test run, yay!
+Successful test run with {{ env "EXTRA_VALUE" }}, yay!
 {{- end }}

 {{ define "body_success" -}}
--- a/test/notifications/run.sh
+++ b/test/notifications/run.sh
@@ -3,6 +3,8 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))

 mkdir -p local

@@ -10,16 +12,15 @@ docker-compose up -d
 sleep 5

 GOTIFY_TOKEN=$(curl -sSLX POST -H 'Content-Type: application/json' -d '{"name":"test"}' http://admin:custom@localhost:8080/application | jq -r '.token')
-echo "[TEST:INFO] Set up Gotify application using token $GOTIFY_TOKEN"
+info "Set up Gotify application using token $GOTIFY_TOKEN"

 docker-compose exec backup backup

 NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
 if [ "$NUM_MESSAGES" != 0 ]; then
-  echo "[TEST:FAIL] Expected no notifications to be sent when not configured"
-  exit 1
+  fail "Expected no notifications to be sent when not configured"
 fi
-echo "[TEST:PASS] No notifications were sent when not configured."
+pass "No notifications were sent when not configured."

 docker-compose down

@@ -29,24 +30,21 @@ docker-compose exec backup backup

 NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
 if [ "$NUM_MESSAGES" != 1 ]; then
-  echo "[TEST:FAIL] Expected one notifications to be sent when configured"
-  exit 1
+  fail "Expected one notifications to be sent when configured"
 fi
-echo "[TEST:PASS] Correct number of notifications were sent when configured."
+pass "Correct number of notifications were sent when configured."

 MESSAGE_TITLE=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages[0].title')
 MESSAGE_BODY=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages[0].message')

-if [ "$MESSAGE_TITLE" != "Successful test run, yay!" ]; then
-  echo "[TEST:FAIL] Unexpected notification title $MESSAGE_TITLE"
-  exit 1
+if [ "$MESSAGE_TITLE" != "Successful test run with extra-value, yay!" ]; then
+  fail "Unexpected notification title $MESSAGE_TITLE"
 fi
-echo "[TEST:PASS] Custom notification title was used."
+pass "Custom notification title was used."

 if [ "$MESSAGE_BODY" != "Backing up /tmp/test.tar.gz succeeded." ]; then
-  echo "[TEST:FAIL] Unexpected notification body $MESSAGE_BODY"
-  exit 1
+  fail "Unexpected notification body $MESSAGE_BODY"
 fi
-echo "[TEST:PASS] Custom notification body was used."
+pass "Custom notification body was used."

 docker-compose down --volumes
--- a/test/ownership/run.sh
+++ b/test/ownership/run.sh
@@ -4,6 +4,8 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))

 mkdir -p local

@@ -12,17 +14,17 @@ sleep 5

 docker-compose exec backup backup

-sudo tar --same-owner -xvf ./local/backup.tar.gz -C /tmp
+tmp_dir=$(mktemp -d)
+sudo tar --same-owner -xvf ./local/backup.tar.gz -C $tmp_dir

-sudo find /tmp/backup/postgres > /dev/null
-echo "[TEST:PASS] Backup contains files at expected location"
+sudo find $tmp_dir/backup/postgres > /dev/null
+pass "Backup contains files at expected location"

-for file in $(sudo find /tmp/backup/postgres); do
+for file in $(sudo find $tmp_dir/backup/postgres); do
  if [ "$(sudo stat -c '%u:%g' $file)" != "70:70" ]; then
-    echo "[TEST:FAIL] Unexpected file ownership for $file: $(sudo stat -c '%u:%g' $file)"
-    exit 1
+    fail "Unexpected file ownership for $file: $(sudo stat -c '%u:%g' $file)"
  fi
 done
-echo "[TEST:PASS] All files and directories in backup preserved their ownership."
+pass "All files and directories in backup preserved their ownership."

 docker-compose down --volumes
--- a/test/compose/docker-compose.yml
+++ b/test/compose/docker-compose.yml
@@ -12,21 +12,11 @@ services:
    volumes:
      - minio_backup_data:/data

-  webdav:
-    image: bytemark/webdav:2.4
-    environment:
-      AUTH_TYPE: Digest
-      USERNAME: test
-      PASSWORD: test
-    volumes:
-      - webdav_backup_data:/var/lib/dav
-
  backup:
    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
    hostname: hostnametoken
    depends_on:
      - minio
-      - webdav
    restart: always
    environment:
      AWS_ACCESS_KEY_ID: test
@@ -36,19 +26,11 @@ services:
      AWS_S3_BUCKET_NAME: backup
      BACKUP_FILENAME_EXPAND: 'true'
      BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
-      BACKUP_LATEST_SYMLINK: test-$$HOSTNAME.latest.tar.gz.gpg
      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
      BACKUP_PRUNING_LEEWAY: 5s
      BACKUP_PRUNING_PREFIX: test
-      GPG_PASSPHRASE: 1234secret
-      WEBDAV_URL: http://webdav/
-      WEBDAV_URL_INSECURE: 'true'
-      WEBDAV_PATH: /my/new/path/
-      WEBDAV_USERNAME: test
-      WEBDAV_PASSWORD: test
    volumes:
-      - ./local:/archive
      - app_data:/backup/app_data:ro
      - /var/run/docker.sock:/var/run/docker.sock

@@ -61,5 +43,5 @@ services:

 volumes:
  minio_backup_data:
-  webdav_backup_data:
+    name: minio_backup_data
  app_data:
--- a/test/s3/run.sh
+++ b/test/s3/run.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+docker-compose up -d
+sleep 5
+
+# A symlink for a known file in the volume is created so the test can check
+# whether symlinks are preserved on backup.
+docker-compose exec backup backup
+
+sleep 5
+
+expect_running_containers "3"
+
+docker run --rm -it \
+  -v minio_backup_data:/minio_data \
+  alpine \
+  ash -c 'tar -xvf /minio_data/backup/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
+
+pass "Found relevant files in untared remote backups."
+
+# The second part of this test checks if backups get deleted when the retention
+# is set to 0 days (which it should not as it would mean all backups get deleted)
+# TODO: find out if we can test actual deletion without having to wait for a day
+BACKUP_RETENTION_DAYS="0" docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+docker run --rm -it \
+  -v minio_backup_data:/minio_data \
+  alpine \
+  ash -c '[ $(find /minio_data/backup/ -type f | wc -l) = "1" ]'
+
+pass "Remote backups have not been deleted."
+
+docker-compose down --volumes
--- a/test/ssh/docker-compose.yml
+++ b/test/ssh/docker-compose.yml
@@ -0,0 +1,47 @@
+version: '3'
+
+services:
+  ssh:
+    image: linuxserver/openssh-server:version-8.6_p1-r3
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - USER_NAME=test
+    volumes:
+      - ./id_rsa.pub:/config/.ssh/authorized_keys
+      - ssh_backup_data:/tmp
+
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    hostname: hostnametoken
+    depends_on:
+      - ssh
+    restart: always
+    environment:
+      BACKUP_FILENAME_EXPAND: 'true'
+      BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
+      BACKUP_PRUNING_LEEWAY: 5s
+      BACKUP_PRUNING_PREFIX: test
+      SSH_HOST_NAME: ssh
+      SSH_PORT: 2222
+      SSH_USER: test
+      SSH_REMOTE_PATH: /tmp
+      SSH_IDENTITY_PASSPHRASE: test1234
+    volumes:
+      - ./id_rsa:/root/.ssh/id_rsa
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  ssh_backup_data:
+    name: ssh_backup_data
+  app_data:
--- a/test/ssh/run.sh
+++ b/test/ssh/run.sh
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+ssh-keygen -t rsa -m pem -b 4096 -N "test1234" -f id_rsa -C "docker-volume-backup@local"
+
+docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+sleep 5
+
+expect_running_containers 3
+
+docker run --rm -it \
+  -v ssh_backup_data:/ssh_data \
+  alpine \
+  ash -c 'tar -xvf /ssh_data/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
+
+pass "Found relevant files in decrypted and untared remote backups."
+
+# The second part of this test checks if backups get deleted when the retention
+# is set to 0 days (which it should not as it would mean all backups get deleted)
+# TODO: find out if we can test actual deletion without having to wait for a day
+BACKUP_RETENTION_DAYS="0" docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+docker run --rm -it \
+  -v ssh_backup_data:/ssh_data \
+  alpine \
+  ash -c '[ $(find /ssh_data/ -type f | wc -l) = "1" ]'
+
+pass "Remote backups have not been deleted."
+
+docker-compose down --volumes
+rm -f id_rsa id_rsa.pub
--- a/test/swarm/docker-compose.yml
+++ b/test/swarm/docker-compose.yml
@@ -64,4 +64,5 @@ services:

 volumes:
  backup_data:
+    name: backup_data
  pg_data:
--- a/test/swarm/run.sh
+++ b/test/swarm/run.sh
@@ -3,13 +3,15 @@
 set -e

 cd $(dirname $0)
+. ../util.sh
+current_test=$(basename $(pwd))

 docker swarm init

 docker stack deploy --compose-file=docker-compose.yml test_stack

 while [ -z $(docker ps -q -f name=backup) ]; do
-  echo "[TEST:INFO] Backup container not ready yet. Retrying."
+  info "Backup container not ready yet. Retrying."
  sleep 1
 done

@@ -18,18 +20,13 @@ sleep 20
 docker exec $(docker ps -q -f name=backup) backup

 docker run --rm -it \
-  -v test_stack_backup_data:/data alpine \
+  -v backup_data:/data alpine \
  ash -c 'tar -xf /data/backup/test.tar.gz && test -f /backup/pg_data/PG_VERSION'

-echo "[TEST:PASS] Found relevant files in untared backup."
+pass "Found relevant files in untared backup."

 sleep 5
-if [ "$(docker ps -q | wc -l)" != "5" ]; then
-  echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
-  docker ps -a
-  exit 1
-fi
-echo "[TEST:PASS] All containers running post backup."
+expect_running_containers "5"

 docker stack rm test_stack
 docker swarm leave --force
--- a/test/util.sh
+++ b/test/util.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+set -e
+
+info () {
+  echo "[test:${current_test:-none}:info] "$1""
+}
+
+pass () {
+  echo "[test:${current_test:-none}:pass] "$1""
+}
+
+fail () {
+  echo "[test:${current_test:-none}:fail] "$1""
+  exit 1
+}
+
+expect_running_containers () {
+  if [ "$(docker ps -q | wc -l)" != "$1" ]; then
+    fail "Expected $1 containers to be running, instead seen: "$(docker ps -a | wc -l)""
+  fi
+  pass "$1 containers running."
+}
--- a/test/webdav/docker-compose.yml
+++ b/test/webdav/docker-compose.yml
@@ -0,0 +1,45 @@
+version: '3'
+
+services:
+  webdav:
+    image: bytemark/webdav:2.4
+    environment:
+      AUTH_TYPE: Digest
+      USERNAME: test
+      PASSWORD: test
+    volumes:
+      - webdav_backup_data:/var/lib/dav
+
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    hostname: hostnametoken
+    depends_on:
+      - webdav
+    restart: always
+    environment:
+      BACKUP_FILENAME_EXPAND: 'true'
+      BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
+      BACKUP_PRUNING_LEEWAY: 5s
+      BACKUP_PRUNING_PREFIX: test
+      WEBDAV_URL: http://webdav/
+      WEBDAV_URL_INSECURE: 'true'
+      WEBDAV_PATH: /my/new/path/
+      WEBDAV_USERNAME: test
+      WEBDAV_PASSWORD: test
+    volumes:
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  webdav_backup_data:
+    name: webdav_backup_data
+  app_data:
--- a/test/webdav/run.sh
+++ b/test/webdav/run.sh
@@ -0,0 +1,40 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+sleep 5
+
+expect_running_containers "3"
+
+docker run --rm -it \
+  -v webdav_backup_data:/webdav_data \
+  alpine \
+  ash -c 'tar -xvf /webdav_data/data/my/new/path/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
+
+pass "Found relevant files in untared remote backup."
+
+# The second part of this test checks if backups get deleted when the retention
+# is set to 0 days (which it should not as it would mean all backups get deleted)
+# TODO: find out if we can test actual deletion without having to wait for a day
+BACKUP_RETENTION_DAYS="0" docker-compose up -d
+sleep 5
+
+docker-compose exec backup backup
+
+docker run --rm -it \
+  -v webdav_backup_data:/webdav_data \
+  alpine \
+  ash -c '[ $(find /webdav_data/data/my/new/path/ -type f | wc -l) = "1" ]'
+
+pass "Remote backups have not been deleted."
+
+docker-compose down --volumes
Author	SHA1	Message	Date
Frederik Ring	b441cf3e2b	Fine grained labels (#115 ) * Refactor label command mechanism to be more flexible * Run all steps wrapped in labeled commands * Rename methods to be in line with lifecycle * Deprecate exec-pre and exec-post labels * Add documentation * Use type alias for lifecycle phases * Fix bad imports * Fix command lookup for deprecated labels * Use more generic naming for lifecycle phase * Fail on erroneous post command * Update documentation	2022-07-10 10:36:56 +02:00
Frederik Ring	82f66565da	Add further docs on backup selection when using conf.d	2022-07-08 14:05:45 +02:00
Frederik Ring	d68814be9d	Add section about shoutrrr CLI tool to README	2022-07-07 22:22:21 +02:00
Erwan LE PRADO	3661a4b49b	feat: Add storage class header (#119 ) * feat: Add storage class header * doc: change the readme * chore: Remove the unnecessary default value	2022-07-06 13:18:12 +02:00
Frederik Ring	e738bd0539	Make crond log to stderr so Docker can forward it (#120 )	2022-07-06 13:16:43 +02:00
Frederik Ring	342ae5910e	Add env template helper (#121 )	2022-07-06 13:16:32 +02:00
Frederik Ring	c2a8cc92fc	Untangle tests (#112 ) * Isolate S3 test case * Isolate webdav test case * Isolate SSH test case * Isolate local storage test case * Isolate gpg test case * Add missing volume mount * Fix file locations for local test case * Remove compose test case, use utils * Use test utils throughout * Use dedicated tmp dir * Fix link location that is being tested * Use dedicated tmp_dirs when working on host fs * Force delete artifact * Fix expected filename * Provide helpful messages on failing tests * Fix filename * Use proper volume names * Fix syntax error, use large resource class * Use named Docker volumes when referencing them in test scripts * Add name of test case to logging output	2022-06-23 14:40:29 +02:00
Frederik Ring	1892d56ff6	Change default value for SSH identity file (#108 ) * Change default value for SSH identity file * Force remove write protected file in tests	2022-06-17 11:28:29 +02:00
İbrahim Akyel	0b205fe6dc	SSH Backup Storage Support (#107 ) * SSH Client implemented * Private key auth implemented Code refactoring * Refactoring * Passphrase renamed to IdentityPassphrase Default private key location changed to .ssh/id	2022-06-17 11:06:15 +02:00
Frederik Ring	8c8a2fa088	Update vulnerable containerd dependency (#104 )	2022-06-07 09:21:40 +02:00
Frederik Ring	a850bf13fe	Fix broken link in README	2022-05-12 08:18:12 +02:00