Add support for Azure Blob Storage (#171)

* Scaffold Azure storage backend that does nothing yet

* Implement copy for Azure Blob Storage

* Set up automated testing for Azure Storage

* Implement pruning for Azure blob storage

* Add documentation for Azure Blob Storage

* Add support for remote path

* Add azure to notifications doc

* Tidy go.mod file

* Allow use of managed identity credential

* Use volume in tests

* Auto append trailing slash to endpoint if needed, clarify docs, tidy mod file

Dockerfile

@@ -9,7 +9,7 @@ RUN go mod download
 WORKDIR /app/cmd/backup
 RUN go build -o backup .
 
-FROM alpine:3.16
+FROM alpine:3.17
 
 WORKDIR /root
 

README.md

@@ -4,7 +4,7 @@
 
 # docker-volume-backup
 
-Backup Docker volumes locally or to any S3 compatible storage.
+Backup Docker volumes locally or to any S3, WebDAV or SSH compatible storage.
 
 The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
 It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV, Azure Blob Storage or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
@@ -33,6 +33,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
   - [Run multiple backup schedules in the same container](#run-multiple-backup-schedules-in-the-same-container)
   - [Define different retention schedules](#define-different-retention-schedules)
   - [Use special characters in notification URLs](#use-special-characters-in-notification-urls)
+  - [Handle file uploads using third party tools](#handle-file-uploads-using-third-party-tools)
 - [Recipes](#recipes)
   - [Backing up to AWS S3](#backing-up-to-aws-s3)
   - [Backing up to Filebase](#backing-up-to-filebase)
@@ -304,7 +305,8 @@ You can populate below template according to your requirements and use it as you
 
 # SSH_IDENTITY_PASSPHRASE="pass"
 
-# The credential's account name when using Azure Blob Storage.
+# The credential's account name when using Azure Blob Storage. This has to be
+# set when using Azure Blob Storage.
 
 # AZURE_STORAGE_ACCOUNT_NAME="account-name"
 
@@ -318,7 +320,7 @@ You can populate below template according to your requirements and use it as you
 # AZURE_STORAGE_CONTAINER_NAME="container-name"
 
 # The service endpoint when using Azure Blob Storage. This is a template that
-# will be passed the account name as shown in the default value below.
+# can be passed the account name as shown in the default value below.
 
 # AZURE_STORAGE_ENDPOINT="https://{{ .AccountName }}.blob.core.windows.net/"
 
@@ -913,6 +915,44 @@ where service is any of the [supported services][shoutrrr-docs], e.g. for SMTP:
 docker run --rm -ti containrrr/shoutrrr generate smtp
 ```
 
+### Handle file uploads using third party tools
+
+If you want to use a non-supported storage backend, or want to use a third party (e.g. rsync, rclone) tool for file uploads, you can build a Docker image containing the required binaries off this one, and call through to these in lifecycle hooks.
+
+For example, if you wanted to use `rsync`, define your Docker image like this:
+
+```Dockerfile
+FROM offen/docker-volume-backup:v2
+
+RUN apk add rsync
+```
+
+Using this image, you can now omit configuring any of the supported storage backends, and instead define your own mechanism in a `docker-volume-backup.copy-post` label:
+
+```yml
+version: '3'
+
+services:
+  backup:
+    image: your-custom-image
+    restart: always
+    environment:
+      BACKUP_FILENAME: "daily-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
+      BACKUP_CRON_EXPRESSION: "0 2 * * *"
+    labels:
+      - docker-volume-backup.copy-post=/bin/sh -c 'rsync $$COMMAND_RUNTIME_ARCHIVE_FILEPATH /destination'
+    volumes:
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  # other services defined here ...
+volumes:
+  app_data:
+```
+
+
+Commands will be invoked with the filepath of the tar archive passed as `COMMAND_RUNTIME_BACKUP_FILEPATH`.
+
 ## Recipes
 
 This section lists configuration for some real-world use cases that you can mix and match according to your needs.

cmd/backup/exec.go

@@ -23,10 +23,14 @@ import (
 
 func (s *script) exec(containerRef string, command string) ([]byte, []byte, error) {
 	args, _ := argv.Argv(command, nil, nil)
+	commandEnv := []string{
+		fmt.Sprintf("COMMAND_RUNTIME_ARCHIVE_FILEPATH=%s", s.file),
+	}
 	execID, err := s.cli.ContainerExecCreate(context.Background(), containerRef, types.ExecConfig{
 		Cmd:          args[0],
 		AttachStdin:  true,
 		AttachStderr: true,
+		Env:          commandEnv,
 	})
 	if err != nil {
 		return nil, nil, fmt.Errorf("exec: error creating container exec: %w", err)

go.mod

@@ -3,6 +3,7 @@ module github.com/offen/docker-volume-backup
 go 1.19
 
 require (
+	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.6.1
 	github.com/containrrr/shoutrrr v0.5.2
 	github.com/cosiner/argv v0.1.0
@@ -21,7 +22,6 @@ require (
 
 require (
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4 // indirect
-	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.1 // indirect
 	github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 // indirect
 	github.com/Microsoft/go-winio v0.5.2 // indirect

go.sum

@@ -2,7 +2,6 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
 cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4 h1:pqrAR74b6EoR4kcxF7L7Wg2B8Jgil9UUZtMvxhEFqWo=
 github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.4/go.mod h1:uGG2W01BaETf0Ozp+QxxKJdMBNRWPdstHG0Fmdwn1/U=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0 h1:QkAcEIAKbNL4KoFr4SathZPhDhF4mVwpBMFlYjyAqy8=
 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0 h1:t/W5MYAuQy81cvM8VUNfRLzhtKpXhVUAN7Cd7KVbTyc=
 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0/go.mod h1:NBanQUfSWiWn3QEpWDTCU0IjBECKOYvl2R8xdRtMtiM=
 github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.1 h1:XUNQ4mw+zJmaA2KXzP9JlQiecy1SI+Eog7xVkPiqIbg=
@@ -11,7 +10,6 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.6.1 h1:YvQv9Mz6T8oR5ypQO
 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.6.1/go.mod h1:c6WvOhtmjNUWbLfOG1qxM/q0SPvQNSVJvolm+C52dIU=
 github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
 github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
-github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 h1:BWe8a+f/t+7KY7zH2mqygeUD0t8hNFXe08p1Pb3/jKE=
 github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 h1:VgSJlZH5u0k2qxSpqyghcFQKmvYckj46uymKK5XzkBM=
 github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0/go.mod h1:BDJ5qMFKx9DugEg3+uQSDCdbYPr5s9vBTrL9P8TpqOU=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
@@ -107,7 +105,6 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
-github.com/golang-jwt/jwt v3.2.1+incompatible h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c=
 github.com/golang-jwt/jwt/v4 v4.4.2 h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
 github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=

internal/storage/azure/azure.go

@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 	"sync"
 	"text/template"
 	"time"
@@ -45,6 +46,7 @@ func NewStorageBackend(opts Config, logFunc storage.Log) (storage.Backend, error
 	if err := endpointTemplate.Execute(&ep, opts); err != nil {
 		return nil, fmt.Errorf("NewStorageBackend: error executing endpoint template: %w", err)
 	}
+	normalizedEndpoint := fmt.Sprintf("%s/", strings.TrimSuffix(ep.String(), "/"))
 
 	var client *azblob.Client
 	if opts.PrimaryAccountKey != "" {
@@ -53,7 +55,7 @@ func NewStorageBackend(opts Config, logFunc storage.Log) (storage.Backend, error
 			return nil, fmt.Errorf("NewStorageBackend: error creating shared key Azure credential: %w", err)
 		}
 
-		client, err = azblob.NewClientWithSharedKeyCredential(ep.String(), cred, nil)
+		client, err = azblob.NewClientWithSharedKeyCredential(normalizedEndpoint, cred, nil)
 		if err != nil {
 			return nil, fmt.Errorf("NewStorageBackend: error creating Azure client: %w", err)
 		}
@@ -62,7 +64,7 @@ func NewStorageBackend(opts Config, logFunc storage.Log) (storage.Backend, error
 		if err != nil {
 			return nil, fmt.Errorf("NewStorageBackend: error creating managed identity credential: %w", err)
 		}
-		client, err = azblob.NewClient(ep.String(), cred, nil)
+		client, err = azblob.NewClient(normalizedEndpoint, cred, nil)
 		if err != nil {
 			return nil, fmt.Errorf("NewStorageBackend: error creating Azure client: %w", err)
 		}

test/azure/docker-compose.yml

@@ -4,7 +4,7 @@ services:
   storage:
     image: mcr.microsoft.com/azure-storage/azurite
     volumes:
-      - ./foo:/data
+      - azurite_backup_data:/data
     command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data
     healthcheck:
         test: nc 127.0.0.1 10000 -z

test/certs/run.sh

@@ -24,10 +24,10 @@ openssl x509 -req -passin pass:test \
 
 openssl x509 -in minio.crt -noout -text
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 sleep 5
 
@@ -40,4 +40,4 @@ docker run --rm -it \
 
 pass "Found relevant files in untared remote backups."
 
-docker-compose down --volumes
+docker compose down --volumes

test/commands/run.sh

@@ -6,10 +6,10 @@ cd $(dirname $0)
 . ../util.sh
 current_test=$(basename $(pwd))
 
-docker-compose up -d
+docker compose up -d
 sleep 30 # mariadb likes to take a bit before responding
 
-docker-compose exec backup backup
+docker compose exec backup backup
 sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' commands_archive) ./local
 
 tar -xvf ./local/test.tar.gz
@@ -28,7 +28,7 @@ if [ -f ./backup/data/post.txt ]; then
 fi
 pass "Did not find unexpected file."
 
-docker-compose down --volumes
+docker compose down --volumes
 sudo rm -rf ./local
 
 

test/confd/run.sh

@@ -8,12 +8,12 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 
 # sleep until a backup is guaranteed to have happened on the 1 minute schedule
 sleep 100
 
-docker-compose down --volumes
+docker compose down --volumes
 
 if [ ! -f ./local/conf.tar.gz ]; then
   fail "Config from file was not used."

test/extend/Dockerfile

@@ -0,0 +1,4 @@
+ARG version=canary
+FROM offen/docker-volume-backup:$version
+
+RUN apk add rsync

test/extend/docker-compose.yml

@@ -0,0 +1,26 @@
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:${TEST_VERSION:-canary}
+    restart: always
+    labels:
+      - docker-volume-backup.copy-post=/bin/sh -c 'mkdir -p /tmp/unpack && tar -xvf $$COMMAND_RUNTIME_ARCHIVE_FILEPATH -C /tmp/unpack && rsync -r /tmp/unpack/backup/app_data /local'
+    environment:
+      BACKUP_FILENAME: test.tar.gz
+      BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
+      EXEC_FORWARD_OUTPUT: "true"
+    volumes:
+      - ./local:/local
+      - app_data:/backup/app_data:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+
+  offen:
+    image: offen/offen:latest
+    labels:
+      - docker-volume-backup.stop-during-backup=true
+    volumes:
+      - app_data:/var/opt/offen
+
+volumes:
+  app_data:

test/extend/run.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -e
+
+cd "$(dirname "$0")"
+. ../util.sh
+current_test=$(basename $(pwd))
+
+mkdir -p local
+
+export TEST_VERSION="${TEST_VERSION:-canary}-with-rsync"
+
+docker build . -t offen/docker-volume-backup:$TEST_VERSION
+
+docker compose up -d
+sleep 5
+
+docker compose exec backup backup
+
+sleep 5
+
+expect_running_containers "2"
+
+if [ ! -f "./local/app_data/offen.db" ]; then
+  fail "Could not find expected file in untared archive."
+fi
+
+docker compose down --volumes

test/gpg/run.sh

@@ -8,10 +8,10 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 expect_running_containers "2"
 
@@ -30,4 +30,4 @@ if [ ! -L ./local/test-latest.tar.gz.gpg ]; then
   fail "Could not find local symlink to latest encrypted backup."
 fi
 
-docker-compose down --volumes
+docker compose down --volumes

test/ignore/run.sh

@@ -8,11 +8,11 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 sleep 5
-docker-compose exec backup backup
+docker compose exec backup backup
 
-docker-compose down --volumes
+docker compose down --volumes
 
 out=$(mktemp -d)
 sudo tar --same-owner -xvf ./local/test.tar.gz -C "$out"

test/local/run.sh

@@ -8,13 +8,13 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
 # A symlink for a known file in the volume is created so the test can check
 # whether symlinks are preserved on backup.
-docker-compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
-docker-compose exec backup backup
+docker compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
+docker compose exec backup backup
 
 sleep 5
 
@@ -42,14 +42,14 @@ pass "Found symlink to latest version in local backup."
 # The second part of this test checks if backups get deleted when the retention
 # is set to 0 days (which it should not as it would mean all backups get deleted)
 # TODO: find out if we can test actual deletion without having to wait for a day
-BACKUP_RETENTION_DAYS="0" docker-compose up -d
+BACKUP_RETENTION_DAYS="0" docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 if [ "$(find ./local -type f | wc -l)" != "1" ]; then
   fail "Backups should not have been deleted, instead seen: "$(find ./local -type f)""
 fi
 pass "Local backups have not been deleted."
 
-docker-compose down --volumes
+docker compose down --volumes

test/notifications/run.sh

@@ -8,13 +8,13 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
 GOTIFY_TOKEN=$(curl -sSLX POST -H 'Content-Type: application/json' -d '{"name":"test"}' http://admin:custom@localhost:8080/application | jq -r '.token')
 info "Set up Gotify application using token $GOTIFY_TOKEN"
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
 if [ "$NUM_MESSAGES" != 0 ]; then
@@ -22,11 +22,11 @@ if [ "$NUM_MESSAGES" != 0 ]; then
 fi
 pass "No notifications were sent when not configured."
 
-docker-compose down
+docker compose down
 
-NOTIFICATION_URLS="gotify://gotify/${GOTIFY_TOKEN}?disableTLS=true" docker-compose up -d
+NOTIFICATION_URLS="gotify://gotify/${GOTIFY_TOKEN}?disableTLS=true" docker compose up -d
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 NUM_MESSAGES=$(curl -sSL http://admin:custom@localhost:8080/message | jq -r '.messages | length')
 if [ "$NUM_MESSAGES" != 1 ]; then
@@ -47,4 +47,4 @@ if [ "$MESSAGE_BODY" != "Backing up /tmp/test.tar.gz succeeded." ]; then
 fi
 pass "Custom notification body was used."
 
-docker-compose down --volumes
+docker compose down --volumes

test/ownership/run.sh

@@ -9,10 +9,10 @@ current_test=$(basename $(pwd))
 
 mkdir -p local
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 tmp_dir=$(mktemp -d)
 sudo tar --same-owner -xvf ./local/backup.tar.gz -C $tmp_dir
@@ -27,4 +27,4 @@ for file in $(sudo find $tmp_dir/backup/postgres); do
 done
 pass "All files and directories in backup preserved their ownership."
 
-docker-compose down --volumes
+docker compose down --volumes

test/s3/run.sh

@@ -6,12 +6,12 @@ cd "$(dirname "$0")"
 . ../util.sh
 current_test=$(basename $(pwd))
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
 # A symlink for a known file in the volume is created so the test can check
 # whether symlinks are preserved on backup.
-docker-compose exec backup backup
+docker compose exec backup backup
 
 sleep 5
 
@@ -27,10 +27,10 @@ pass "Found relevant files in untared remote backups."
 # The second part of this test checks if backups get deleted when the retention
 # is set to 0 days (which it should not as it would mean all backups get deleted)
 # TODO: find out if we can test actual deletion without having to wait for a day
-BACKUP_RETENTION_DAYS="0" docker-compose up -d
+BACKUP_RETENTION_DAYS="0" docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 docker run --rm -it \
   -v minio_backup_data:/minio_data \
@@ -39,4 +39,4 @@ docker run --rm -it \
 
 pass "Remote backups have not been deleted."
 
-docker-compose down --volumes
+docker compose down --volumes

test/ssh/run.sh

@@ -8,10 +8,10 @@ current_test=$(basename $(pwd))
 
 ssh-keygen -t rsa -m pem -b 4096 -N "test1234" -f id_rsa -C "docker-volume-backup@local"
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 sleep 5
 
@@ -27,10 +27,10 @@ pass "Found relevant files in decrypted and untared remote backups."
 # The second part of this test checks if backups get deleted when the retention
 # is set to 0 days (which it should not as it would mean all backups get deleted)
 # TODO: find out if we can test actual deletion without having to wait for a day
-BACKUP_RETENTION_DAYS="0" docker-compose up -d
+BACKUP_RETENTION_DAYS="0" docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 docker run --rm -it \
   -v ssh_backup_data:/ssh_data \
@@ -39,5 +39,5 @@ docker run --rm -it \
 
 pass "Remote backups have not been deleted."
 
-docker-compose down --volumes
+docker compose down --volumes
 rm -f id_rsa id_rsa.pub

test/webdav/run.sh

@@ -6,10 +6,10 @@ cd "$(dirname "$0")"
 . ../util.sh
 current_test=$(basename $(pwd))
 
-docker-compose up -d
+docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 sleep 5
 
@@ -25,10 +25,10 @@ pass "Found relevant files in untared remote backup."
 # The second part of this test checks if backups get deleted when the retention
 # is set to 0 days (which it should not as it would mean all backups get deleted)
 # TODO: find out if we can test actual deletion without having to wait for a day
-BACKUP_RETENTION_DAYS="0" docker-compose up -d
+BACKUP_RETENTION_DAYS="0" docker compose up -d
 sleep 5
 
-docker-compose exec backup backup
+docker compose exec backup backup
 
 docker run --rm -it \
   -v webdav_backup_data:/webdav_data \
@@ -37,4 +37,4 @@ docker run --rm -it \
 
 pass "Remote backups have not been deleted."
 
-docker-compose down --volumes
+docker compose down --volumes