Compare commits

...

88 Commits

Author SHA1 Message Date
Frederik Ring
1db896f7cf Tweak README, improve client naming, tidy go.mod file 2022-01-22 13:35:13 +01:00
Kaerbr
6ded00aa06 Support Nextcloud / WebDav (#48)
* add studio-b12/gowebdav to be able to upload to webdav server

* make sure all env variables are present for webdav upload

* implement file upload to WebDav server

directory defaults to the base directory

* docs: add the new feature to the documentation

* if no WebDav env variable are given throw no error

* docs: use more elegant english :D

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* docs: use official spelling of "WebDAV"

* perf: golang likes to return early instead of having an else block

* use WEBDAV_PATH instead of WEBDAV_DIRECTORY

* use split_words for more convenience

like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support

* simplify

* feat: add pruning of files in WebDAV remote

Based on / Inspired by the minio/S3 implementation of pruning remote files.

* remove logging from the development

* test: first try implementing tests

Sandly I have to use the remote pipeline -- local wont work for me.

* test: adapt used volume names

* test: specify image only once!

* test: minio AND webdav data should be present

* test: backups on WebDAV remote are laying in the root directory

* test: the webdav server stores date in /var/lib/dav

* trying with data subfolder

* test: 1 container was added so the number raised from 3 to 4

* webdav  subfolder is "data" not "backup"

* fix: password AND username must be defined

not password OR username

* improve logging

* feat: if the given path on the server isnt preset it will be created

* test: add creation of new folder for webdav to tests

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2022-01-22 13:29:21 +01:00
Hendrik Niefeld
6b79f1914b Update README.md 2022-01-06 16:09:33 +01:00
Hendrik Niefeld
40ff2e00c9 Update README.md 2022-01-06 16:07:00 +01:00
Frederik Ring
760cc9cebc Add issue template 2021-12-29 13:10:43 +01:00
Frederik Ring
1f9582df51 Fix handling of empty directories (#44)
* Add test checking whether empty directories are included in backups

* Update targz library to include fix
2021-12-29 10:10:12 +01:00
Frederik Ring
32575c831e Also expand env vars in pruning prefix if configured 2021-12-23 09:22:56 +01:00
Frederik Ring
c062710ce8 Allow for env substitution in backup filename (#39) 2021-12-22 14:39:46 +01:00
Frederik Ring
3a7dfe8e60 Add note about double quoting issue in older compose versions 2021-12-18 13:24:14 +01:00
Frederik Ring
9ec33510e7 Extend docs on notifications 2021-12-18 10:31:12 +01:00
Frederik Ring
4207146fb6 Refactor calling of hooks on exit 2021-12-18 10:31:12 +01:00
Frederik Ring
1f727f698f Run hooks in order of severity 2021-12-18 10:31:12 +01:00
Frederik Ring
88c90a206c Use int comparison for checking hooks 2021-12-18 10:31:12 +01:00
Frederik Ring
8bad0656b3 Enable notifications on multiple levels 2021-12-18 10:31:12 +01:00
Frederik Ring
08d78a0bd6 allow sending notifications to multiple channels 2021-12-18 10:31:12 +01:00
Frederik Ring
5a6ce81b58 update github.com/otai/copy, use PreserveOwner option 2021-11-29 08:40:55 +01:00
Frederik Ring
dfd0d617e4 install bugfix releases where available 2021-11-28 20:12:23 +01:00
Frederik Ring
7bc5b2ccef fix minor error scoping mistakes 2021-11-28 20:06:24 +01:00
Frederik Ring
b6ad624115 leverage docker cache for downloading go deps 2021-11-23 08:04:48 +01:00
Frederik Ring
210c7d4540 Reuse hook mechanism for scheduling clean up tasks (#33)
* reuse hook mechanism for scheduling clean up tasks

* register hooks before creating files or dirs

* fix logging order

* use typed hook levels
2021-11-08 19:10:10 +01:00
Frederik Ring
3c06bf8102 run cli test using BACKUP_FROM_SNAPSHOT 2021-11-08 08:44:59 +01:00
schwannden
411c39ee72 create a snapshot before creating tar archive (#32)
* create a snapshot before creating tar archive

* safeguard snapshot removal and make snapshot optional

* fix typo, make sure remove snapshot failure triggers failure hook

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-11-08 08:39:18 +01:00
Frederik Ring
0c666d0c88 use lstat when checking whether file is a symlink 2021-11-03 18:07:55 +01:00
Frederik Ring
a0402b407d fix fileinfo mode comparison when checking for symlinks 2021-11-03 18:03:44 +01:00
Frederik Ring
3193e88fc0 os.FileInfo cannot be used for deleting files as it does not contain a full path 2021-11-02 06:40:37 +01:00
Frederik Ring
c391230be6 Merge pull request #31 from offen/exclude-symlink-candidates
Exclude symlinks from candidates when pruning local files
2021-10-31 20:07:51 +01:00
Frederik Ring
f946f36fb0 exclude symlinks from candidates when pruning local files
Previously, symlinks would be included in the set of candidates, but would
be skipped when pruning. This could lead to a wrong number of candidates
being printed in the log messages.
2021-10-29 09:00:37 +02:00
Frederik Ring
5245b5882f update README, save some indentation 2021-10-28 19:55:39 +02:00
schwannden
7f0f173115 adding option to skip tls verification error (#30)
* adding option to skip tls verification error

* merge options

* removed merged option from README

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-10-28 19:51:35 +02:00
Frederik Ring
ad7ec58322 add syntax highlighting 2021-10-23 17:45:57 +02:00
Frederik Ring
b7ab2fbacc add section about container timezones to the README 2021-10-23 17:44:30 +02:00
Frederik Ring
789fc656e8 Merge pull request #27 from offen/latest-symlink
Automatically create symlink to latest local backup if configured
2021-10-01 18:47:16 +02:00
Frederik Ring
c59b40f2df automatically create symlink to latest local backup if configured 2021-10-01 18:19:24 +02:00
Frederik Ring
cff418e735 fix README grammar 2021-10-01 08:48:20 +02:00
Frederik Ring
d7ccdd79fc Merge pull request #26 from offen/instance-profile
Allow s3 authentication via IAM role
2021-09-30 19:32:54 +02:00
Frederik Ring
bd73a2b5e4 allow s3 authentication via IAM role 2021-09-30 19:24:43 +02:00
Frederik Ring
6cf5cf47e7 Merge pull request #25 from offen/delete-on-failure
Ensure script always tries to remove local artifacts even when backup failed
2021-09-13 09:33:12 +02:00
Frederik Ring
53c257065e ensure script always tries to remove local artifacts even when backup failed 2021-09-12 10:48:19 +02:00
Frederik Ring
184b7a1e18 add docs on one off backups using docker cli 2021-09-11 11:21:48 +02:00
Frederik Ring
69a94f226b tweak configuration reference for email settings 2021-09-10 11:58:33 +02:00
Frederik Ring
160a47e90b allow registering hooks at different levels 2021-09-09 16:55:49 +02:00
Frederik Ring
59660ec5c7 include exit log message in notification 2021-09-09 11:08:05 +02:00
Frederik Ring
af3e69b7a8 fix typo in README 2021-09-09 09:19:37 +02:00
Frederik Ring
5d400cb943 Merge pull request #24 from offen/failure-email
Enable sending out email notifications on failed backups
2021-09-09 09:10:20 +02:00
Frederik Ring
88368197c1 implement email notifications on failed backup runs 2021-09-09 09:00:23 +02:00
Frederik Ring
e46968ed79 call error hooks on script failure 2021-09-09 08:12:07 +02:00
Frederik Ring
2c06f81503 collect all log output in buffer so it could be used in notifications 2021-09-09 07:24:18 +02:00
Frederik Ring
55d030a06a Merge pull request #22 from offen/targz-fork
Fix handling of symlinks in backup targets
2021-09-06 18:15:34 +02:00
Frederik Ring
fefc34c6aa tidy go mod file 2021-09-04 15:54:09 +02:00
Frederik Ring
5922820ada add test for checking behavior on symlinks 2021-09-04 10:30:34 +02:00
Frederik Ring
8aba98c012 use forked version of package targz 2021-09-04 10:08:06 +02:00
Frederik Ring
70daa0308a Merge pull request #19 from offen/golang-version
v2 Rewrite
2021-08-30 19:57:36 +02:00
Frederik Ring
ede94bcd88 display all error messages instead of first one 2021-08-29 19:39:51 +02:00
Frederik Ring
aae97a5617 try restarting even when stopping some containers failed 2021-08-29 18:51:05 +02:00
Frederik Ring
825cbb50ef always use background context directly 2021-08-29 18:26:40 +02:00
Frederik Ring
bea203af3d improve documentation 2021-08-29 18:16:04 +02:00
Frederik Ring
6034e6a902 print proper local archive in log message 2021-08-29 08:36:45 +02:00
Frederik Ring
d0eca0a179 fix container stop execution order 2021-08-26 16:22:24 +02:00
Frederik Ring
a0fe2cf42d handle errors on container restart 2021-08-26 12:50:22 +02:00
Frederik Ring
5334ff1a5a refactor script initialization 2021-08-25 07:48:20 +02:00
Frederik Ring
e73256ad70 do not use start time as deadline 2021-08-24 09:15:43 +02:00
Frederik Ring
e0c4adc563 move handling of config to script layer 2021-08-24 09:01:44 +02:00
Frederik Ring
2469597848 fix lockfile mechanism 2021-08-23 18:46:49 +02:00
Frederik Ring
b1c4bee85d use buffered reader to write to encryption mechanism 2021-08-23 17:34:13 +02:00
Frederik Ring
ec87bd27e7 do not use scanner to write file in chunks 2021-08-23 15:19:50 +02:00
Frederik Ring
f4f4fa9e74 use full filepath when pruning local backups 2021-08-23 14:56:04 +02:00
Frederik Ring
7086c6e645 read backup in small chunks when encrypting 2021-08-23 14:48:33 +02:00
Frederik Ring
411a62a6c7 shorten log messages 2021-08-23 14:48:33 +02:00
Frederik Ring
5a2bf48ec6 make sure backup also runs when socket isn't present 2021-08-23 14:48:33 +02:00
Frederik Ring
07b06cf0ba read all configuration in init 2021-08-23 14:48:33 +02:00
Frederik Ring
4c80494433 use go native strftime version 2021-08-23 14:48:33 +02:00
Frederik Ring
7244725c5b fix location of success message for having created local backup 2021-08-23 14:48:33 +02:00
Frederik Ring
935de92f2e only tag proper releases as latest 2021-08-23 14:48:33 +02:00
Frederik Ring
d195e8967f improve logging messages 2021-08-23 14:48:33 +02:00
Frederik Ring
188c14c00f add insecure option, update docs 2021-08-23 14:48:33 +02:00
Frederik Ring
da9458724f adapt repo layout to go 2021-08-23 14:48:33 +02:00
Frederik Ring
435583168b add logging 2021-08-23 14:48:33 +02:00
Frederik Ring
67499d776c refactor deferred cleanup actions to always run 2021-08-23 14:48:33 +02:00
Frederik Ring
8c99ec0bdf implement pruning from remote storage 2021-08-23 14:48:33 +02:00
Frederik Ring
f2739b583e add gpg encryption 2021-08-23 14:48:32 +02:00
Frederik Ring
78e4e3813b implement deletion of local backups 2021-08-23 14:48:10 +02:00
Frederik Ring
4d9482a8b4 implement lock file to ensure backup runs mutually exclusive 2021-08-23 14:47:34 +02:00
Frederik Ring
0c6ac05789 implement copy to remote storage 2021-08-23 14:47:34 +02:00
Frederik Ring
8b110fd789 scaffold script flow 2021-08-23 14:47:34 +02:00
Frederik Ring
efb52aa806 try porting docker related parts to golang 2021-08-23 14:47:34 +02:00
Frederik Ring
4c84674650 Merge pull request #20 from offen/gpg-testcase
Add testcase for gpg encryption
2021-08-23 14:47:11 +02:00
Frederik Ring
6fe81cdf2d add testcase for gpg encryption 2021-08-23 14:42:50 +02:00
Frederik Ring
b7ba0e08df prefix mtime param with a +, use -name param for passing pattern to find 2021-08-20 21:51:45 +02:00
15 changed files with 2830 additions and 392 deletions

View File

@@ -11,6 +11,10 @@ jobs:
name: Build name: Build
command: | command: |
docker build . -t offen/docker-volume-backup:canary docker build . -t offen/docker-volume-backup:canary
- run:
name: Install gnupg
command: |
sudo apt-get install -y gnupg
- run: - run:
name: Run tests name: Run tests
working_directory: ~/docker-volume-backup/test working_directory: ~/docker-volume-backup/test
@@ -39,10 +43,13 @@ jobs:
docker context create docker-volume-backup docker context create docker-volume-backup
docker buildx create docker-volume-backup --name docker-volume-backup --use docker buildx create docker-volume-backup --name docker-volume-backup --use
docker buildx inspect --bootstrap docker buildx inspect --bootstrap
tag_args="-t offen/docker-volume-backup:$CIRCLE_TAG"
if [[ "$CIRCLE_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
# prerelease tags like `v2.0.0-alpha.1` should not be released as `latest`
tag_args="$tag_args -t offen/docker-volume-backup:latest"
fi
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 \ docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 \
-t offen/docker-volume-backup:$CIRCLE_TAG \ $tag_args . --push
-t offen/docker-volume-backup:latest \
. --push
workflows: workflows:
version: 2 version: 2

18
.editorconfig Normal file
View File

@@ -0,0 +1,18 @@
# EditorConfig is awesome: http://EditorConfig.org
# top-most EditorConfig file
root = true
# Unix-style newlines with a newline ending every file
[*]
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 2
[*.md]
trim_trailing_whitespace = false
[*.go]
indent_style = tab

20
.github/ISSUE_TEMPLATE.md vendored Normal file
View File

@@ -0,0 +1,20 @@
* **I'm submitting a ...**
- [ ] bug report
- [ ] feature request
- [ ] support request
* **What is the current behavior?**
* **If the current behavior is a bug, please provide the configuration and steps to reproduce and if possible a minimal demo of the problem.**
* **What is the expected behavior?**
* **What is the motivation / use case for changing the behavior?**
* **Please tell us about your environment:**
- Image version:
- Docker version:
- docker-compose version:
* **Other information** (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)

View File

@@ -2,22 +2,22 @@
# SPDX-License-Identifier: MPL-2.0 # SPDX-License-Identifier: MPL-2.0
FROM golang:1.17-alpine as builder FROM golang:1.17-alpine as builder
ARG MC_VERSION=RELEASE.2021-06-13T17-48-22Z
RUN go install -ldflags "-X github.com/minio/mc/cmd.ReleaseTag=$MC_VERSION" github.com/minio/mc@$MC_VERSION WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/backup/main.go ./cmd/backup/main.go
RUN go build -o backup cmd/backup/main.go
FROM alpine:3.14 FROM alpine:3.14
WORKDIR /root WORKDIR /root
RUN apk add --update ca-certificates docker openrc gnupg RUN apk add --update ca-certificates
RUN update-ca-certificates
RUN rc-update add docker boot
COPY --from=builder /go/bin/mc /usr/bin/mc COPY --from=builder /app/backup /usr/bin/backup
RUN mc --version
COPY src/backup.sh src/entrypoint.sh /root/ COPY ./entrypoint.sh /root/
RUN chmod +x backup.sh && mv backup.sh /usr/bin/backup \ RUN chmod +x entrypoint.sh
&& chmod +x entrypoint.sh
ENTRYPOINT ["/root/entrypoint.sh"] ENTRYPOINT ["/root/entrypoint.sh"]

22
NOTICE
View File

@@ -1,22 +0,0 @@
Copyright 2021 Offen Authors <hioffen@posteo.de>
This project contains software that is Copyright (c) 2018 Futurice
Licensed under the Terms of the MIT License:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

776
README.md
View File

@@ -1,115 +1,55 @@
<a href="https://www.offen.dev/">
<img src="https://offen.github.io/press-kit/offen-material/gfx-GitHub-Offen-logo.svg" alt="Offen logo" title="Offen" width="150px"/>
</a>
# docker-volume-backup # docker-volume-backup
Backup Docker volumes locally or to any S3 compatible storage. Backup Docker volumes locally or to any S3 compatible storage.
The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a sidecar container to an existing Docker setup. It handles recurring backups of Docker volumes to a local directory or any S3 compatible storage (or both) and rotates away old backups if configured. The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3 or WebDAV compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
## Configuration <!-- MarkdownTOC -->
Backup targets, schedule and retention are configured in environment variables: - [Quickstart](#quickstart)
- [Recurring backups in a compose setup](#recurring-backups-in-a-compose-setup)
- [One-off backups using Docker CLI](#one-off-backups-using-docker-cli)
- [Configuration reference](#configuration-reference)
- [How to](#how-to)
- [Stopping containers during backup](#stopping-containers-during-backup)
- [Automatically pruning old backups](#automatically-pruning-old-backups)
- [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs)
- [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg)
- [Restoring a volume from a backup](#restoring-a-volume-from-a-backup)
- [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in)
- [Using with Docker Swarm](#using-with-docker-swarm)
- [Manually triggering a backup](#manually-triggering-a-backup)
- [Update deprecated email configuration](#update-deprecated-email-configuration)
- [Recipes](#recipes)
- [Backing up to AWS S3](#backing-up-to-aws-s3)
- [Backing up to MinIO](#backing-up-to-minio)
- [Backing up to WebDAV](#backing-up-to-webdav)
- [Backing up locally](#backing-up-locally)
- [Backing up to AWS S3 as well as locally](#backing-up-to-aws-s3-as-well-as-locally)
- [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
- [Rotating away backups that are older than 7 days](#rotating-away-backups-that-are-older-than-7-days)
- [Encrypting your backups using GPG](#encrypting-your-backups-using-gpg)
- [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup)
- [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup)
```ini <!-- /MarkdownTOC -->
########### BACKUP SCHEDULE
# Backups run on the given cron schedule and use the filename defined in the ---
# template expression.
BACKUP_CRON_EXPRESSION="0 2 * * *" Code and documentation for `v1` versions are found on [this branch][v1-branch].
BACKUP_FILENAME="backup-%Y-%m-%dT%H-%M-%S.tar.gz"
########### BACKUP STORAGE [v1-branch]: https://github.com/offen/docker-volume-backup/tree/v1
# Define credentials for authenticating against the backup storage and a bucket ## Quickstart
# name. Although all of these values are `AWS`-prefixed, the setup can be used
# with any S3 compatible storage.
AWS_ACCESS_KEY_ID="<xxx>" ### Recurring backups in a compose setup
AWS_SECRET_ACCESS_KEY="<xxx>"
AWS_S3_BUCKET_NAME="<xxx>"
# This is the FQDN of your storage server, e.g. `storage.example.com`. Add a `backup` service to your compose setup and mount the volumes you would like to see backed up:
# Do not set this when working against AWS S3. If you need to set a
# specific protocol, you will need to use the option below.
# AWS_ENDPOINT="<xxx>"
# The protocol to be used when communicating with your storage server.
# Defaults to "https". You can set this to "http" when communicating with
# a different Docker container on the same host for example.
# AWS_ENDPOINT_PROTO="https"
# In addition to backing up you can also store backups locally. Pass in
# a local path to store your backups here if needed. You likely want to
# mount a local folder or Docker volume into that location when running
# the container. Local paths can also be subject to pruning of old
# backups as defined below.
# BACKUP_ARCHIVE="/archive"
########### BACKUP PRUNING
# **IMPORTANT, PLEASE READ THIS BEFORE USING THIS FEATURE**:
# The mechanism used for pruning backups is not very sophisticated
# and applies its rules to **all files in the target directory** by default,
# which means that if you are storing your backups next to other files,
# these might become subject to deletion too. When using this option
# make sure the backup files are stored in a directory used exclusively
# for storing them or to configure BACKUP_PRUNING_PREFIX to limit
# removal to certain files.
# Define this value to enable automatic pruning of old backups. The value
# declares the number of days for which a backup is kept.
# BACKUP_RETENTION_DAYS="7"
# In case the duration a backup takes fluctuates noticeably in your setup
# you can adjust this setting to make sure there are no race conditions
# between the backup finishing and the pruning not deleting backups that
# sit on the very edge of the time window. Set this value to a duration
# that is expected to be bigger than the maximum difference of backups.
# Valid values have a suffix of (s)econds, (m)inutes, (h)ours, or (d)ays.
# BACKUP_PRUNING_LEEWAY="10m"
# In case your target bucket or directory contains other files than the ones
# managed by this container, you can limit the scope of rotation by setting
# a prefix value. This would usually be the non-parametrized part of your
# BACKUP_FILENAME. E.g. if BACKUP_FILENAME is `db-backup-%Y-%m-%dT%H-%M-%S.tar.gz`,
# you can set BACKUP_PRUNING_PREFIX to `db-backup-` and make sure
# unrelated files are not affected.
# BACKUP_PRUNING_PREFIX="backup-"
########### BACKUP ENCRYPTION
# Backups can be encrypted using gpg in case a passphrase is given
# GPG_PASSPHRASE="<xxx>"
########### STOPPING CONTAINERS DURING BACKUP
# Containers can be stopped by applying a
# `docker-volume-backup.stop-during-backup` label. By default, all containers
# that are labeled with `true` will be stopped. If you need more fine grained
# control (e.g. when running multiple containers based on this image), you can
# override this default by specifying a different value here.
# BACKUP_STOP_CONTAINER_LABEL="service1"
########### MINIO CLIENT CONFIGURATION
# Pass these additional flags to all MinIO client `mc` invocations.
# This can be used for example to pass `--insecure` when using self
# signed certificates, or passing `--debug` to gain insights on
# unexpected behavior.
# MC_GLOBAL_OPTIONS="<xxx>"
```
## Example in a docker-compose setup
Most likely, you will use this image as a sidecar container in an existing docker-compose setup like this:
```yml ```yml
version: '3' version: '3'
@@ -127,26 +67,399 @@ services:
- docker-volume-backup.stop-during-backup=true - docker-volume-backup.stop-during-backup=true
backup: backup:
# In production, it is advised to lock your image tag to a proper
# release version instead of using `latest`.
# Check https://github.com/offen/docker-volume-backup/releases
# for a list of available releases.
image: offen/docker-volume-backup:latest image: offen/docker-volume-backup:latest
restart: always restart: always
env_file: ./backup.env env_file: ./backup.env # see below for configuration reference
volumes: volumes:
- data:/backup/my-app-backup:ro
# Mounting the Docker socket allows the script to stop and restart # Mounting the Docker socket allows the script to stop and restart
# the container during backup. You can omit this if you don't want # the container during backup. You can omit this if you don't want
# to stop the container # to stop the container
- /var/run/docker.sock:/var/run/docker.sock:ro - /var/run/docker.sock:/var/run/docker.sock:ro
- data:/backup/my-app-backup:ro
# If you mount a local directory or volume to `/archive` a local # If you mount a local directory or volume to `/archive` a local
# copy of the backup will be stored there. You can override the # copy of the backup will be stored there. You can override the
# location inside of the container by setting `BACKUP_ARCHIVE` # location inside of the container by setting `BACKUP_ARCHIVE`.
# - /path/to/local_backups:/archive # You can omit this if you do not want to keep local backups.
- /path/to/local_backups:/archive
volumes: volumes:
data: data:
``` ```
## Using with Docker Swarm ### One-off backups using Docker CLI
By default, Docker Swarm will restart stopped containers automatically, even when manually stopped. If you plan to have your containers / services stopped during backup, this means you need to apply the `on-failure` restart policy to your service's definitions. A restart policy of `always` is not compatible with this tool. To run a one time backup, mount the volume you would like to see backed up into a container and run the `backup` command:
```console
docker run --rm \
-v data:/backup/data \
--env AWS_ACCESS_KEY_ID="<xxx>" \
--env AWS_SECRET_ACCESS_KEY="<xxx>" \
--env AWS_S3_BUCKET_NAME="<xxx>" \
--entrypoint backup \
offen/docker-volume-backup:latest
```
Alternatively, pass a `--env-file` in order to use a full config as described below.
## Configuration reference
Backup targets, schedule and retention are configured in environment variables.
You can populate below template according to your requirements and use it as your `env_file`:
```ini
########### BACKUP SCHEDULE
# Backups run on the given cron schedule in `busybox` flavor. If no
# value is set, `@daily` will be used. If you do not want the cron
# to ever run, use `0 0 5 31 2 ?`.
# BACKUP_CRON_EXPRESSION="0 2 * * *"
# The name of the backup file including the `.tar.gz` extension.
# Format verbs will be replaced as in `strftime`. Omitting them
# will result in the same filename for every backup run, which means previous
# versions will be overwritten on subsequent runs. The default results
# in filenames like `backup-2021-08-29T04-00-00.tar.gz`.
# BACKUP_FILENAME="backup-%Y-%m-%dT%H-%M-%S.tar.gz"
# Setting BACKUP_FILENAME_EXPAND to true allows for environment variable
# placeholders in BACKUP_FILENAME, BACKUP_LATEST_SYMLINK and in
# BACKUP_PRUNING_PREFIX that will get expanded at runtime,
# e.g. `backup-$HOSTNAME-%Y-%m-%dT%H-%M-%S.tar.gz`. Expansion happens before
# interpolating strftime tokens. It is disabled by default.
# Please note that you will need to escape the `$` when providing the value
# in a docker-compose.yml file, i.e. using $$VAR instead of $VAR.
# BACKUP_FILENAME_TEMPLATE="true"
# When storing local backups, a symlink to the latest backup can be created
# in case a value is given for this key. This has no effect on remote backups.
# BACKUP_LATEST_SYMLINK="backup.latest.tar.gz"
# Whether to copy the content of backup folder before creating the tar archive.
# In the rare scenario where the content of the source backup volume is continously
# updating, but we do not wish to stop the container while performing the backup,
# this setting can be used to ensure the integrity of the tar.gz file.
# BACKUP_FROM_SNAPSHOT="false"
########### BACKUP STORAGE
# The name of the remote bucket that should be used for storing backups. If
# this is not set, no remote backups will be stored.
# AWS_S3_BUCKET_NAME="backup-bucket"
# Define credentials for authenticating against the backup storage and a bucket
# name. Although all of these keys are `AWS`-prefixed, the setup can be used
# with any S3 compatible storage.
# AWS_ACCESS_KEY_ID="<xxx>"
# AWS_SECRET_ACCESS_KEY="<xxx>"
# Instead of providing static credentials, you can also use IAM instance profiles
# or similar to provide authentication. Some possible configuration options on AWS:
# - EC2: http://169.254.169.254
# - ECS: http://169.254.170.2
# AWS_IAM_ROLE_ENDPOINT="http://169.254.169.254"
# This is the FQDN of your storage server, e.g. `storage.example.com`.
# Do not set this when working against AWS S3 (the default value is
# `s3.amazonaws.com`). If you need to set a specific (non-https) protocol, you
# will need to use the option below.
# AWS_ENDPOINT="storage.example.com"
# The protocol to be used when communicating with your storage server.
# Defaults to "https". You can set this to "http" when communicating with
# a different Docker container on the same host for example.
# AWS_ENDPOINT_PROTO="https"
# Setting this variable to `true` will disable verification of
# SSL certificates. You shouldn't use this unless you use self-signed
# certificates for your remote storage backend. This can only be used
# when AWS_ENDPOINT_PROTO is set to `https`.
# AWS_ENDPOINT_INSECURE="true"
# You can also backup files to any WebDAV server:
# The URL of the remote WebDAV server
# WEBDAV_URL="https://webdav.example.com"
# The Directory to place the backups to on the WebDAV server.
# If the path is not present on the server it will be created.
# WEBDAV_PATH="/my/directory/"
# The username for the WebDAV server
# WEBDAV_USERNAME="user"
# The password for the WebDAV server
# WEBDAV_PASSWORD="password"
# In addition to storing backups remotely, you can also keep local copies.
# Pass a container-local path to store your backups if needed. You also need to
# mount a local folder or Docker volume into that location (`/archive`
# by default) when running the container. In case the specified directory does
# not exist (nothing is mounted) in the container when the backup is running,
# local backups will be skipped. Local paths are also be subject to pruning of
# old backups as defined below.
# BACKUP_ARCHIVE="/archive"
########### BACKUP PRUNING
# **IMPORTANT, PLEASE READ THIS BEFORE USING THIS FEATURE**:
# The mechanism used for pruning old backups is not very sophisticated
# and applies its rules to **all files in the target directory** by default,
# which means that if you are storing your backups next to other files,
# these might become subject to deletion too. When using this option
# make sure the backup files are stored in a directory used exclusively
# for such files, or to configure BACKUP_PRUNING_PREFIX to limit
# removal to certain files.
# Define this value to enable automatic rotation of old backups. The value
# declares the number of days for which a backup is kept.
# BACKUP_RETENTION_DAYS="7"
# In case the duration a backup takes fluctuates noticeably in your setup
# you can adjust this setting to make sure there are no race conditions
# between the backup finishing and the rotation not deleting backups that
# sit on the edge of the time window. Set this value to a duration
# that is expected to be bigger than the maximum difference of backups.
# Valid values have a suffix of (s)econds, (m)inutes or (h)ours. By default,
# one minute is used.
# BACKUP_PRUNING_LEEWAY="1m"
# In case your target bucket or directory contains other files than the ones
# managed by this container, you can limit the scope of rotation by setting
# a prefix value. This would usually be the non-parametrized part of your
# BACKUP_FILENAME. E.g. if BACKUP_FILENAME is `db-backup-%Y-%m-%dT%H-%M-%S.tar.gz`,
# you can set BACKUP_PRUNING_PREFIX to `db-backup-` and make sure
# unrelated files are not affected by the rotation mechanism.
# BACKUP_PRUNING_PREFIX="backup-"
########### BACKUP ENCRYPTION
# Backups can be encrypted using gpg in case a passphrase is given.
# GPG_PASSPHRASE="<xxx>"
########### STOPPING CONTAINERS DURING BACKUP
# Containers can be stopped by applying a
# `docker-volume-backup.stop-during-backup` label. By default, all containers
# that are labeled with `true` will be stopped. If you need more fine grained
# control (e.g. when running multiple containers based on this image), you can
# override this default by specifying a different value here.
# BACKUP_STOP_CONTAINER_LABEL="service1"
########### NOTIFICATIONS
# Notifications (email, Slack, etc.) can be sent out when a backup run finishes.
# Configuration is provided as a comma-separated list of URLs as consumed
# by `shoutrrr`: https://containrrr.dev/shoutrrr/v0.5/services/overview/
# When providing multiple URLs or an URL that contains a comma, the values
# can be URL encoded to avoid ambiguities.
# The below URL demonstrates how to send an email using the provided SMTP
# configuration and credentials.
# NOTIFICATION_URLS=smtp://username:password@host:587/?fromAddress=sender@example.com&toAddresses=recipient@example.com
# By default, notifications would only be sent out when a backup run fails
# To receive notifications for every run, set `NOTIFICATION_LEVEL` to `info`
# instead of the default `error`.
# NOTIFICATION_LEVEL="error"
########### EMAIL NOTIFICATIONS
# ************************************************************************
# Providing notification configuration like this has been deprecated
# and will be removed in the next major version. Please use NOTIFICATION_URLS
# as documented above instead.
# ************************************************************************
# In case SMTP credentials are provided, notification emails can be sent out when
# a backup run finished. These emails will contain the start time, the error
# message on failure and all prior log output.
# The recipient(s) of the notification. Supply a comma separated list
# of adresses if you want to notify multiple recipients. If this is
# not set, no emails will be sent.
# EMAIL_NOTIFICATION_RECIPIENT="you@example.com"
# The "From" header of the sent email. Defaults to `noreply@nohost`.
# EMAIL_NOTIFICATION_SENDER="no-reply@example.com"
# Configuration and credentials for the SMTP server to be used.
# EMAIL_SMTP_PORT defaults to 587.
# EMAIL_SMTP_HOST="posteo.de"
# EMAIL_SMTP_PASSWORD="<xxx>"
# EMAIL_SMTP_USERNAME="no-reply@example.com"
# EMAIL_SMTP_PORT="<port>"
```
In case you encouter double quoted values in your configuration you might be running an [older version of `docker-compose`].
You can work around this by either updating `docker-compose` or unquoting your configuration values.
[compose-issue]: https://github.com/docker/compose/issues/2854
## How to
### Stopping containers during backup
In many cases, it will be desirable to stop the services that are consuming the volume you want to backup in order to ensure data integrity.
This image can automatically stop and restart containers and services (in case you are running Docker in Swarm mode).
By default, any container that is labeled `docker-volume-backup.stop-during-backup=true` will be stopped before the backup is being taken and restarted once it has finished.
In case you need more fine grained control about which containers should be stopped (e.g. when backing up multiple volumes on different schedules), you can set the `BACKUP_STOP_CONTAINER_LABEL` environment variable and then use the same value for labeling:
```yml
version: '3'
services:
app:
# definition for app ...
labels:
- docker-volume-backup.stop-during-backup=service1
backup:
image: offen/docker-volume-backup:latest
environment:
BACKUP_STOP_CONTAINER_LABEL: service1
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Automatically pruning old backups
When `BACKUP_RETENTION_DAYS` is configured, the image will check if there are any backups in the remote bucket or local archive that are older than the given retention value and rotate these backups away.
Be aware that this mechanism looks at __all files in the target bucket or archive__, which means that other files that are older than the given deadline are deleted as well. In case you need to use a target that cannot be used exclusively for your backups, you can configure `BACKUP_PRUNING_PREFIX` to limit which files are considered eligible for deletion:
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
BACKUP_PRUNING_PREFIX: backup-
BACKUP_RETENTION_DAYS: 7
volumes:
- ${HOME}/backups:/archive
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Send email notifications on failed backup runs
To send out email notifications on failed backup runs, provide SMTP credentials, a sender and a recipient:
```yml
version: '3'
services:
backup:
image: offen/docker-volume-backup:latest
environment:
# ... other configuration values go here
NOTIFICATION_URLS=smtp://me:secret@smtp.example.com:587/?fromAddress=no-reply@example.com&toAddresses=you@example.com
```
Notification backends other than email are also supported.
Refer to the documentation of [shoutrrr][shoutrrr-docs] to find out about options and configuration.
[shoutrrr-docs]: https://containrrr.dev/shoutrrr/v0.5/services/overview/
### Encrypting your backup using GPG
The image supports encrypting backups using GPG out of the box.
In case a `GPG_PASSPHRASE` environment variable is set, the backup will be encrypted using the given key and saved as a `.gpg` file instead.
Assuming you have `gpg` installed, you can decrypt such a backup using (your OS will prompt for the passphrase before decryption can happen):
```console
gpg -o backup.tar.gz -d backup.tar.gz.gpg
```
### Restoring a volume from a backup
In case you need to restore a volume from a backup, the most straight forward procedure to do so would be:
- Stop the container(s) that are using the volume
- Untar the backup you want to restore
```console
tar -C /tmp -xvf backup.tar.gz
```
- Using a temporary one-off container, mount the volume (the example assumes it's named `data`) and copy over the backup. Make sure you copy the correct path level (this depends on how you mount your volume into the backup container), you might need to strip some leading elements
```console
docker run -d --name backup_restore -v data:/backup_restore alpine
docker cp /tmp/backup/data-backup backup_restore:/backup_restore
docker stop backup_restore
docker rm backup_restore
```
- Restart the container(s) that are using the volume
Depending on your setup and the application(s) you are running, this might involve other steps to be taken still.
### Set the timezone the container runs in
By default a container based on this image will run in the UTC timezone.
As the image is designed to be as small as possible, additional timezone data is not included.
In case you want to run your cron rules in your local timezone (respecting DST and similar), you can mount your Docker host's `/etc/timezone` and `/etc/localtime` in read-only mode:
```yml
version: '3'
services:
backup:
image: offen/docker-volume-backup:latest
volumes:
- data:/backup/my-app-backup:ro
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
volumes:
data:
```
### Using with Docker Swarm
By default, Docker Swarm will restart stopped containers automatically, even when manually stopped.
If you plan to have your containers / services stopped during backup, this means you need to apply the `on-failure` restart policy to your service's definitions.
A restart policy of `always` is not compatible with this tool.
--- ---
@@ -162,7 +475,7 @@ services:
memory: 25M memory: 25M
``` ```
## Manually triggering a backup ### Manually triggering a backup
You can manually trigger a backup run outside of the defined cron schedule by executing the `backup` command inside the container: You can manually trigger a backup run outside of the defined cron schedule by executing the `backup` command inside the container:
@@ -170,15 +483,254 @@ You can manually trigger a backup run outside of the defined cron schedule by ex
docker exec <container_ref> backup docker exec <container_ref> backup
``` ```
--- ### Update deprecated email configuration
Starting with version 2.6.0, configuring email notifications using `EMAIL_*` keys has been deprecated.
Instead of providing multiple values using multiple keys, you can now provide a single URL for `NOTIFICATION_URLS`.
Before:
```ini
EMAIL_NOTIFICATION_RECIPIENT="you@example.com"
EMAIL_NOTIFICATION_SENDER="no-reply@example.com"
EMAIL_SMTP_HOST="posteo.de"
EMAIL_SMTP_PASSWORD="secret"
EMAIL_SMTP_USERNAME="me"
EMAIL_SMTP_PORT="587"
```
After:
```ini
NOTIFICATION_URLS=smtp://me:secret@posteo.de:587/?fromAddress=no-reply@example.com&toAddresses=you@example.com
```
## Recipes
This section lists configuration for some real-world use cases that you can mix and match according to your needs.
### Backing up to AWS S3
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Backing up to MinIO
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
AWS_ENDPOINT: minio.example.com
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: MINIOACCESSKEY
AWS_SECRET_ACCESS_KEY: MINIOSECRETKEY
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Backing up to WebDAV
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
WEBDAV_URL: https://webdav.mydomain.me
WEBDAV_PATH: /my/directory/
WEBDAV_USERNAME: user
WEBDAV_PASSWORD: password
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Backing up locally
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
BACKUP_LATEST_SYMLINK: backup-latest.tar.gz
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${HOME}/backups:/archive
volumes:
data:
```
### Backing up to AWS S3 as well as locally
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${HOME}/backups:/archive
volumes:
data:
```
### Running on a custom cron schedule
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
# take a backup on every hour
BACKUP_CRON_EXPRESSION: "0 * * * *"
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Rotating away backups that are older than 7 days
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
BACKUP_FILENAME: backup-%Y-%m-%dT%H-%M-%S.tar.gz
BACKUP_PRUNING_PREFIX: backup-
BACKUP_RETENTION_DAYS: 7
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Encrypting your backups using GPG
```yml
version: '3'
services:
# ... define other services using the `data` volume here
backup:
image: offen/docker-volume-backup:latest
environment:
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
GPG_PASSPHRASE: somesecretstring
volumes:
- data:/backup/my-app-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data:
```
### Running multiple instances in the same setup
```yml
version: '3'
services:
# ... define other services using the `data_1` and `data_2` volumes here
backup_1: &backup_service
image: offen/docker-volume-backup:latest
environment: &backup_environment
BACKUP_CRON_EXPRESSION: "0 2 * * *"
AWS_BUCKET_NAME: backup-bucket
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Label the container using the `data_1` volume as `docker-volume-backup.stop-during-backup=service1`
BACKUP_STOP_CONTAINER_LABEL: service1
volumes:
- data_1:/backup/data-1-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
backup_2:
<<: *backup_service
environment:
<<: *backup_environment
# Label the container using the `data_2` volume as `docker-volume-backup.stop-during-backup=service2`
BACKUP_CRON_EXPRESSION: "0 3 * * *"
BACKUP_STOP_CONTAINER_LABEL: service2
volumes:
- data_2:/backup/data-2-backup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
data_1:
data_2:
```
## Differences to `futurice/docker-volume-backup` ## Differences to `futurice/docker-volume-backup`
This image is heavily inspired by the `futurice/docker-volume-backup`. We decided to publish this image as a simpler and more lightweight alternative because of the following requirements: This image is heavily inspired by `futurice/docker-volume-backup`. We decided to publish this image as a simpler and more lightweight alternative because of the following requirements:
- The original image is based on `ubuntu`, making it very heavy. This version is roughly 1/3 in compressed size. - The original image is based on `ubuntu` and requires additional tools, making it heavy.
- This image makes use of the MinIO client `mc` instead of the full blown AWS CLI for uploading backups. This version is roughly 1/25 in compressed size (it's ~12MB).
- The original image proposed to handle backup rotation through AWS S3 lifecycle policies. This image adds the option to rotate old backups through the same script so this functionality can also be offered for non-AWS storage backends like MinIO. - The original image uses a shell script, when this version is written in Go, which makes it easier to extend and maintain (more verbose too).
- InfluxDB specific functionality was removed. - The original image proposed to handle backup rotation through AWS S3 lifecycle policies.
This image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO.
Local copies of backups can also be pruned once they reach a certain age.
- InfluxDB specific functionality from the original image was removed.
- `arm64` and `arm/v7` architectures are supported. - `arm64` and `arm/v7` architectures are supported.
- Docker in Swarm mode is supported. - Docker in Swarm mode is supported.
- Notifications on failed backups are supported
- IAM authentication through instance profiles is supported

895
cmd/backup/main.go Normal file
View File

@@ -0,0 +1,895 @@
// Copyright 2021 - Offen Authors <hioffen@posteo.de>
// SPDX-License-Identifier: MPL-2.0
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"sort"
"strings"
"time"
"github.com/containrrr/shoutrrr"
"github.com/containrrr/shoutrrr/pkg/router"
sTypes "github.com/containrrr/shoutrrr/pkg/types"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/api/types/swarm"
"github.com/docker/docker/client"
"github.com/gofrs/flock"
"github.com/kelseyhightower/envconfig"
"github.com/leekchan/timeutil"
"github.com/m90/targz"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
"github.com/otiai10/copy"
"github.com/sirupsen/logrus"
"github.com/studio-b12/gowebdav"
"golang.org/x/crypto/openpgp"
)
func main() {
unlock := lock("/var/lock/dockervolumebackup.lock")
defer unlock()
s, err := newScript()
if err != nil {
panic(err)
}
defer func() {
if pArg := recover(); pArg != nil {
if err, ok := pArg.(error); ok {
if hookErr := s.runHooks(err); hookErr != nil {
s.logger.Errorf("An error occurred calling the registered hooks: %s", hookErr)
}
os.Exit(1)
}
panic(pArg)
}
if err := s.runHooks(nil); err != nil {
s.logger.Errorf(
"Backup procedure ran successfully, but an error ocurred calling the registered hooks: %v",
err,
)
os.Exit(1)
}
s.logger.Info("Finished running backup tasks.")
}()
s.must(func() error {
restartContainers, err := s.stopContainers()
// The mechanism for restarting containers is not using hooks as it
// should happen as soon as possible (i.e. before uploading backups or
// similar).
defer func() {
s.must(restartContainers())
}()
if err != nil {
return err
}
return s.takeBackup()
}())
s.must(s.encryptBackup())
s.must(s.copyBackup())
s.must(s.pruneOldBackups())
}
// script holds all the stateful information required to orchestrate a
// single backup run.
type script struct {
cli *client.Client
minioClient *minio.Client
webdavClient *gowebdav.Client
logger *logrus.Logger
sender *router.ServiceRouter
hooks []hook
hookLevel hookLevel
start time.Time
file string
output *bytes.Buffer
c *config
}
type config struct {
BackupSources string `split_words:"true" default:"/backup"`
BackupFilename string `split_words:"true" default:"backup-%Y-%m-%dT%H-%M-%S.tar.gz"`
BackupFilenameExpand bool `split_words:"true"`
BackupLatestSymlink string `split_words:"true"`
BackupArchive string `split_words:"true" default:"/archive"`
BackupRetentionDays int32 `split_words:"true" default:"-1"`
BackupPruningLeeway time.Duration `split_words:"true" default:"1m"`
BackupPruningPrefix string `split_words:"true"`
BackupStopContainerLabel string `split_words:"true" default:"true"`
BackupFromSnapshot bool `split_words:"true"`
AwsS3BucketName string `split_words:"true"`
AwsEndpoint string `split_words:"true" default:"s3.amazonaws.com"`
AwsEndpointProto string `split_words:"true" default:"https"`
AwsEndpointInsecure bool `split_words:"true"`
AwsAccessKeyID string `envconfig:"AWS_ACCESS_KEY_ID"`
AwsSecretAccessKey string `split_words:"true"`
AwsIamRoleEndpoint string `split_words:"true"`
GpgPassphrase string `split_words:"true"`
NotificationURLs []string `envconfig:"NOTIFICATION_URLS"`
NotificationLevel string `split_words:"true" default:"error"`
EmailNotificationRecipient string `split_words:"true"`
EmailNotificationSender string `split_words:"true" default:"noreply@nohost"`
EmailSMTPHost string `envconfig:"EMAIL_SMTP_HOST"`
EmailSMTPPort int `envconfig:"EMAIL_SMTP_PORT" default:"587"`
EmailSMTPUsername string `envconfig:"EMAIL_SMTP_USERNAME"`
EmailSMTPPassword string `envconfig:"EMAIL_SMTP_PASSWORD"`
WebdavUrl string `split_words:"true"`
WebdavPath string `split_words:"true" default:"/"`
WebdavUsername string `split_words:"true"`
WebdavPassword string `split_words:"true"`
}
var msgBackupFailed = "backup run failed"
// newScript creates all resources needed for the script to perform actions against
// remote resources like the Docker engine or remote storage locations. All
// reading from env vars or other configuration sources is expected to happen
// in this method.
func newScript() (*script, error) {
stdOut, logBuffer := buffer(os.Stdout)
s := &script{
c: &config{},
logger: &logrus.Logger{
Out: stdOut,
Formatter: new(logrus.TextFormatter),
Hooks: make(logrus.LevelHooks),
Level: logrus.InfoLevel,
},
start: time.Now(),
output: logBuffer,
}
if err := envconfig.Process("", s.c); err != nil {
return nil, fmt.Errorf("newScript: failed to process configuration values: %w", err)
}
s.file = path.Join("/tmp", s.c.BackupFilename)
if s.c.BackupFilenameExpand {
s.file = os.ExpandEnv(s.file)
s.c.BackupLatestSymlink = os.ExpandEnv(s.c.BackupLatestSymlink)
s.c.BackupPruningPrefix = os.ExpandEnv(s.c.BackupPruningPrefix)
}
s.file = timeutil.Strftime(&s.start, s.file)
_, err := os.Stat("/var/run/docker.sock")
if !os.IsNotExist(err) {
cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
if err != nil {
return nil, fmt.Errorf("newScript: failed to create docker client")
}
s.cli = cli
}
if s.c.AwsS3BucketName != "" {
var creds *credentials.Credentials
if s.c.AwsAccessKeyID != "" && s.c.AwsSecretAccessKey != "" {
creds = credentials.NewStaticV4(
s.c.AwsAccessKeyID,
s.c.AwsSecretAccessKey,
"",
)
} else if s.c.AwsIamRoleEndpoint != "" {
creds = credentials.NewIAM(s.c.AwsIamRoleEndpoint)
} else {
return nil, errors.New("newScript: AWS_S3_BUCKET_NAME is defined, but no credentials were provided")
}
options := minio.Options{
Creds: creds,
Secure: s.c.AwsEndpointProto == "https",
}
if s.c.AwsEndpointInsecure {
if !options.Secure {
return nil, errors.New("newScript: AWS_ENDPOINT_INSECURE = true is only meaningful for https")
}
transport, err := minio.DefaultTransport(true)
if err != nil {
return nil, fmt.Errorf("newScript: failed to create default minio transport")
}
transport.TLSClientConfig.InsecureSkipVerify = true
options.Transport = transport
}
mc, err := minio.New(s.c.AwsEndpoint, &options)
if err != nil {
return nil, fmt.Errorf("newScript: error setting up minio client: %w", err)
}
s.minioClient = mc
}
if s.c.WebdavUrl != "" {
if s.c.WebdavUsername == "" || s.c.WebdavPassword == "" {
return nil, errors.New("newScript: WEBDAV_URL is defined, but no credentials were provided")
} else {
webdavClient := gowebdav.NewClient(s.c.WebdavUrl, s.c.WebdavUsername, s.c.WebdavPassword)
s.webdavClient = webdavClient
}
}
if s.c.EmailNotificationRecipient != "" {
emailURL := fmt.Sprintf(
"smtp://%s:%s@%s:%d/?from=%s&to=%s",
s.c.EmailSMTPUsername,
s.c.EmailSMTPPassword,
s.c.EmailSMTPHost,
s.c.EmailSMTPPort,
s.c.EmailNotificationSender,
s.c.EmailNotificationRecipient,
)
s.c.NotificationURLs = append(s.c.NotificationURLs, emailURL)
s.logger.Warn(
"Using EMAIL_* keys for providing notification configuration has been deprecated and will be removed in the next major version.",
)
s.logger.Warn(
"Please use NOTIFICATION_URLS instead. Refer to the README for an upgrade guide.",
)
}
hookLevel, ok := hookLevels[s.c.NotificationLevel]
if !ok {
return nil, fmt.Errorf("newScript: unknown NOTIFICATION_LEVEL %s", s.c.NotificationLevel)
}
s.hookLevel = hookLevel
if len(s.c.NotificationURLs) > 0 {
sender, senderErr := shoutrrr.CreateSender(s.c.NotificationURLs...)
if senderErr != nil {
return nil, fmt.Errorf("newScript: error creating sender: %w", senderErr)
}
s.sender = sender
// To prevent duplicate notifications, ensure the regsistered callbacks
// run mutually exclusive.
s.registerHook(hookLevelError, func(err error) error {
if err == nil {
return nil
}
return s.notifyFailure(err)
})
s.registerHook(hookLevelInfo, func(err error) error {
if err != nil {
return nil
}
return s.notifySuccess()
})
}
return s, nil
}
var noop = func() error { return nil }
// registerHook adds the given action at the given level.
func (s *script) registerHook(level hookLevel, action func(err error) error) {
s.hooks = append(s.hooks, hook{level, action})
}
// notifyFailure sends a notification about a failed backup run
func (s *script) notifyFailure(err error) error {
body := fmt.Sprintf(
"Running docker-volume-backup failed with error: %s\n\nLog output of the failed run was:\n\n%s\n", err, s.output.String(),
)
title := fmt.Sprintf("Failure running docker-volume-backup at %s", s.start.Format(time.RFC3339))
if err := s.sendNotification(title, body); err != nil {
return fmt.Errorf("notifyFailure: error notifying: %w", err)
}
return nil
}
// notifyFailure sends a notification about a successful backup run
func (s *script) notifySuccess() error {
title := fmt.Sprintf("Success running docker-volume-backup at %s", s.start.Format(time.RFC3339))
body := fmt.Sprintf(
"Running docker-volume-backup succeeded.\n\nLog output was:\n\n%s\n", s.output.String(),
)
if err := s.sendNotification(title, body); err != nil {
return fmt.Errorf("notifySuccess: error notifying: %w", err)
}
return nil
}
// sendNotification sends a notification to all configured third party services
func (s *script) sendNotification(title, body string) error {
var errs []error
for _, result := range s.sender.Send(body, &sTypes.Params{"title": title}) {
if result != nil {
errs = append(errs, result)
}
}
if len(errs) != 0 {
return fmt.Errorf("sendNotification: error sending message: %w", join(errs...))
}
return nil
}
// stopContainers stops all Docker containers that are marked as to being
// stopped during the backup and returns a function that can be called to
// restart everything that has been stopped.
func (s *script) stopContainers() (func() error, error) {
if s.cli == nil {
return noop, nil
}
allContainers, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
Quiet: true,
})
if err != nil {
return noop, fmt.Errorf("stopContainersAndRun: error querying for containers: %w", err)
}
containerLabel := fmt.Sprintf(
"docker-volume-backup.stop-during-backup=%s",
s.c.BackupStopContainerLabel,
)
containersToStop, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
Quiet: true,
Filters: filters.NewArgs(filters.KeyValuePair{
Key: "label",
Value: containerLabel,
}),
})
if err != nil {
return noop, fmt.Errorf("stopContainersAndRun: error querying for containers to stop: %w", err)
}
if len(containersToStop) == 0 {
return noop, nil
}
s.logger.Infof(
"Stopping %d container(s) labeled `%s` out of %d running container(s).",
len(containersToStop),
containerLabel,
len(allContainers),
)
var stoppedContainers []types.Container
var stopErrors []error
for _, container := range containersToStop {
if err := s.cli.ContainerStop(context.Background(), container.ID, nil); err != nil {
stopErrors = append(stopErrors, err)
} else {
stoppedContainers = append(stoppedContainers, container)
}
}
var stopError error
if len(stopErrors) != 0 {
stopError = fmt.Errorf(
"stopContainersAndRun: %d error(s) stopping containers: %w",
len(stopErrors),
join(stopErrors...),
)
}
return func() error {
servicesRequiringUpdate := map[string]struct{}{}
var restartErrors []error
for _, container := range stoppedContainers {
if swarmServiceName, ok := container.Labels["com.docker.swarm.service.name"]; ok {
servicesRequiringUpdate[swarmServiceName] = struct{}{}
continue
}
if err := s.cli.ContainerStart(context.Background(), container.ID, types.ContainerStartOptions{}); err != nil {
restartErrors = append(restartErrors, err)
}
}
if len(servicesRequiringUpdate) != 0 {
services, _ := s.cli.ServiceList(context.Background(), types.ServiceListOptions{})
for serviceName := range servicesRequiringUpdate {
var serviceMatch swarm.Service
for _, service := range services {
if service.Spec.Name == serviceName {
serviceMatch = service
break
}
}
if serviceMatch.ID == "" {
return fmt.Errorf("stopContainersAndRun: couldn't find service with name %s", serviceName)
}
serviceMatch.Spec.TaskTemplate.ForceUpdate = 1
if _, err := s.cli.ServiceUpdate(
context.Background(), serviceMatch.ID,
serviceMatch.Version, serviceMatch.Spec, types.ServiceUpdateOptions{},
); err != nil {
restartErrors = append(restartErrors, err)
}
}
}
if len(restartErrors) != 0 {
return fmt.Errorf(
"stopContainersAndRun: %d error(s) restarting containers and services: %w",
len(restartErrors),
join(restartErrors...),
)
}
s.logger.Infof(
"Restarted %d container(s) and the matching service(s).",
len(stoppedContainers),
)
return nil
}, stopError
}
// takeBackup creates a tar archive of the configured backup location and
// saves it to disk.
func (s *script) takeBackup() error {
backupSources := s.c.BackupSources
if s.c.BackupFromSnapshot {
backupSources = filepath.Join("/tmp", s.c.BackupSources)
// copy before compressing guard against a situation where backup folder's content are still growing.
s.registerHook(hookLevelPlumbing, func(error) error {
if err := remove(backupSources); err != nil {
return fmt.Errorf("takeBackup: error removing snapshot: %w", err)
}
s.logger.Infof("Removed snapshot `%s`.", backupSources)
return nil
})
if err := copy.Copy(s.c.BackupSources, backupSources, copy.Options{
PreserveTimes: true,
PreserveOwner: true,
}); err != nil {
return fmt.Errorf("takeBackup: error creating snapshot: %w", err)
}
s.logger.Infof("Created snapshot of `%s` at `%s`.", s.c.BackupSources, backupSources)
}
tarFile := s.file
s.registerHook(hookLevelPlumbing, func(error) error {
if err := remove(tarFile); err != nil {
return fmt.Errorf("takeBackup: error removing tar file: %w", err)
}
s.logger.Infof("Removed tar file `%s`.", tarFile)
return nil
})
if err := targz.Compress(backupSources, tarFile); err != nil {
return fmt.Errorf("takeBackup: error compressing backup folder: %w", err)
}
s.logger.Infof("Created backup of `%s` at `%s`.", backupSources, tarFile)
return nil
}
// encryptBackup encrypts the backup file using PGP and the configured passphrase.
// In case no passphrase is given it returns early, leaving the backup file
// untouched.
func (s *script) encryptBackup() error {
if s.c.GpgPassphrase == "" {
return nil
}
gpgFile := fmt.Sprintf("%s.gpg", s.file)
s.registerHook(hookLevelPlumbing, func(error) error {
if err := remove(gpgFile); err != nil {
return fmt.Errorf("encryptBackup: error removing gpg file: %w", err)
}
s.logger.Infof("Removed GPG file `%s`.", gpgFile)
return nil
})
outFile, err := os.Create(gpgFile)
defer outFile.Close()
if err != nil {
return fmt.Errorf("encryptBackup: error opening out file: %w", err)
}
_, name := path.Split(s.file)
dst, err := openpgp.SymmetricallyEncrypt(outFile, []byte(s.c.GpgPassphrase), &openpgp.FileHints{
IsBinary: true,
FileName: name,
}, nil)
defer dst.Close()
if err != nil {
return fmt.Errorf("encryptBackup: error encrypting backup file: %w", err)
}
src, err := os.Open(s.file)
if err != nil {
return fmt.Errorf("encryptBackup: error opening backup file `%s`: %w", s.file, err)
}
if _, err := io.Copy(dst, src); err != nil {
return fmt.Errorf("encryptBackup: error writing ciphertext to file: %w", err)
}
s.file = gpgFile
s.logger.Infof("Encrypted backup using given passphrase, saving as `%s`.", s.file)
return nil
}
// copyBackup makes sure the backup file is copied to both local and remote locations
// as per the given configuration.
func (s *script) copyBackup() error {
_, name := path.Split(s.file)
if s.minioClient != nil {
if _, err := s.minioClient.FPutObject(context.Background(), s.c.AwsS3BucketName, name, s.file, minio.PutObjectOptions{
ContentType: "application/tar+gzip",
}); err != nil {
return fmt.Errorf("copyBackup: error uploading backup to remote storage: %w", err)
}
s.logger.Infof("Uploaded a copy of backup `%s` to bucket `%s`.", s.file, s.c.AwsS3BucketName)
}
if s.webdavClient != nil {
bytes, err := os.ReadFile(s.file)
if err != nil {
return fmt.Errorf("copyBackup: error reading the file to be uploaded: %w", err)
}
if err := s.webdavClient.MkdirAll(s.c.WebdavPath, 0644); err != nil {
return fmt.Errorf("copyBackup: error creating directory '%s' on WebDAV server: %w", s.c.WebdavPath, err)
}
if err := s.webdavClient.Write(filepath.Join(s.c.WebdavPath, name), bytes, 0644); err != nil {
return fmt.Errorf("copyBackup: error uploading the file to WebDAV server: %w", err)
}
s.logger.Infof("Uploaded a copy of backup `%s` to WebDAV-URL '%s' at path '%s'.", s.file, s.c.WebdavUrl, s.c.WebdavPath)
}
if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
if err := copyFile(s.file, path.Join(s.c.BackupArchive, name)); err != nil {
return fmt.Errorf("copyBackup: error copying file to local archive: %w", err)
}
s.logger.Infof("Stored copy of backup `%s` in local archive `%s`.", s.file, s.c.BackupArchive)
if s.c.BackupLatestSymlink != "" {
symlink := path.Join(s.c.BackupArchive, s.c.BackupLatestSymlink)
if _, err := os.Lstat(symlink); err == nil {
os.Remove(symlink)
}
if err := os.Symlink(name, symlink); err != nil {
return fmt.Errorf("copyBackup: error creating latest symlink: %w", err)
}
s.logger.Infof("Created/Updated symlink `%s` for latest backup.", s.c.BackupLatestSymlink)
}
}
return nil
}
// pruneOldBackups rotates away backups from local and remote storages using
// the given configuration. In case the given configuration would delete all
// backups, it does nothing instead.
func (s *script) pruneOldBackups() error {
if s.c.BackupRetentionDays < 0 {
return nil
}
if s.c.BackupPruningLeeway != 0 {
s.logger.Infof("Sleeping for %s before pruning backups.", s.c.BackupPruningLeeway)
time.Sleep(s.c.BackupPruningLeeway)
}
deadline := time.Now().AddDate(0, 0, -int(s.c.BackupRetentionDays))
// Prune minio/S3 backups
if s.minioClient != nil {
candidates := s.minioClient.ListObjects(context.Background(), s.c.AwsS3BucketName, minio.ListObjectsOptions{
WithMetadata: true,
Prefix: s.c.BackupPruningPrefix,
})
var matches []minio.ObjectInfo
var lenCandidates int
for candidate := range candidates {
lenCandidates++
if candidate.Err != nil {
return fmt.Errorf(
"pruneOldBackups: error looking up candidates from remote storage: %w",
candidate.Err,
)
}
if candidate.LastModified.Before(deadline) {
matches = append(matches, candidate)
}
}
if len(matches) != 0 && len(matches) != lenCandidates {
objectsCh := make(chan minio.ObjectInfo)
go func() {
for _, match := range matches {
objectsCh <- match
}
close(objectsCh)
}()
errChan := s.minioClient.RemoveObjects(context.Background(), s.c.AwsS3BucketName, objectsCh, minio.RemoveObjectsOptions{})
var removeErrors []error
for result := range errChan {
if result.Err != nil {
removeErrors = append(removeErrors, result.Err)
}
}
if len(removeErrors) != 0 {
return fmt.Errorf(
"pruneOldBackups: %d error(s) removing files from remote storage: %w",
len(removeErrors),
join(removeErrors...),
)
}
s.logger.Infof(
"Pruned %d out of %d remote backup(s) as their age exceeded the configured retention period of %d days.",
len(matches),
lenCandidates,
s.c.BackupRetentionDays,
)
} else if len(matches) != 0 && len(matches) == lenCandidates {
s.logger.Warnf(
"The current configuration would delete all %d remote backup copies.",
len(matches),
)
s.logger.Warn("Refusing to do so, please check your configuration.")
} else {
s.logger.Infof("None of %d remote backup(s) were pruned.", lenCandidates)
}
}
// Prune WebDAV backups
if s.webdavClient != nil {
candidates, err := s.webdavClient.ReadDir(s.c.WebdavPath)
if err != nil {
return fmt.Errorf("pruneOldBackups: error looking up candidates from remote storage: %w", err)
}
var matches []fs.FileInfo
var lenCandidates int
for _, candidate := range candidates {
lenCandidates++
if candidate.ModTime().Before(deadline) {
matches = append(matches, candidate)
}
}
if len(matches) != 0 && len(matches) != lenCandidates {
for _, match := range matches {
if err := s.webdavClient.Remove(filepath.Join(s.c.WebdavPath, match.Name())); err != nil {
return fmt.Errorf("pruneOldBackups: error removing a file from remote storage: %w", err)
}
s.logger.Infof("Pruned %s from WebDAV: %s", match.Name(), filepath.Join(s.c.WebdavUrl, s.c.WebdavPath))
}
s.logger.Infof("Pruned %d out of %d remote backup(s) as their age exceeded the configured retention period of %d days.", len(matches), lenCandidates, s.c.BackupRetentionDays)
} else if len(matches) != 0 && len(matches) == lenCandidates {
s.logger.Warnf("The current configuration would delete all %d remote backup copies.", len(matches))
s.logger.Warn("Refusing to do so, please check your configuration.")
} else {
s.logger.Infof("None of %d remote backup(s) were pruned.", lenCandidates)
}
}
// Prune local backups
if _, err := os.Stat(s.c.BackupArchive); !os.IsNotExist(err) {
globPattern := path.Join(
s.c.BackupArchive,
fmt.Sprintf("%s*", s.c.BackupPruningPrefix),
)
globMatches, err := filepath.Glob(globPattern)
if err != nil {
return fmt.Errorf(
"pruneOldBackups: error looking up matching files using pattern %s: %w",
globPattern,
err,
)
}
var candidates []string
for _, candidate := range globMatches {
fi, err := os.Lstat(candidate)
if err != nil {
return fmt.Errorf(
"pruneOldBackups: error calling Lstat on file %s: %w",
candidate,
err,
)
}
if fi.Mode()&os.ModeSymlink != os.ModeSymlink {
candidates = append(candidates, candidate)
}
}
var matches []string
for _, candidate := range candidates {
fi, err := os.Stat(candidate)
if err != nil {
return fmt.Errorf(
"pruneOldBackups: error calling stat on file %s: %w",
candidate,
err,
)
}
if fi.ModTime().Before(deadline) {
matches = append(matches, candidate)
}
}
if len(matches) != 0 && len(matches) != len(candidates) {
var removeErrors []error
for _, match := range matches {
if err := os.Remove(match); err != nil {
removeErrors = append(removeErrors, err)
}
}
if len(removeErrors) != 0 {
return fmt.Errorf(
"pruneOldBackups: %d error(s) deleting local files, starting with: %w",
len(removeErrors),
join(removeErrors...),
)
}
s.logger.Infof(
"Pruned %d out of %d local backup(s) as their age exceeded the configured retention period of %d days.",
len(matches),
len(candidates),
s.c.BackupRetentionDays,
)
} else if len(matches) != 0 && len(matches) == len(candidates) {
s.logger.Warnf(
"The current configuration would delete all %d local backup copies.",
len(matches),
)
s.logger.Warn("Refusing to do so, please check your configuration.")
} else {
s.logger.Infof("None of %d local backup(s) were pruned.", len(candidates))
}
}
return nil
}
// runHooks runs all hooks that have been registered using the
// given levels in the defined ordering. In case executing a hook returns an
// error, the following hooks will still be run before the function returns.
func (s *script) runHooks(err error) error {
sort.SliceStable(s.hooks, func(i, j int) bool {
return s.hooks[i].level < s.hooks[j].level
})
var actionErrors []error
for _, hook := range s.hooks {
if hook.level > s.hookLevel {
continue
}
if actionErr := hook.action(err); actionErr != nil {
actionErrors = append(actionErrors, fmt.Errorf("runHooks: error running hook: %w", actionErr))
}
}
if len(actionErrors) != 0 {
return join(actionErrors...)
}
return nil
}
// must exits the script run prematurely in case the given error
// is non-nil.
func (s *script) must(err error) {
if err != nil {
s.logger.Errorf("Fatal error running backup: %s", err)
panic(err)
}
}
// remove removes the given file or directory from disk.
func remove(location string) error {
fi, err := os.Lstat(location)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("remove: error checking for existence of `%s`: %w", location, err)
}
if fi.IsDir() {
err = os.RemoveAll(location)
} else {
err = os.Remove(location)
}
if err != nil {
return fmt.Errorf("remove: error removing `%s`: %w", location, err)
}
return nil
}
// lock opens a lockfile at the given location, keeping it locked until the
// caller invokes the returned release func. When invoked while the file is
// still locked the function panics.
func lock(lockfile string) func() error {
fileLock := flock.New(lockfile)
acquired, err := fileLock.TryLock()
if err != nil {
panic(err)
}
if !acquired {
panic("unable to acquire file lock")
}
return fileLock.Unlock
}
// copy creates a copy of the file located at `dst` at `src`.
func copyFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return err
}
_, err = io.Copy(out, in)
if err != nil {
out.Close()
return err
}
return out.Close()
}
// join takes a list of errors and joins them into a single error
func join(errs ...error) error {
if len(errs) == 1 {
return errs[0]
}
var msgs []string
for _, err := range errs {
if err == nil {
continue
}
msgs = append(msgs, err.Error())
}
return errors.New("[" + strings.Join(msgs, ", ") + "]")
}
// buffer takes an io.Writer and returns a wrapped version of the
// writer that writes to both the original target as well as the returned buffer
func buffer(w io.Writer) (io.Writer, *bytes.Buffer) {
buffering := &bufferingWriter{buf: bytes.Buffer{}, writer: w}
return buffering, &buffering.buf
}
type bufferingWriter struct {
buf bytes.Buffer
writer io.Writer
}
func (b *bufferingWriter) Write(p []byte) (n int, err error) {
if n, err := b.buf.Write(p); err != nil {
return n, fmt.Errorf("bufferingWriter: error writing to buffer: %w", err)
}
return b.writer.Write(p)
}
// hook contains a queued action that can be trigger them when the script
// reaches a certain point (e.g. unsuccessful backup)
type hook struct {
level hookLevel
action func(err error) error
}
type hookLevel int
const (
hookLevelPlumbing hookLevel = iota
hookLevelError
hookLevelInfo
)
var hookLevels = map[string]hookLevel{
"info": hookLevelInfo,
"error": hookLevelError,
}

14
entrypoint.sh Normal file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# Copyright 2021 - Offen Authors <hioffen@posteo.de>
# SPDX-License-Identifier: MPL-2.0
set -e
BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"
echo "Installing cron.d entry with expression $BACKUP_CRON_EXPRESSION."
echo "$BACKUP_CRON_EXPRESSION backup 2>&1" | crontab -
echo "Starting cron in foreground."
crond -f -l 8

59
go.mod Normal file
View File

@@ -0,0 +1,59 @@
module github.com/offen/docker-volume-backup
go 1.17
require (
github.com/containrrr/shoutrrr v0.5.2
github.com/docker/docker v20.10.11+incompatible
github.com/gofrs/flock v0.8.1
github.com/kelseyhightower/envconfig v1.4.0
github.com/leekchan/timeutil v0.0.0-20150802142658-28917288c48d
github.com/m90/targz v0.0.0-20211229090208-2f22c2d9278e
github.com/minio/minio-go/v7 v7.0.16
github.com/otiai10/copy v1.7.0
github.com/sirupsen/logrus v1.8.1
github.com/studio-b12/gowebdav v0.0.0-20211109083228-3f8721cd4b6f
golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
)
require (
github.com/Microsoft/go-winio v0.4.17 // indirect
github.com/containerd/containerd v1.5.5 // indirect
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/fatih/color v1.10.0 // indirect
github.com/fsnotify/fsnotify v1.4.9 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.13.6 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/mattn/go-colorable v0.1.8 // indirect
github.com/mattn/go-isatty v0.0.12 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/nxadm/tail v1.4.6 // indirect
github.com/onsi/ginkgo v1.14.2 // indirect
github.com/onsi/gomega v1.10.3 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rs/xid v1.3.0 // indirect
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 // indirect
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1 // indirect
golang.org/x/text v0.3.6 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a // indirect
google.golang.org/grpc v1.33.2 // indirect
google.golang.org/protobuf v1.26.0 // indirect
gopkg.in/ini.v1 v1.65.0 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
)

1093
go.sum Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -1,176 +0,0 @@
#!/bin/sh
# Copyright 2021 - Offen Authors <hioffen@posteo.de>
# SPDX-License-Identifier: MPL-2.0
# Portions of this file are taken from github.com/futurice/docker-volume-backup
# See NOTICE for information about authors and licensing.
source env.sh
function info {
echo -e "\n[INFO] $1\n"
}
info "Preparing backup"
DOCKER_SOCK="/var/run/docker.sock"
if [ -S "$DOCKER_SOCK" ]; then
TEMPFILE="$(mktemp)"
docker ps -q \
--filter "label=docker-volume-backup.stop-during-backup=$BACKUP_STOP_CONTAINER_LABEL" \
> "$TEMPFILE"
CONTAINERS_TO_STOP="$(cat $TEMPFILE | tr '\n' ' ')"
CONTAINERS_TO_STOP_TOTAL="$(cat $TEMPFILE | wc -l)"
CONTAINERS_TOTAL="$(docker ps -q | wc -l)"
rm "$TEMPFILE"
echo "$CONTAINERS_TOTAL containers running on host in total."
echo "$CONTAINERS_TO_STOP_TOTAL containers marked to be stopped during backup."
else
CONTAINERS_TO_STOP_TOTAL="0"
CONTAINERS_TOTAL="0"
echo "Cannot access \"$DOCKER_SOCK\", won't look for containers to stop."
fi
if [ "$CONTAINERS_TO_STOP_TOTAL" != "0" ]; then
info "Stopping containers"
docker stop $CONTAINERS_TO_STOP
fi
info "Creating backup"
BACKUP_FILENAME="$(date +"$BACKUP_FILENAME")"
tar -czvf "$BACKUP_FILENAME" $BACKUP_SOURCES # allow the var to expand, in case we have multiple sources
if [ ! -z "$GPG_PASSPHRASE" ]; then
info "Encrypting backup"
gpg --symmetric --cipher-algo aes256 --batch --passphrase "$GPG_PASSPHRASE" \
-o "${BACKUP_FILENAME}.gpg" $BACKUP_FILENAME
rm $BACKUP_FILENAME
BACKUP_FILENAME="${BACKUP_FILENAME}.gpg"
fi
if [ "$CONTAINERS_TO_STOP_TOTAL" != "0" ]; then
info "Starting containers/services back up"
# The container might be part of a stack when running in swarm mode, so
# its parent service needs to be restarted instead once backup is finished.
SERVICES_REQUIRING_UPDATE=""
for CONTAINER_ID in $CONTAINERS_TO_STOP; do
SWARM_SERVICE_NAME=$(
docker inspect \
--format "{{ index .Config.Labels \"com.docker.swarm.service.name\" }}" \
$CONTAINER_ID
)
if [ -z "$SWARM_SERVICE_NAME" ]; then
echo "Restarting $(docker start $CONTAINER_ID)"
else
echo "Removing $(docker rm $CONTAINER_ID)"
# Multiple containers might belong to the same service, so they will
# be restarted only after all names are known.
SERVICES_REQUIRING_UPDATE="${SERVICES_REQUIRING_UPDATE} ${SWARM_SERVICE_NAME}"
fi
done
if [ -n "$SERVICES_REQUIRING_UPDATE" ]; then
for SERVICE_NAME in $(echo -n "$SERVICES_REQUIRING_UPDATE" | tr ' ' '\n' | sort -u); do
docker service update --force $SERVICE_NAME
done
fi
fi
copy_backup () {
mc cp $MC_GLOBAL_OPTIONS "$BACKUP_FILENAME" "$1"
}
if [ ! -z "$AWS_S3_BUCKET_NAME" ]; then
info "Uploading backup to remote storage"
echo "Will upload to bucket \"$AWS_S3_BUCKET_NAME\"."
copy_backup "backup-target/$AWS_S3_BUCKET_NAME"
echo "Upload finished."
fi
if [ -d "$BACKUP_ARCHIVE" ]; then
info "Copying backup to local archive"
echo "Will copy to \"$BACKUP_ARCHIVE\"."
copy_backup "$BACKUP_ARCHIVE"
echo "Finished copying."
fi
if [ -f "$BACKUP_FILENAME" ]; then
info "Cleaning up"
rm -vf "$BACKUP_FILENAME"
fi
info "Backup finished"
echo "Will wait for next scheduled backup."
probe_expired () {
local target=$1
local is_local=$2
if [ -z "$is_local" ]; then
mc rm $MC_GLOBAL_OPTIONS --fake --recursive --force \
--older-than "${BACKUP_RETENTION_DAYS}d" \
"$target"
else
find $target* -type f -mtime $BACKUP_RETENTION_DAYS
fi
}
probe_all () {
local target=$1
local is_local=$2
if [ -z "$is_local" ]; then
mc ls $MC_GLOBAL_OPTIONS "$target"
else
find $target* -type f
fi
}
delete () {
local target=$1
local is_local=$2
if [ -z "$is_local" ]; then
mc rm $MC_GLOBAL_OPTIONS --recursive --force \
--older-than "${BACKUP_RETENTION_DAYS}d" \
"$target"
else
find $target* -delete -type f -mtime $BACKUP_RETENTION_DAYS
fi
}
prune () {
local target=$1
local is_local=$2
if [ ! -z "$BACKUP_PRUNING_PREFIX" ]; then
target="$target/${BACKUP_PRUNING_PREFIX}"
fi
rule_applies_to=$(probe_expired "$target" "$is_local" | wc -l)
if [ "$rule_applies_to" == "0" ]; then
echo "No backups found older than the configured retention period of ${BACKUP_RETENTION_DAYS} days."
echo "Doing nothing."
else
total=$(probe_all "$target" "$is_local" | wc -l)
if [ "$rule_applies_to" == "$total" ]; then
echo "Using a retention of ${BACKUP_RETENTION_DAYS} days would prune all currently existing backups, will not continue."
echo "If this is what you want, please remove files manually instead of using this script."
else
delete "$target" "$is_local"
echo "Successfully pruned ${rule_applies_to} backups older than ${BACKUP_RETENTION_DAYS} days."
fi
fi
}
if [ ! -z "$BACKUP_RETENTION_DAYS" ]; then
info "Pruning old backups"
echo "Sleeping ${BACKUP_PRUNING_LEEWAY} before checking eligibility."
sleep "$BACKUP_PRUNING_LEEWAY"
if [ ! -z "$AWS_S3_BUCKET_NAME" ]; then
info "Pruning old backups from remote storage"
prune "backup-target/$AWS_S3_BUCKET_NAME"
fi
if [ -d "$BACKUP_ARCHIVE" ]; then
info "Pruning old backups from local archive"
prune "$BACKUP_ARCHIVE" "local"
fi
fi

View File

@@ -1,46 +0,0 @@
#!/bin/sh
# Copyright 2021 - Offen Authors <hioffen@posteo.de>
# SPDX-License-Identifier: MPL-2.0
# Portions of this file are taken from github.com/futurice/docker-volume-backup
# See NOTICE for information about authors and licensing.
set -e
# Write cronjob env to file, fill in sensible defaults, and read them back in
cat <<EOF > env.sh
BACKUP_SOURCES="${BACKUP_SOURCES:-/backup}"
BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"
BACKUP_FILENAME="${BACKUP_FILENAME:-backup-%Y-%m-%dT%H-%M-%S.tar.gz}"
BACKUP_ARCHIVE="${BACKUP_ARCHIVE:-/archive}"
BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-}"
BACKUP_PRUNING_LEEWAY="${BACKUP_PRUNING_LEEWAY:-10m}"
BACKUP_PRUNING_PREFIX="${BACKUP_PRUNING_PREFIX:-}"
BACKUP_STOP_CONTAINER_LABEL="${BACKUP_STOP_CONTAINER_LABEL:-true}"
AWS_S3_BUCKET_NAME="${AWS_S3_BUCKET_NAME:-}"
AWS_ENDPOINT="${AWS_ENDPOINT:-s3.amazonaws.com}"
AWS_ENDPOINT_PROTO="${AWS_ENDPOINT_PROTO:-https}"
GPG_PASSPHRASE="${GPG_PASSPHRASE:-}"
MC_GLOBAL_OPTIONS="${MC_GLOBAL_OPTIONS:-}"
EOF
chmod a+x env.sh
source env.sh
if [ ! -z "$AWS_ACCESS_KEY_ID" ] && [ ! -z "$AWS_SECRET_ACCESS_KEY" ]; then
mc $MC_GLOBAL_OPTIONS alias set backup-target \
"$AWS_ENDPOINT_PROTO://$AWS_ENDPOINT" \
"$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY"
fi
# Add our cron entry, and direct stdout & stderr to Docker commands stdout
echo "Installing cron.d entry with expression $BACKUP_CRON_EXPRESSION."
echo "$BACKUP_CRON_EXPRESSION backup 2>&1" | crontab -
# Let cron take the wheel
echo "Starting cron in foreground."
crond -f -l 8

View File

@@ -7,6 +7,7 @@ cd $(dirname $0)
docker network create test_network docker network create test_network
docker volume create backup_data docker volume create backup_data
docker volume create app_data docker volume create app_data
docker volume create empty_data
docker run -d \ docker run -d \
--name minio \ --name minio \
@@ -23,16 +24,15 @@ docker exec minio mkdir -p /data/backup
docker run -d \ docker run -d \
--name offen \ --name offen \
--network test_network \ --network test_network \
--label "docker-volume-backup.stop-during-backup=true" \
-v app_data:/var/opt/offen/ \ -v app_data:/var/opt/offen/ \
offen/offen:latest offen/offen:latest
sleep 10 sleep 10
docker run -d \ docker run --rm \
--name backup \
--network test_network \ --network test_network \
-v app_data:/backup/app_data \ -v app_data:/backup/app_data \
-v empty_data:/backup/empty_data \
-v /var/run/docker.sock:/var/run/docker.sock \ -v /var/run/docker.sock:/var/run/docker.sock \
--env AWS_ACCESS_KEY_ID=test \ --env AWS_ACCESS_KEY_ID=test \
--env AWS_SECRET_ACCESS_KEY=GMusLtUmILge2by+z890kQ \ --env AWS_SECRET_ACCESS_KEY=GMusLtUmILge2by+z890kQ \
@@ -40,18 +40,17 @@ docker run -d \
--env AWS_ENDPOINT_PROTO=http \ --env AWS_ENDPOINT_PROTO=http \
--env AWS_S3_BUCKET_NAME=backup \ --env AWS_S3_BUCKET_NAME=backup \
--env BACKUP_FILENAME=test.tar.gz \ --env BACKUP_FILENAME=test.tar.gz \
--env BACKUP_CRON_EXPRESSION="0 0 5 31 2 ?" \ --env "BACKUP_FROM_SNAPSHOT=true" \
--entrypoint backup \
offen/docker-volume-backup:$TEST_VERSION offen/docker-volume-backup:$TEST_VERSION
docker exec backup backup
docker run --rm -it \ docker run --rm -it \
-v backup_data:/data alpine \ -v backup_data:/data alpine \
ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db' ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db && test -d /backup/empty_data'
echo "[TEST:PASS] Found relevant files in untared backup." echo "[TEST:PASS] Found relevant files in untared backup."
if [ "$(docker ps -q | wc -l)" != "3" ]; then if [ "$(docker ps -q | wc -l)" != "2" ]; then
echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:" echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
docker ps docker ps
exit 1 exit 1
@@ -59,6 +58,6 @@ fi
echo "[TEST:PASS] All containers running post backup." echo "[TEST:PASS] All containers running post backup."
docker rm $(docker stop minio offen backup) docker rm $(docker stop minio offen)
docker volume rm backup_data app_data docker volume rm backup_data app_data
docker network rm test_network docker network rm test_network

View File

@@ -10,12 +10,23 @@ services:
MINIO_SECRET_KEY: GMusLtUmILge2by+z890kQ MINIO_SECRET_KEY: GMusLtUmILge2by+z890kQ
entrypoint: /bin/ash -c 'mkdir -p /data/backup && minio server /data' entrypoint: /bin/ash -c 'mkdir -p /data/backup && minio server /data'
volumes: volumes:
- backup_data:/data - minio_backup_data:/data
webdav:
image: bytemark/webdav:2.4
environment:
AUTH_TYPE: Digest
USERNAME: test
PASSWORD: test
volumes:
- webdav_backup_data:/var/lib/dav
backup: &default_backup_service backup: &default_backup_service
image: offen/docker-volume-backup:${TEST_VERSION} image: offen/docker-volume-backup:${TEST_VERSION}
hostname: hostnametoken
depends_on: depends_on:
- minio - minio
- webdav
restart: always restart: always
environment: environment:
AWS_ACCESS_KEY_ID: test AWS_ACCESS_KEY_ID: test
@@ -23,12 +34,18 @@ services:
AWS_ENDPOINT: minio:9000 AWS_ENDPOINT: minio:9000
AWS_ENDPOINT_PROTO: http AWS_ENDPOINT_PROTO: http
AWS_S3_BUCKET_NAME: backup AWS_S3_BUCKET_NAME: backup
BACKUP_FILENAME: test.tar.gz BACKUP_FILENAME_EXPAND: 'true'
BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
BACKUP_LATEST_SYMLINK: test-$$HOSTNAME.latest.tar.gz.gpg
BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ? BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7} BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
BACKUP_FORCE_PRUNE: ${BACKUP_FORCE_PRUNE:-}
BACKUP_PRUNING_LEEWAY: 5s BACKUP_PRUNING_LEEWAY: 5s
BACKUP_PRUNING_PREFIX: test BACKUP_PRUNING_PREFIX: test
GPG_PASSPHRASE: 1234secret
WEBDAV_URL: http://webdav/
WEBDAV_PATH: /my/new/path/
WEBDAV_USERNAME: test
WEBDAV_PASSWORD: test
volumes: volumes:
- ./local:/archive - ./local:/archive
- app_data:/backup/app_data:ro - app_data:/backup/app_data:ro
@@ -41,7 +58,7 @@ services:
volumes: volumes:
- app_data:/var/opt/offen - app_data:/var/opt/offen
volumes: volumes:
backup_data: minio_backup_data:
webdav_backup_data:
app_data: app_data:

View File

@@ -9,19 +9,27 @@ mkdir -p local
docker-compose up -d docker-compose up -d
sleep 5 sleep 5
docker-compose exec offen ln -s /var/opt/offen/offen.db /var/opt/offen/db.link
docker-compose exec backup backup docker-compose exec backup backup
docker run --rm -it \ docker run --rm -it \
-v compose_backup_data:/data alpine \ -v compose_minio_backup_data:/minio_data \
ash -c 'tar -xf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db' -v compose_webdav_backup_data:/webdav_data alpine \
ash -c 'apk add gnupg && \
echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /minio_data/backup/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db && \
echo 1234secret | gpg -d --pinentry-mode loopback --passphrase-fd 0 --yes /webdav_data/data/my/new/path/test-hostnametoken.tar.gz.gpg > /tmp/test-hostnametoken.tar.gz && tar -xf /tmp/test-hostnametoken.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db'
echo "[TEST:PASS] Found relevant files in untared remote backup." echo "[TEST:PASS] Found relevant files in untared remote backups."
tar -xf ./local/test.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db test -L ./local/test-hostnametoken.latest.tar.gz.gpg
echo 1234secret | gpg -d --yes --passphrase-fd 0 ./local/test-hostnametoken.tar.gz.gpg > ./local/decrypted.tar.gz
tar -xf ./local/decrypted.tar.gz -C /tmp && test -f /tmp/backup/app_data/offen.db
rm ./local/decrypted.tar.gz
test -L /tmp/backup/app_data/db.link
echo "[TEST:PASS] Found relevant files in untared local backup." echo "[TEST:PASS] Found relevant files in untared local backup."
if [ "$(docker-compose ps -q | wc -l)" != "3" ]; then if [ "$(docker-compose ps -q | wc -l)" != "4" ]; then
echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:" echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
docker-compose ps docker-compose ps
exit 1 exit 1
@@ -29,8 +37,6 @@ fi
echo "[TEST:PASS] All containers running post backup." echo "[TEST:PASS] All containers running post backup."
docker-compose down
# The second part of this test checks if backups get deleted when the retention # The second part of this test checks if backups get deleted when the retention
# is set to 0 days (which it should not as it would mean all backups get deleted) # is set to 0 days (which it should not as it would mean all backups get deleted)
# TODO: find out if we can test actual deletion without having to wait for a day # TODO: find out if we can test actual deletion without having to wait for a day
@@ -40,8 +46,10 @@ sleep 5
docker-compose exec backup backup docker-compose exec backup backup
docker run --rm -it \ docker run --rm -it \
-v compose_backup_data:/data alpine \ -v compose_minio_backup_data:/minio_data \
ash -c '[ $(find /data/backup/ -type f | wc -l) = "1" ]' -v compose_webdav_backup_data:/webdav_data alpine \
ash -c '[ $(find /minio_data/backup/ -type f | wc -l) = "1" ] && \
[ $(find /webdav_data/data/my/new/path/ -type f | wc -l) = "1" ]'
echo "[TEST:PASS] Remote backups have not been deleted." echo "[TEST:PASS] Remote backups have not been deleted."