Merge pull request #26 from offen/instance-profile

Allow s3 authentication via IAM role

README.md

@@ -3,15 +3,18 @@
 Backup Docker volumes locally or to any S3 compatible storage.
 
 The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
-It handles __recurring or one-off backups of Docker volumes__ to a __local directory__ or __any S3 compatible storage__ (or both), and __rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__.
+It handles __recurring or one-off backups of Docker volumes__ to a __local directory__ or __any S3 compatible storage__ (or both), and __rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
 
 <!-- MarkdownTOC -->
 
 - [Quickstart](#quickstart)
+  - [Recurring backups in a compose setup](#recurring-backups-in-a-compose-setup)
+  - [One-off backups using Docker CLI](#one-off-backups-using-docker-cli)
 - [Configuration reference](#configuration-reference)
 - [How to](#how-to)
   - [Stopping containers during backup](#stopping-containers-during-backup)
   - [Automatically pruning old backups](#automatically-pruning-old-backups)
+  - [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs)
   - [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg)
   - [Restoring a volume from a backup](#restoring-a-volume-from-a-backup)
   - [Using with Docker Swarm](#using-with-docker-swarm)
@@ -37,6 +40,8 @@ Code and documentation for `v1` versions are found on [this branch][v1-branch].
 
 ## Quickstart
 
+### Recurring backups in a compose setup
+
 Add a `backup` service to your compose setup and mount the volumes you would like to see backed up:
 
 ```yml
@@ -55,6 +60,10 @@ services:
       - docker-volume-backup.stop-during-backup=true
 
   backup:
+    # In production, it is advised to lock your image tag to a proper
+    # release version instead of using `latest`.
+    # Check https://github.com/offen/docker-volume-backup/releases
+    # for a list of available releases.
     image: offen/docker-volume-backup:latest
     restart: always
     env_file: ./backup.env # see below for configuration reference
@@ -73,6 +82,22 @@ volumes:
   data:
 ```
 
+### One-off backups using Docker CLI
+
+To run a one time backup, mount the volume you would like to see backed up into a container and run the `backup` command:
+
+```console
+docker run --rm \
+  -v data:/backup/data \
+  --env AWS_ACCESS_KEY_ID="<xxx>" \
+  --env AWS_SECRET_ACCESS_KEY="<xxx>" \
+  --env AWS_S3_BUCKET_NAME="<xxx>" \
+  --entrypoint backup \
+  offen/docker-volume-backup:latest
+```
+
+Alternatively, pass a `--env-file` in order to use a full config as described below.
+
 ## Configuration reference
 
 Backup targets, schedule and retention are configured in environment variables.
@@ -109,6 +134,13 @@ You can populate below template according to your requirements and use it as you
 # AWS_ACCESS_KEY_ID="<xxx>"
 # AWS_SECRET_ACCESS_KEY="<xxx>"
 
+# Instead of providing static credentials, you can also use IAM instance profiles
+# or similar to provide authentication. Some possible configuration options on AWS:
+# - EC2: http://169.254.169.254
+# - ECS: http://169.254.170.2
+
+# AWS_IAM_ROLE_ENDPOINT="http://169.254.169.254"
+
 # This is the FQDN of your storage server, e.g. `storage.example.com`.
 # Do not set this when working against AWS S3 (the default value is 
 # `s3.amazonaws.com`). If you need to set a specific (non-https) protocol, you
@@ -188,6 +220,30 @@ You can populate below template according to your requirements and use it as you
 # override this default by specifying a different value here.
 
 # BACKUP_STOP_CONTAINER_LABEL="service1"
+
+########### EMAIL NOTIFICATIONS ON FAILED BACKUP RUNS
+
+# In case SMTP credentials are provided, notification emails can be sent out on
+# failed backup runs. These emails will contain the start time, the error
+# message and all log output prior to the failure.
+
+# The recipient(s) of the notification. Supply a comma separated list
+# of adresses if you want to notify multiple recipients. If this is
+# not set, no emails will be sent.
+
+# EMAIL_NOTIFICATION_RECIPIENT="you@example.com"
+
+# The "From" header of the sent email. Defaults to `noreply@nohost`.
+
+# EMAIL_NOTIFICATION_SENDER="no-reply@example.com"
+
+# Configuration and credentials for the SMTP server to be used.
+# EMAIL_SMTP_PORT defaults to 587.
+
+# EMAIL_SMTP_HOST="posteo.de"
+# EMAIL_SMTP_PASSWORD="<xxx>"
+# EMAIL_SMTP_USERNAME="no-reply@example.com"
+# EMAIL_SMTP_PORT="<port>"
 ```
 
 ## How to
@@ -247,6 +303,25 @@ volumes:
   data:
 ```
 
+### Send email notifications on failed backup runs
+
+To send out email notifications on failed backup runs, provide SMTP credentials, a sender and a recipient:
+
+```yml
+version: '3'
+
+services:
+  backup:
+    image: offen/docker-volume-backup:latest
+    environment:
+      # ... other configuration values go here
+      EMAIL_SMTP_HOST: "smtp.example.com"
+      EMAIL_SMTP_PASSWORD: "password"
+      EMAIL_SMTP_USERNAME: "username"
+      EMAIL_NOTIFICATION_SENDER: "noreply@example.com"
+      EMAIL_NOTIFICATION_RECIPIENT: "notifications@example.com"
+```
+
 ### Encrypting your backup using GPG
 
 The image supports encrypting backups using GPG out of the box.

cmd/backup/main.go

@@ -4,6 +4,7 @@
 package main
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"fmt"
@@ -18,6 +19,7 @@ import (
 	"github.com/docker/docker/api/types/filters"
 	"github.com/docker/docker/api/types/swarm"
 	"github.com/docker/docker/client"
+	"github.com/go-gomail/gomail"
 	"github.com/gofrs/flock"
 	"github.com/kelseyhightower/envconfig"
 	"github.com/leekchan/timeutil"
@@ -37,6 +39,15 @@ func main() {
 		panic(err)
 	}
 
+	defer func() {
+		if err := recover(); err != nil {
+			if e, ok := err.(error); ok && strings.Contains(e.Error(), msgBackupFailed) {
+				os.Exit(1)
+			}
+			panic(err)
+		}
+	}()
+
 	s.must(func() error {
 		restartContainers, err := s.stopContainers()
 		defer func() {
@@ -48,9 +59,14 @@ func main() {
 		return s.takeBackup()
 	}())
 
-	s.must(s.encryptBackup())
-	s.must(s.copyBackup())
+	s.must(func() error {
+		defer func() {
 			s.must(s.removeArtifacts())
+		}()
+		s.must(s.encryptBackup())
+		return s.copyBackup()
+	}())
+
 	s.must(s.pruneOldBackups())
 	s.logger.Info("Finished running backup tasks.")
 }
@@ -61,9 +77,11 @@ type script struct {
 	cli    *client.Client
 	mc     *minio.Client
 	logger *logrus.Logger
+	hooks  []hook
 
 	start  time.Time
 	file   string
+	output *bytes.Buffer
 
 	c *config
 }
@@ -82,23 +100,34 @@ type config struct {
 	AwsEndpointInsecure        bool          `split_words:"true"`
 	AwsAccessKeyID             string        `envconfig:"AWS_ACCESS_KEY_ID"`
 	AwsSecretAccessKey         string        `split_words:"true"`
+	AwsIamRoleEndpoint         string        `split_words:"true"`
 	GpgPassphrase              string        `split_words:"true"`
+	EmailNotificationRecipient string        `split_words:"true"`
+	EmailNotificationSender    string        `split_words:"true" default:"noreply@nohost"`
+	EmailSMTPHost              string        `envconfig:"EMAIL_SMTP_HOST"`
+	EmailSMTPPort              int           `envconfig:"EMAIL_SMTP_PORT" default:"587"`
+	EmailSMTPUsername          string        `envconfig:"EMAIL_SMTP_USERNAME"`
+	EmailSMTPPassword          string        `envconfig:"EMAIL_SMTP_PASSWORD"`
 }
 
+var msgBackupFailed = "backup run failed"
+
 // newScript creates all resources needed for the script to perform actions against
 // remote resources like the Docker engine or remote storage locations. All
 // reading from env vars or other configuration sources is expected to happen
 // in this method.
 func newScript() (*script, error) {
+	stdOut, logBuffer := buffer(os.Stdout)
 	s := &script{
 		c: &config{},
 		logger: &logrus.Logger{
-			Out:       os.Stdout,
+			Out:       stdOut,
 			Formatter: new(logrus.TextFormatter),
 			Hooks:     make(logrus.LevelHooks),
 			Level:     logrus.InfoLevel,
 		},
 		start:  time.Now(),
+		output: logBuffer,
 	}
 
 	if err := envconfig.Process("", s.c); err != nil {
@@ -117,12 +146,21 @@ func newScript() (*script, error) {
 	}
 
 	if s.c.AwsS3BucketName != "" {
-		mc, err := minio.New(s.c.AwsEndpoint, &minio.Options{
-			Creds: credentials.NewStaticV4(
+		var creds *credentials.Credentials
+		if s.c.AwsAccessKeyID != "" && s.c.AwsSecretAccessKey != "" {
+			creds = credentials.NewStaticV4(
 				s.c.AwsAccessKeyID,
 				s.c.AwsSecretAccessKey,
 				"",
-			),
+			)
+		} else if s.c.AwsIamRoleEndpoint != "" {
+			creds = credentials.NewIAM(s.c.AwsIamRoleEndpoint)
+		} else {
+			return nil, errors.New("newScript: AWS_S3_BUCKET_NAME is defined, but no credentials were provided")
+		}
+
+		mc, err := minio.New(s.c.AwsEndpoint, &minio.Options{
+			Creds:  creds,
 			Secure: !s.c.AwsEndpointInsecure && s.c.AwsEndpointProto == "https",
 		})
 		if err != nil {
@@ -131,6 +169,28 @@ func newScript() (*script, error) {
 		s.mc = mc
 	}
 
+	if s.c.EmailNotificationRecipient != "" {
+		s.hooks = append(s.hooks, hook{hookLevelFailure, func(err error, start time.Time, logOutput string) error {
+			mailer := gomail.NewDialer(
+				s.c.EmailSMTPHost, s.c.EmailSMTPPort, s.c.EmailSMTPUsername, s.c.EmailSMTPPassword,
+			)
+
+			subject := fmt.Sprintf(
+				"Failure running docker-volume-backup at %s", start.Format(time.RFC3339),
+			)
+			body := fmt.Sprintf(
+				"Running docker-volume-backup failed with error: %s\n\nLog output of the failed run was:\n\n%s\n", err, logOutput,
+			)
+
+			message := gomail.NewMessage()
+			message.SetHeader("From", s.c.EmailNotificationSender)
+			message.SetHeader("To", s.c.EmailNotificationRecipient)
+			message.SetHeader("Subject", subject)
+			message.SetBody("text/plain", body)
+			return mailer.DialAndSend(message)
+		}})
+	}
+
 	return s, nil
 }
 
@@ -326,10 +386,17 @@ func (s *script) copyBackup() error {
 
 // removeArtifacts removes the backup file from disk.
 func (s *script) removeArtifacts() error {
-	if err := os.Remove(s.file); err != nil {
-		return fmt.Errorf("removeArtifacts: error removing file: %w", err)
+	_, err := os.Stat(s.file)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
 		}
-	s.logger.Info("Removed local artifacts.")
+		return fmt.Errorf("removeArtifacts: error calling stat on file %s: %w", s.file, err)
+	}
+	if err := os.Remove(s.file); err != nil {
+		return fmt.Errorf("removeArtifacts: error removing file %s: %w", s.file, err)
+	}
+	s.logger.Infof("Removed local artifacts %s.", s.file)
 	return nil
 }
 
@@ -468,11 +535,35 @@ func (s *script) pruneOldBackups() error {
 	return nil
 }
 
-// must exits the script run non-zero and prematurely in case the given error
-// is non-nil.
+// runHooks runs all hooks that have been registered using the
+// given level. In case executing a hook returns an error, the following
+// hooks will still be run before the function returns an error.
+func (s *script) runHooks(err error, targetLevel string) error {
+	var actionErrors []error
+	for _, hook := range s.hooks {
+		if hook.level != targetLevel {
+			continue
+		}
+		if err := hook.action(err, s.start, s.output.String()); err != nil {
+			actionErrors = append(actionErrors, err)
+		}
+	}
+	if len(actionErrors) != 0 {
+		return join(actionErrors...)
+	}
+	return nil
+}
+
+// must exits the script run prematurely in case the given error
+// is non-nil. If failure hooks have been registered on the script object, they
+// will be called, passing the failure and previous log output.
 func (s *script) must(err error) {
 	if err != nil {
-		s.logger.Fatalf("Fatal error running backup: %s", err)
+		s.logger.Errorf("Fatal error running backup: %s", err)
+		if hookErr := s.runHooks(err, hookLevelFailure); hookErr != nil {
+			s.logger.Errorf("An error occurred calling the registered failure hooks: %s", hookErr)
+		}
+		panic(errors.New(msgBackupFailed))
 	}
 }
 
@@ -526,3 +617,33 @@ func join(errs ...error) error {
 	}
 	return errors.New("[" + strings.Join(msgs, ", ") + "]")
 }
+
+// buffer takes an io.Writer and returns a wrapped version of the
+// writer that writes to both the original target as well as the returned buffer
+func buffer(w io.Writer) (io.Writer, *bytes.Buffer) {
+	buffering := &bufferingWriter{buf: bytes.Buffer{}, writer: w}
+	return buffering, &buffering.buf
+}
+
+type bufferingWriter struct {
+	buf    bytes.Buffer
+	writer io.Writer
+}
+
+func (b *bufferingWriter) Write(p []byte) (n int, err error) {
+	if n, err := b.buf.Write(p); err != nil {
+		return n, fmt.Errorf("bufferingWriter: error writing to buffer: %w", err)
+	}
+	return b.writer.Write(p)
+}
+
+// hook contains a queued action that can be trigger them when the script
+// reaches a certain point (e.g. unsuccessful backup)
+type hook struct {
+	level  string
+	action func(err error, start time.Time, logOutput string) error
+}
+
+const (
+	hookLevelFailure = "failure"
+)

go.mod

@@ -20,6 +20,7 @@ require (
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/docker/go-units v0.4.0 // indirect
 	github.com/dustin/go-humanize v1.0.0 // indirect
+	github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/protobuf v1.5.0 // indirect
 	github.com/google/uuid v1.2.0 // indirect
@@ -41,5 +42,6 @@ require (
 	google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a // indirect
 	google.golang.org/grpc v1.33.2 // indirect
 	google.golang.org/protobuf v1.26.0 // indirect
+	gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
 	gopkg.in/ini.v1 v1.57.0 // indirect
 )

go.sum

@@ -254,6 +254,8 @@ github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeME
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
+github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df h1:Bao6dhmbTA1KFVxmJ6nBoMuOJit2yjEgLJpIMYpop0E=
+github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df/go.mod h1:GJr+FCSXshIwgHBtLglIg9M2l2kQSi6QjVAngtzI08Y=
 github.com/go-ini/ini v1.25.4/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@@ -908,6 +910,8 @@ google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/l
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc h1:2gGKlE2+asNV9m7xrywl36YYNnBG5ZQ0r/BOOxqPpmk=
+gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc/go.mod h1:m7x9LTH6d71AHyAX77c9yqWCCa3UKHcVEj9y7hAtKDk=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20141024133853-64131543e789/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

test/cli/run.sh

@@ -29,8 +29,7 @@ docker run -d \
 
 sleep 10
 
-docker run -d \
-  --name backup \
+docker run --rm \
   --network test_network \
   -v app_data:/backup/app_data \
   -v /var/run/docker.sock:/var/run/docker.sock \
@@ -40,18 +39,16 @@ docker run -d \
   --env AWS_ENDPOINT_PROTO=http \
   --env AWS_S3_BUCKET_NAME=backup \
   --env BACKUP_FILENAME=test.tar.gz \
-  --env BACKUP_CRON_EXPRESSION="0 0 5 31 2 ?" \
+  --entrypoint backup \
   offen/docker-volume-backup:$TEST_VERSION
 
-docker exec backup backup
-
 docker run --rm -it \
   -v backup_data:/data alpine \
   ash -c 'tar -xvf /data/backup/test.tar.gz && test -f /backup/app_data/offen.db'
 
 echo "[TEST:PASS] Found relevant files in untared backup."
 
-if [ "$(docker ps -q | wc -l)" != "3" ]; then
+if [ "$(docker ps -q | wc -l)" != "2" ]; then
   echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
   docker ps
   exit 1