Release version 0.24.0

Add option for Releases

CHANGES.rst

@@ -1,49 +1,98 @@
 Changelog
 =========
 
-0.21.0 (2018-11-28)
+0.24.0 (2019-06-27)
+-------------------
+------------------------
+- QKT-45: include assets - update readme. [Ethan Timm]
+
+  update readme with flag information for including assets alongside their respective releases
+- Make assets it's own flag. [Harrison Wright]
+- Fix super call for python2. [Harrison Wright]
+- Fix redirect to s3. [Harrison Wright]
+- WIP: download assets. [Harrison Wright]
+- QKT-42: releases - add readme info. [ethan]
+- QKT-42 update: shorter command flag. [ethan]
+- QKT-42: support saving release information. [ethan]
+- Fix pull details. [Harrison Wright]
+
+
+0.23.0 (2019-06-04)
+-------------------
+- Avoid to crash in case of HTTP 502 error. [Gael de Chalendar]
+
+  Survive also on socket.error connections like on HTTPError or URLError.
+
+  This should solve issue #110.
+
+
+0.22.2 (2019-02-21)
 -------------------
 
+Fix
+~~~
+- Warn instead of error. [Jose Diaz-Gonzalez]
+
+  Refs #106
+
+
+0.22.1 (2019-02-21)
+-------------------
+- Log URL error https://github.com/josegonzalez/python-github-
+  backup/issues/105. [JOHN STETIC]
+
+
+0.22.0 (2019-02-01)
+-------------------
+- Remove unnecessary sys.exit call. [W. Harrison Wright]
+- Add org check to avoid incorrect log output. [W. Harrison Wright]
+- Fix accidental system exit with better logging strategy. [W. Harrison
+  Wright]
+
+
+0.21.1 (2018-12-25)
+-------------------
+- Mark options which are not included in --all. [Bernd]
+
+  As discussed in Issue #100
+
+
+0.21.0 (2018-11-28)
+-------------------
 - Correctly download repos when user arg != authenticated user. [W.
   Harrison Wright]
 
+
 0.20.1 (2018-09-29)
 -------------------
-
 - Clone the specified user's gists, not the authenticated user. [W.
   Harrison Wright]
-
 - Clone the specified user's starred repos, not the authenticated user.
   [W. Harrison Wright]
 
+
 0.20.0 (2018-03-24)
 -------------------
-
 - Chore: drop Python 2.6. [Jose Diaz-Gonzalez]
-
 - Feat: simplify release script. [Jose Diaz-Gonzalez]
 
+
 0.19.2 (2018-03-24)
 -------------------
 
 Fix
 ~~~
-
 - Cleanup pep8 violations. [Jose Diaz-Gonzalez]
 
+
 0.19.0 (2018-03-24)
 -------------------
-
 - Add additional output for the current request. [Robin Gloster]
 
   This is useful to have some progress indication for huge repositories.
-
-
 - Add option to backup additional PR details. [Robin Gloster]
 
   Some payload is only included when requesting a single pull request
-
-
 - Mark string as binary in comparison for skip_existing. [Johannes
   Bornhold]
 
@@ -54,66 +103,53 @@ Fix
 
 0.18.0 (2018-02-22)
 -------------------
-
 - Add option to fetch followers/following JSON data. [Stephen Greene]
 
+
 0.17.0 (2018-02-20)
 -------------------
-
 - Short circuit gists backup process. [W. Harrison Wright]
-
 - Formatting. [W. Harrison Wright]
-
 - Add ability to backup gists. [W. Harrison Wright]
 
+
 0.16.0 (2018-01-22)
 -------------------
-
 - Change option to --all-starred. [W. Harrison Wright]
-
 - JK don't update documentation. [W. Harrison Wright]
-
 - Put starred clone repoistories under a new option. [W. Harrison
   Wright]
-
 - Add comment. [W. Harrison Wright]
-
 - Add ability to clone starred repos. [W. Harrison Wright]
 
+
 0.14.1 (2017-10-11)
 -------------------
-
 - Fix arg not defined error. [Edward Pfremmer]
 
-  Ref: https://github.com/josegonzalez/python-github-backup/issues/69
 
 0.14.0 (2017-10-11)
 -------------------
-
 - Added a check to see if git-lfs is installed when doing an LFS clone.
   [pieterclaerhout]
-
 - Added support for LFS clones. [pieterclaerhout]
-
 - Add pypi info to readme. [Albert Wang]
-
 - Explicitly support python 3 in package description. [Albert Wang]
-
 - Add couple examples to help new users. [Yusuf Tran]
 
+
 0.13.2 (2017-05-06)
 -------------------
-
 - Fix remotes while updating repository. [Dima Gerasimov]
 
+
 0.13.1 (2017-04-11)
 -------------------
-
 - Fix error when repository has no updated_at value. [Nicolai Ehemann]
 
+
 0.13.0 (2017-04-05)
 -------------------
-
 - Add OS check for OSX specific keychain args. [Martin O'Reilly]
 
   Keychain arguments are only supported on Mac OSX.
@@ -122,8 +158,6 @@ Fix
   error message rather than a "No password item matching the
   provided name and account could be found in the osx keychain"
   error message
-
-
 - Add support for storing PAT in OSX keychain. [Martin O'Reilly]
 
   Added additional optional arguments and README guidance for storing
@@ -133,62 +167,48 @@ Fix
 
 0.12.1 (2017-03-27)
 -------------------
-
 - Avoid remote branch name churn. [Chris Adams]
 
   This avoids the backup output having lots of "[new branch]" messages
   because removing the old remote name removed all of the existing branch
   references.
-
-
 - Fix detection of bare git directories. [Andrzej Maczuga]
 
+
 0.12.0 (2016-11-22)
 -------------------
 
 Fix
 ~~~
-
 - Properly import version from github_backup package. [Jose Diaz-
   Gonzalez]
-
 - Support alternate git status output. [Jose Diaz-Gonzalez]
 
 Other
 ~~~~~
-
 - Pep8: E501 line too long (83 > 79 characters) [Jose Diaz-Gonzalez]
-
 - Pep8: E128 continuation line under-indented for visual indent. [Jose
   Diaz-Gonzalez]
-
 - Support archivization using bare git clones. [Andrzej Maczuga]
-
 - Fix typo, 3x. [Terrell Russell]
 
+
 0.11.0 (2016-10-26)
 -------------------
-
 - Support --token file:///home/user/token.txt (fixes gh-51) [Björn
   Dahlgren]
-
 - Fix some linting. [Albert Wang]
-
 - Fix byte/string conversion for python 3. [Albert Wang]
-
 - Support python 3. [Albert Wang]
-
 - Encode special characters in password. [Remi Rampin]
-
 - Don't pretend program name is "Github Backup" [Remi Rampin]
-
 - Don't install over insecure connection. [Remi Rampin]
 
   The git:// protocol is unauthenticated and unencrypted, and no longer advertised by GitHub. Using HTTPS shouldn't impact performance.
 
+
 0.10.3 (2016-08-21)
 -------------------
-
 - Fixes #29. [Jonas Michel]
 
   Reporting an error when the user's rate limit is exceeded causes
@@ -196,8 +216,6 @@ Other
   sleep. Instead of generating an explicit error we just want to
   inform the user that the script is going to sleep until their rate
   limit count resets.
-
-
 - Fixes #29. [Jonas Michel]
 
   The errors list was not being cleared out after resuming a backup
@@ -208,14 +226,13 @@ Other
 
 0.10.2 (2016-08-21)
 -------------------
-
 - Add a note regarding git version requirement. [Jose Diaz-Gonzalez]
 
   Closes #37
 
+
 0.10.0 (2016-08-18)
 -------------------
-
 - Implement incremental updates. [Robert Bradshaw]
 
   Guarded with an --incremental flag.
@@ -228,12 +245,11 @@ Other
 
 0.9.0 (2016-03-29)
 ------------------
-
 - Fix cloning private repos with basic auth or token. [Kazuki Suda]
 
+
 0.8.0 (2016-02-14)
 ------------------
-
 - Don't store issues which are actually pull requests. [Enrico Tröger]
 
   This prevents storing pull requests twice since the Github API returns
@@ -244,43 +260,31 @@ Other
 
 0.7.0 (2016-02-02)
 ------------------
-
 - Softly fail if not able to read hooks. [Albert Wang]
-
 - Add note about 2-factor auth. [Albert Wang]
-
 - Make user repository search go through endpoint capable of reading
   private repositories. [Albert Wang]
-
 - Prompt for password if only username given. [Alex Hall]
 
+
 0.6.0 (2015-11-10)
 ------------------
-
 - Force proper remote url. [Jose Diaz-Gonzalez]
-
 - Improve error handling in case of HTTP errors. [Enrico Tröger]
 
   In case of a HTTP status code 404, the returned 'r' was never assigned.
   In case of URL errors which are not timeouts, we probably should bail
   out.
-
-
 - Add --hooks to also include web hooks into the backup. [Enrico Tröger]
-
 - Create the user specified output directory if it does not exist.
   [Enrico Tröger]
 
   Fixes #17.
-
-
 - Add missing auth argument to _get_response() [Enrico Tröger]
 
   When running unauthenticated and Github starts rate-limiting the client,
   github-backup crashes because the used auth variable in _get_response()
   was not available. This change should fix it.
-
-
 - Add repository URL to error message for non-existing repositories.
   [Enrico Tröger]
 
@@ -291,40 +295,28 @@ Other
 
 0.5.0 (2015-10-10)
 ------------------
-
 - Add release script. [Jose Diaz-Gonzalez]
-
 - Refactor to both simplify codepath as well as follow PEP8 standards.
   [Jose Diaz-Gonzalez]
-
 - Retry 3 times when the connection times out. [Mathijs Jonker]
-
 - Made unicode output defalut. [Kirill Grushetsky]
-
 - Import alphabetised. [Kirill Grushetsky]
-
 - Preserve Unicode characters in the output file. [Kirill Grushetsky]
 
   Added option to preserve Unicode characters in the output file
-
 - Josegonzales/python-github-backup#12 Added backup of labels and
   milestones. [aensley]
-
 - Fixed indent. [Mathijs Jonker]
-
 - Skip unitialized repo's. [mjonker-embed]
 
   These gave me errors which caused mails from crontab.
-
 - Added prefer-ssh. [mjonker-embed]
 
   Was needed for my back-up setup, code includes this but readme wasn't updated
-
 - Retry API requests which failed due to rate-limiting. [Chris Adams]
 
   This allows operation to continue, albeit at a slower pace,
   if you have enough data to trigger the API rate limits
-
 - Logging_subprocess: always log when a command fails. [Chris Adams]
 
   Previously git clones could fail without any indication
@@ -334,21 +326,15 @@ Other
   Now a non-zero return code will always output a message to
   stderr and will display the executed command so it can be
   rerun for troubleshooting.
-
-
 - Switch to using ssh_url. [Chris Adams]
 
   The previous commit used the wrong URL for a private repo. This was
   masked by the lack of error loging in logging_subprocess (which will be
   in a separate branch)
-
-
 - Add an option to prefer checkouts over SSH. [Chris Adams]
 
   This is really useful with private repos to avoid being nagged
   for credentials for every repository
-
-
 - Add pull request support. [Kevin Laude]
 
   Back up reporitory pull requests by passing the --include-pulls
@@ -360,8 +346,6 @@ Other
 
   Pull requests are automatically backed up when the --all argument is
   uesd.
-
-
 - Add GitHub Enterprise support. [Kevin Laude]
 
   Pass the -H or --github-host argument with a GitHub Enterprise hostname
@@ -371,35 +355,21 @@ Other
 
 0.2.0 (2014-09-22)
 ------------------
-
 - Add support for retrieving repositories. Closes #1. [Jose Diaz-
   Gonzalez]
-
 - Fix PEP8 violations. [Jose Diaz-Gonzalez]
-
 - Add authorization to header only if specified by user. [Ioannis
   Filippidis]
-
 - Fill out readme more. [Jose Diaz-Gonzalez]
-
 - Fix import. [Jose Diaz-Gonzalez]
-
 - Properly name readme. [Jose Diaz-Gonzalez]
-
 - Create MANIFEST.in. [Jose Diaz-Gonzalez]
-
 - Create .gitignore. [Jose Diaz-Gonzalez]
-
 - Create setup.py. [Jose Diaz-Gonzalez]
-
 - Create requirements.txt. [Jose Diaz-Gonzalez]
-
 - Create __init__.py. [Jose Diaz-Gonzalez]
-
 - Create LICENSE.txt. [Jose Diaz-Gonzalez]
-
 - Create README.md. [Jose Diaz-Gonzalez]
-
 - Create github-backup. [Jose Diaz-Gonzalez]
 
 

README.rst

@@ -32,8 +32,9 @@ CLI Usage is as follows::
                   [--watched] [--followers] [--following] [--all]
                   [--issues] [--issue-comments] [--issue-events] [--pulls]
                   [--pull-comments] [--pull-commits] [--labels] [--hooks]
-                  [--milestones] [--repositories] [--bare] [--lfs]
-                  [--wikis] [--gists] [--starred-gists] [--skip-existing]
+                  [--milestones] [--repositories] [--releases] [--assets]
+                  [--bare] [--lfs] [--wikis] [--gists] [--starred-gists]
+                  [--skip-existing]
                   [-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
                   [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
                   [--prefer-ssh] [-v]
@@ -76,6 +77,8 @@ CLI Usage is as follows::
                             authenticated)
       --milestones          include milestones in backup
       --repositories        include repository clone in backup
+      --releases            include repository releases' information without assets or binaries
+      --assets              include assets alongside release information; only applies if including releases
       --bare                clone bare repositories
       --lfs                 clone LFS repositories (requires Git LFS to be
                             installed, https://git-lfs.github.com)

bin/github-backup

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 
 from __future__ import print_function
+import socket
 
 import argparse
 import base64
@@ -17,6 +18,7 @@ import subprocess
 import sys
 import time
 import platform
+PY2 = False
 try:
     # python 3
     from urllib.parse import urlparse
@@ -25,14 +27,19 @@ try:
     from urllib.error import HTTPError, URLError
     from urllib.request import urlopen
     from urllib.request import Request
+    from urllib.request import HTTPRedirectHandler
+    from urllib.request import build_opener
 except ImportError:
     # python 2
+    PY2 = True
     from urlparse import urlparse
     from urllib import quote as urlquote
     from urllib import urlencode
     from urllib2 import HTTPError, URLError
     from urllib2 import urlopen
     from urllib2 import Request
+    from urllib2 import HTTPRedirectHandler
+    from urllib2 import build_opener
 
 from github_backup import __version__
 
@@ -40,16 +47,17 @@ FNULL = open(os.devnull, 'w')
 
 
 def log_error(message):
-    if type(message) == str:
-        message = [message]
-
-    for msg in message:
-        sys.stderr.write("{0}\n".format(msg))
-
+    """
+    Log message (str) or messages (List[str]) to stderr and exit with status 1
+    """
+    log_warning(message)
     sys.exit(1)
 
 
 def log_info(message):
+    """
+    Log message (str) or messages (List[str]) to stdout
+    """
     if type(message) == str:
         message = [message]
 
@@ -57,6 +65,17 @@ def log_info(message):
         sys.stdout.write("{0}\n".format(msg))
 
 
+def log_warning(message):
+    """
+    Log message (str) or messages (List[str]) to stderr
+    """
+    if type(message) == str:
+        message = [message]
+
+    for msg in message:
+        sys.stderr.write("{0}\n".format(msg))
+
+
 def logging_subprocess(popenargs,
                        logger,
                        stdout_log_level=logging.DEBUG,
@@ -163,11 +182,11 @@ def parse_args():
     parser.add_argument('--all-starred',
                         action='store_true',
                         dest='all_starred',
-                        help='include starred repositories in backup')
+                        help='include starred repositories in backup [*]')
     parser.add_argument('--watched',
                         action='store_true',
                         dest='include_watched',
-                        help='include watched repositories in backup')
+                        help='include JSON output of watched repositories in backup')
     parser.add_argument('--followers',
                         action='store_true',
                         dest='include_followers',
@@ -179,7 +198,7 @@ def parse_args():
     parser.add_argument('--all',
                         action='store_true',
                         dest='include_everything',
-                        help='include everything in backup')
+                        help='include everything in backup (not including [*])')
     parser.add_argument('--issues',
                         action='store_true',
                         dest='include_issues',
@@ -207,7 +226,7 @@ def parse_args():
     parser.add_argument('--pull-details',
                         action='store_true',
                         dest='include_pull_details',
-                        help='include more pull request details in backup')
+                        help='include more pull request details in backup [*]')
     parser.add_argument('--labels',
                         action='store_true',
                         dest='include_labels',
@@ -231,7 +250,7 @@ def parse_args():
     parser.add_argument('--lfs',
                         action='store_true',
                         dest='lfs_clone',
-                        help='clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com)')
+                        help='clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com) [*]')
     parser.add_argument('--wikis',
                         action='store_true',
                         dest='include_wiki',
@@ -239,11 +258,11 @@ def parse_args():
     parser.add_argument('--gists',
                         action='store_true',
                         dest='include_gists',
-                        help='include gists in backup')
+                        help='include gists in backup [*]')
     parser.add_argument('--starred-gists',
                         action='store_true',
                         dest='include_starred_gists',
-                        help='include starred gists in backup')
+                        help='include starred gists in backup [*]')
     parser.add_argument('--skip-existing',
                         action='store_true',
                         dest='skip_existing',
@@ -273,11 +292,11 @@ def parse_args():
     parser.add_argument('-P', '--private',
                         action='store_true',
                         dest='private',
-                        help='include private repositories')
+                        help='include private repositories [*]')
     parser.add_argument('-F', '--fork',
                         action='store_true',
                         dest='fork',
-                        help='include forked repositories')
+                        help='include forked repositories [*]')
     parser.add_argument('--prefer-ssh',
                         action='store_true',
                         help='Clone repositories using SSH instead of HTTPS')
@@ -290,6 +309,15 @@ def parse_args():
     parser.add_argument('--keychain-account',
                         dest='osx_keychain_item_account',
                         help='OSX ONLY: account field of password item in OSX keychain that holds the personal access or OAuth token')
+    parser.add_argument('--releases',
+                        action='store_true',
+                        dest='include_releases',
+                        help='include release information, not including assets or binaries'
+                        )
+    parser.add_argument('--assets',
+                        action='store_true',
+                        dest='include_assets',
+                        help='include assets alongside release information; only applies if including releases')
     return parser.parse_args()
 
 
@@ -392,6 +420,16 @@ def retrieve_data(args, template, query_args=None, single_request=False):
 
         status_code = int(r.getcode())
 
+        retries = 0
+        while retries < 3 and status_code == 502:
+            print('API request returned HTTP 502: Bad Gateway. Retrying in 5 seconds')
+            retries += 1
+            time.sleep(5)
+            request = _construct_request(per_page, page, query_args, template, auth)  # noqa
+            r, errors = _get_response(request, auth, template)
+
+            status_code = int(r.getcode())
+
         if status_code != 200:
             template = 'API request returned HTTP {0}: {1}'
             errors.append(template.format(status_code, r.reason))
@@ -433,7 +471,13 @@ def _get_response(request, auth, template):
         except HTTPError as exc:
             errors, should_continue = _request_http_error(exc, auth, errors)  # noqa
             r = exc
-        except URLError:
+        except URLError as e:
+            log_warning(e.reason)
+            should_continue = _request_url_error(template, retry_timeout)
+            if not should_continue:
+                raise
+        except socket.error as e:
+            log_warning(e.strerror)
             should_continue = _request_url_error(template, retry_timeout)
             if not should_continue:
                 raise
@@ -503,6 +547,39 @@ def _request_url_error(template, retry_timeout):
     return False
 
 
+class S3HTTPRedirectHandler(HTTPRedirectHandler):
+    """
+    A subclassed redirect handler for downloading Github assets from S3.
+
+    urllib will add the Authorization header to the redirected request to S3, which will result in a 400,
+    so we should remove said header on redirect.
+    """
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        if PY2:
+            # HTTPRedirectHandler is an old style class
+            request = HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
+        else:
+            request = super(S3HTTPRedirectHandler, self).redirect_request(req, fp, code, msg, headers, newurl)
+        del request.headers['Authorization']
+        return request
+
+
+def download_file(url, path, auth):
+    request = Request(url)
+    request.add_header('Accept', 'application/octet-stream')
+    request.add_header('Authorization', 'Basic '.encode('ascii') + auth)
+    opener = build_opener(S3HTTPRedirectHandler)
+    response = opener.open(request)
+
+    chunk_size = 16 * 1024
+    with open(path, 'wb') as f:
+        while True:
+            chunk = response.read(chunk_size)
+            if not chunk:
+                break
+            f.write(chunk)
+
+
 def get_authenticated_user(args):
     template = 'https://{0}/user'.format(get_github_api_host(args))
     data = retrieve_data(args, template, single_request=True)
@@ -513,7 +590,6 @@ def check_git_lfs_install():
     exit_code = subprocess.call(['git', 'lfs', 'version'])
     if exit_code != 0:
         log_error('The argument --lfs requires you to have Git LFS installed.\nYou can get it from https://git-lfs.github.com.')
-        sys.exit(1)
 
 
 def retrieve_repositories(args, authenticated_user):
@@ -524,8 +600,8 @@ def retrieve_repositories(args, authenticated_user):
         template = 'https://{0}/user/repos'.format(
             get_github_api_host(args))
     else:
-        if args.private:
-            log_error('Authenticated user is different from user being backed up, thus private repositories cannot be accessed')
+        if args.private and not args.organization:
+            log_warning('Authenticated user is different from user being backed up, thus private repositories cannot be accessed')
         template = 'https://{0}/users/{1}/repos'.format(
             get_github_api_host(args),
             args.user)
@@ -671,6 +747,10 @@ def backup_repositories(args, output_directory, repositories):
         if args.include_hooks or args.include_everything:
             backup_hooks(args, repo_cwd, repository, repos_template)
 
+        if args.include_releases or args.include_everything:
+            backup_releases(args, repo_cwd, repository, repos_template,
+                            include_assets=args.include_assets or args.include_everything)
+
     if args.incremental:
         open(last_update_path, 'w').write(last_update)
 
@@ -773,7 +853,7 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
                     args,
                     _pulls_template + '/{}'.format(pull['number']),
                     single_request=True
-                )
+                )[0]
 
     log_info('Saving {0} pull requests to disk'.format(
         len(list(pulls.keys()))))
@@ -852,6 +932,33 @@ def backup_hooks(args, repo_cwd, repository, repos_template):
         log_info("Unable to read hooks, skipping")
 
 
+def backup_releases(args, repo_cwd, repository, repos_template, include_assets=False):
+    repository_fullname = repository['full_name']
+
+    # give release files somewhere to live & log intent
+    release_cwd = os.path.join(repo_cwd, 'releases')
+    log_info('Retrieving {0} releases'.format(repository_fullname))
+    mkdir_p(repo_cwd, release_cwd)
+
+    query_args = {}
+
+    release_template = '{0}/{1}/releases'.format(repos_template, repository_fullname)
+    releases = retrieve_data(args, release_template, query_args=query_args)
+
+    # for each release, store it
+    log_info('Saving {0} releases to disk'.format(len(releases)))
+    for release in releases:
+        release_name = release['tag_name']
+        output_filepath = os.path.join(release_cwd, '{0}.json'.format(release_name))
+        with codecs.open(output_filepath, 'w+', encoding='utf-8') as f:
+            json_dump(release, f)
+
+        if include_assets:
+            assets = retrieve_data(args, release['assets_url'])
+            for asset in assets:
+                download_file(asset['url'], os.path.join(release_cwd, asset['name']), get_auth(args))
+
+
 def fetch_repository(name,
                      remote_url,
                      local_dir,

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = '0.21.0'
+__version__ = '0.24.0'