Release version 0.25.0

Issue 119: Change retrieve_data to be a generator

CHANGES.rst

@@ -1,33 +1,297 @@
 Changelog
 =========
 
+0.25.0 (2019-07-03)
+-------------------
+------------------------
+- Issue 119: Change retrieve_data to be a generator. [2a]
+
+  See issue #119.
+
+
+0.24.0 (2019-06-27)
+-------------------
+- QKT-45: include assets - update readme. [Ethan Timm]
+
+  update readme with flag information for including assets alongside their respective releases
+- Make assets it's own flag. [Harrison Wright]
+- Fix super call for python2. [Harrison Wright]
+- Fix redirect to s3. [Harrison Wright]
+- WIP: download assets. [Harrison Wright]
+- QKT-42: releases - add readme info. [ethan]
+- QKT-42 update: shorter command flag. [ethan]
+- QKT-42: support saving release information. [ethan]
+- Fix pull details. [Harrison Wright]
+
+
+0.23.0 (2019-06-04)
+-------------------
+- Avoid to crash in case of HTTP 502 error. [Gael de Chalendar]
+
+  Survive also on socket.error connections like on HTTPError or URLError.
+
+  This should solve issue #110.
+
+
+0.22.2 (2019-02-21)
+-------------------
+
+Fix
+~~~
+- Warn instead of error. [Jose Diaz-Gonzalez]
+
+  Refs #106
+
+
+0.22.1 (2019-02-21)
+-------------------
+- Log URL error https://github.com/josegonzalez/python-github-
+  backup/issues/105. [JOHN STETIC]
+
+
+0.22.0 (2019-02-01)
+-------------------
+- Remove unnecessary sys.exit call. [W. Harrison Wright]
+- Add org check to avoid incorrect log output. [W. Harrison Wright]
+- Fix accidental system exit with better logging strategy. [W. Harrison
+  Wright]
+
+
+0.21.1 (2018-12-25)
+-------------------
+- Mark options which are not included in --all. [Bernd]
+
+  As discussed in Issue #100
+
+
+0.21.0 (2018-11-28)
+-------------------
+- Correctly download repos when user arg != authenticated user. [W.
+  Harrison Wright]
+
+
+0.20.1 (2018-09-29)
+-------------------
+- Clone the specified user's gists, not the authenticated user. [W.
+  Harrison Wright]
+- Clone the specified user's starred repos, not the authenticated user.
+  [W. Harrison Wright]
+
+
+0.20.0 (2018-03-24)
+-------------------
+- Chore: drop Python 2.6. [Jose Diaz-Gonzalez]
+- Feat: simplify release script. [Jose Diaz-Gonzalez]
+
+
+0.19.2 (2018-03-24)
+-------------------
+
+Fix
+~~~
+- Cleanup pep8 violations. [Jose Diaz-Gonzalez]
+
+
+0.19.0 (2018-03-24)
+-------------------
+- Add additional output for the current request. [Robin Gloster]
+
+  This is useful to have some progress indication for huge repositories.
+- Add option to backup additional PR details. [Robin Gloster]
+
+  Some payload is only included when requesting a single pull request
+- Mark string as binary in comparison for skip_existing. [Johannes
+  Bornhold]
+
+  Found out that the flag "--skip-existing" did not work out as expected on Python
+  3.6. Tracked it down to the comparison which has to be against a string of bytes
+  in Python3.
+
+
+0.18.0 (2018-02-22)
+-------------------
+- Add option to fetch followers/following JSON data. [Stephen Greene]
+
+
+0.17.0 (2018-02-20)
+-------------------
+- Short circuit gists backup process. [W. Harrison Wright]
+- Formatting. [W. Harrison Wright]
+- Add ability to backup gists. [W. Harrison Wright]
+
+
+0.16.0 (2018-01-22)
+-------------------
+- Change option to --all-starred. [W. Harrison Wright]
+- JK don't update documentation. [W. Harrison Wright]
+- Put starred clone repoistories under a new option. [W. Harrison
+  Wright]
+- Add comment. [W. Harrison Wright]
+- Add ability to clone starred repos. [W. Harrison Wright]
+
+
+0.14.1 (2017-10-11)
+-------------------
+- Fix arg not defined error. [Edward Pfremmer]
+
+
+0.14.0 (2017-10-11)
+-------------------
+- Added a check to see if git-lfs is installed when doing an LFS clone.
+  [pieterclaerhout]
+- Added support for LFS clones. [pieterclaerhout]
+- Add pypi info to readme. [Albert Wang]
+- Explicitly support python 3 in package description. [Albert Wang]
+- Add couple examples to help new users. [Yusuf Tran]
+
+
+0.13.2 (2017-05-06)
+-------------------
+- Fix remotes while updating repository. [Dima Gerasimov]
+
+
+0.13.1 (2017-04-11)
+-------------------
+- Fix error when repository has no updated_at value. [Nicolai Ehemann]
+
+
+0.13.0 (2017-04-05)
+-------------------
+- Add OS check for OSX specific keychain args. [Martin O'Reilly]
+
+  Keychain arguments are only supported on Mac OSX.
+  Added check for operating system so we give a
+  "Keychain arguments are only supported on Mac OSX"
+  error message rather than a "No password item matching the
+  provided name and account could be found in the osx keychain"
+  error message
+- Add support for storing PAT in OSX keychain. [Martin O'Reilly]
+
+  Added additional optional arguments and README guidance for storing
+  and accessing a Github personal access token (PAT) in the OSX
+  keychain
+
+
+0.12.1 (2017-03-27)
+-------------------
+- Avoid remote branch name churn. [Chris Adams]
+
+  This avoids the backup output having lots of "[new branch]" messages
+  because removing the old remote name removed all of the existing branch
+  references.
+- Fix detection of bare git directories. [Andrzej Maczuga]
+
+
+0.12.0 (2016-11-22)
+-------------------
+
+Fix
+~~~
+- Properly import version from github_backup package. [Jose Diaz-
+  Gonzalez]
+- Support alternate git status output. [Jose Diaz-Gonzalez]
+
+Other
+~~~~~
+- Pep8: E501 line too long (83 > 79 characters) [Jose Diaz-Gonzalez]
+- Pep8: E128 continuation line under-indented for visual indent. [Jose
+  Diaz-Gonzalez]
+- Support archivization using bare git clones. [Andrzej Maczuga]
+- Fix typo, 3x. [Terrell Russell]
+
+
+0.11.0 (2016-10-26)
+-------------------
+- Support --token file:///home/user/token.txt (fixes gh-51) [Björn
+  Dahlgren]
+- Fix some linting. [Albert Wang]
+- Fix byte/string conversion for python 3. [Albert Wang]
+- Support python 3. [Albert Wang]
+- Encode special characters in password. [Remi Rampin]
+- Don't pretend program name is "Github Backup" [Remi Rampin]
+- Don't install over insecure connection. [Remi Rampin]
+
+  The git:// protocol is unauthenticated and unencrypted, and no longer advertised by GitHub. Using HTTPS shouldn't impact performance.
+
+
+0.10.3 (2016-08-21)
+-------------------
+- Fixes #29. [Jonas Michel]
+
+  Reporting an error when the user's rate limit is exceeded causes
+  the script to terminate after resuming execution from a rate limit
+  sleep. Instead of generating an explicit error we just want to
+  inform the user that the script is going to sleep until their rate
+  limit count resets.
+- Fixes #29. [Jonas Michel]
+
+  The errors list was not being cleared out after resuming a backup
+  from a rate limit sleep. When the backup was resumed, the non-empty
+  errors list caused the backup to quit after the next `retrieve_data`
+  request.
+
+
+0.10.2 (2016-08-21)
+-------------------
+- Add a note regarding git version requirement. [Jose Diaz-Gonzalez]
+
+  Closes #37
+
+
+0.10.0 (2016-08-18)
+-------------------
+- Implement incremental updates. [Robert Bradshaw]
+
+  Guarded with an --incremental flag.
+
+  Stores the time of the last update and only downloads issue and
+  pull request data since this time.  All other data is relatively
+  small (likely fetched with a single request) and so is simply
+  re-populated from scratch as before.
+
+
+0.9.0 (2016-03-29)
+------------------
+- Fix cloning private repos with basic auth or token. [Kazuki Suda]
+
+
+0.8.0 (2016-02-14)
+------------------
+- Don't store issues which are actually pull requests. [Enrico Tröger]
+
+  This prevents storing pull requests twice since the Github API returns
+  pull requests also as issues. Those issues will be skipped but only if
+  retrieving pull requests is requested as well.
+  Closes #23.
+
+
+0.7.0 (2016-02-02)
+------------------
+- Softly fail if not able to read hooks. [Albert Wang]
+- Add note about 2-factor auth. [Albert Wang]
+- Make user repository search go through endpoint capable of reading
+  private repositories. [Albert Wang]
+- Prompt for password if only username given. [Alex Hall]
+
+
 0.6.0 (2015-11-10)
 ------------------
-
 - Force proper remote url. [Jose Diaz-Gonzalez]
-
 - Improve error handling in case of HTTP errors. [Enrico Tröger]
 
   In case of a HTTP status code 404, the returned 'r' was never assigned.
   In case of URL errors which are not timeouts, we probably should bail
   out.
-
-
 - Add --hooks to also include web hooks into the backup. [Enrico Tröger]
-
 - Create the user specified output directory if it does not exist.
   [Enrico Tröger]
 
   Fixes #17.
-
-
 - Add missing auth argument to _get_response() [Enrico Tröger]
 
   When running unauthenticated and Github starts rate-limiting the client,
   github-backup crashes because the used auth variable in _get_response()
   was not available. This change should fix it.
-
-
 - Add repository URL to error message for non-existing repositories.
   [Enrico Tröger]
 
@@ -38,40 +302,28 @@ Changelog
 
 0.5.0 (2015-10-10)
 ------------------
-
 - Add release script. [Jose Diaz-Gonzalez]
-
 - Refactor to both simplify codepath as well as follow PEP8 standards.
   [Jose Diaz-Gonzalez]
-
 - Retry 3 times when the connection times out. [Mathijs Jonker]
-
 - Made unicode output defalut. [Kirill Grushetsky]
-
 - Import alphabetised. [Kirill Grushetsky]
-
 - Preserve Unicode characters in the output file. [Kirill Grushetsky]
 
   Added option to preserve Unicode characters in the output file
-
 - Josegonzales/python-github-backup#12 Added backup of labels and
   milestones. [aensley]
-
 - Fixed indent. [Mathijs Jonker]
-
 - Skip unitialized repo's. [mjonker-embed]
 
   These gave me errors which caused mails from crontab.
-
 - Added prefer-ssh. [mjonker-embed]
 
   Was needed for my back-up setup, code includes this but readme wasn't updated
-
 - Retry API requests which failed due to rate-limiting. [Chris Adams]
 
   This allows operation to continue, albeit at a slower pace,
   if you have enough data to trigger the API rate limits
-
 - Logging_subprocess: always log when a command fails. [Chris Adams]
 
   Previously git clones could fail without any indication
@@ -81,21 +333,15 @@ Changelog
   Now a non-zero return code will always output a message to
   stderr and will display the executed command so it can be
   rerun for troubleshooting.
-
-
 - Switch to using ssh_url. [Chris Adams]
 
   The previous commit used the wrong URL for a private repo. This was
   masked by the lack of error loging in logging_subprocess (which will be
   in a separate branch)
-
-
 - Add an option to prefer checkouts over SSH. [Chris Adams]
 
   This is really useful with private repos to avoid being nagged
   for credentials for every repository
-
-
 - Add pull request support. [Kevin Laude]
 
   Back up reporitory pull requests by passing the --include-pulls
@@ -107,8 +353,6 @@ Changelog
 
   Pull requests are automatically backed up when the --all argument is
   uesd.
-
-
 - Add GitHub Enterprise support. [Kevin Laude]
 
   Pass the -H or --github-host argument with a GitHub Enterprise hostname
@@ -118,35 +362,21 @@ Changelog
 
 0.2.0 (2014-09-22)
 ------------------
-
 - Add support for retrieving repositories. Closes #1. [Jose Diaz-
   Gonzalez]
-
 - Fix PEP8 violations. [Jose Diaz-Gonzalez]
-
 - Add authorization to header only if specified by user. [Ioannis
   Filippidis]
-
 - Fill out readme more. [Jose Diaz-Gonzalez]
-
 - Fix import. [Jose Diaz-Gonzalez]
-
 - Properly name readme. [Jose Diaz-Gonzalez]
-
 - Create MANIFEST.in. [Jose Diaz-Gonzalez]
-
 - Create .gitignore. [Jose Diaz-Gonzalez]
-
 - Create setup.py. [Jose Diaz-Gonzalez]
-
 - Create requirements.txt. [Jose Diaz-Gonzalez]
-
 - Create __init__.py. [Jose Diaz-Gonzalez]
-
 - Create LICENSE.txt. [Jose Diaz-Gonzalez]
-
 - Create README.md. [Jose Diaz-Gonzalez]
-
 - Create github-backup. [Jose Diaz-Gonzalez]
 
 

README.rst

@@ -2,8 +2,15 @@
 github-backup
 =============
 
+|PyPI| |Python Versions|
+
 backup a github user or organization
 
+Requirements
+============
+
+- GIT 1.9+
+
 Installation
 ============
 
@@ -13,25 +20,29 @@ Using PIP via PyPI::
 
 Using PIP via Github::
 
-    pip install git+git://github.com/josegonzalez/python-github-backup.git#egg=github-backup
+    pip install git+https://github.com/josegonzalez/python-github-backup.git#egg=github-backup
 
 Usage
 =====
 
 CLI Usage is as follows::
 
-    Github Backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN]
+    github-backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN]
-                     [-o OUTPUT_DIRECTORY] [--starred] [--watched] [--all]
+                  [-o OUTPUT_DIRECTORY] [-i] [--starred] [--all-starred]
-                     [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--watched] [--followers] [--following] [--all]
-                     [--pull-comments] [--pull-commits] [--repositories]
+                  [--issues] [--issue-comments] [--issue-events] [--pulls]
-                     [--wikis] [--labels] [--hooks] [--skip-existing]
+                  [--pull-comments] [--pull-commits] [--labels] [--hooks]
-                     [-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
+                  [--milestones] [--repositories] [--releases] [--assets]
-                     [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
+                  [--bare] [--lfs] [--wikis] [--gists] [--starred-gists]
-                     [--prefer-ssh] [-v]
+                  [--skip-existing]
-                     USER
+                  [-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
+                  [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
+                  [--prefer-ssh] [-v]
+                  [--keychain-name OSX_KEYCHAIN_ITEM_NAME]
+                  [--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT]
+                  USER
 
-
+    Backup a github account
-    Backup a github users account
 
     positional arguments:
       USER                  github username
@@ -41,13 +52,19 @@ CLI Usage is as follows::
       -u USERNAME, --username USERNAME
                             username for basic auth
       -p PASSWORD, --password PASSWORD
-                            password for basic auth
+                            password for basic auth. If a username is given but
+                            not a password, the password will be prompted for.
       -t TOKEN, --token TOKEN
-                            personal access or OAuth token
+                            personal access or OAuth token, or path to token
+                            (file://...)
       -o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                             directory at which to backup the repositories
-      --starred             include starred repositories in backup
+      -i, --incremental     incremental backup
+      --starred             include JSON output of starred repositories in backup
+      --all-starred         include starred repositories in backup
       --watched             include watched repositories in backup
+      --followers           include JSON output of followers in backup
+      --following           include JSON output of following users in backup
       --all                 include everything in backup
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
@@ -55,10 +72,19 @@ CLI Usage is as follows::
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-commits        include pull request commits in backup
-      --repositories        include repository clone in backup
-      --wikis               include wiki clone in backup
       --labels              include labels in backup
-      --hooks               include web hooks in backup (works only when authenticated)
+      --hooks               include hooks in backup (works only when
+                            authenticated)
+      --milestones          include milestones in backup
+      --repositories        include repository clone in backup
+      --releases            include repository releases' information without assets or binaries
+      --assets              include assets alongside release information; only applies if including releases
+      --bare                clone bare repositories
+      --lfs                 clone LFS repositories (requires Git LFS to be
+                            installed, https://git-lfs.github.com)
+      --wikis               include wiki clone in backup
+      --gists               include gists in backup
+      --starred-gists       include starred gists in backup
       --skip-existing       skip project if a backup directory exists
       -L [LANGUAGES [LANGUAGES ...]], --languages [LANGUAGES [LANGUAGES ...]]
                             only allow these languages
@@ -66,13 +92,67 @@ CLI Usage is as follows::
                             python regex to match names against
       -H GITHUB_HOST, --github-host GITHUB_HOST
                             GitHub Enterprise hostname
-      -O, --organization    whether or not this is a query for an organization
+      -O, --organization    whether or not this is an organization user
       -R REPOSITORY, --repository REPOSITORY
                             name of repository to limit backup to
       -P, --private         include private repositories
       -F, --fork            include forked repositories
       --prefer-ssh          Clone repositories using SSH instead of HTTPS
       -v, --version         show program's version number and exit
+      --keychain-name OSX_KEYCHAIN_ITEM_NAME
+                            OSX ONLY: name field of password item in OSX keychain
+                            that holds the personal access or OAuth token
+      --keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT
+                            OSX ONLY: account field of password item in OSX
+                            keychain that holds the personal access or OAuth token
 
 
 The package can be used to backup an *entire* organization or repository, including issues and wikis in the most appropriate format (clones for wikis, json files for issues).
+
+Authentication
+==============
+
+Note: Password-based authentication will fail if you have two-factor authentication enabled.
+
+Using the Keychain on Mac OSX
+=============================
+Note: On Mac OSX the token can be stored securely in the user's keychain. To do this:
+
+1. Open Keychain from "Applications -> Utilities -> Keychain Access"
+2. Add a new password item using "File -> New Password Item"
+3. Enter a name in the "Keychain Item Name" box. You must provide this name to github-backup using the --keychain-name argument.
+4. Enter an account name in the "Account Name" box, enter your Github username as set above. You must provide this name to github-backup using the --keychain-account argument.
+5. Enter your Github personal access token in the "Password" box
+
+Note:  When you run github-backup, you will be asked whether you want to allow "security" to use your confidential information stored in your keychain. You have two options:
+
+1. **Allow:** In this case you will need to click "Allow" each time you run `github-backup`
+2. **Always Allow:** In this case, you will not be asked for permission when you run `github-backup` in future. This is less secure, but is required if you want to schedule `github-backup` to run automatically
+
+About Git LFS
+=============
+
+When you use the "--lfs" option, you will need to make sure you have Git LFS installed.
+
+Instructions on how to do this can be found on https://git-lfs.github.com.
+
+Examples
+========
+
+Backup all repositories::
+
+    export ACCESS_TOKEN=SOME-GITHUB-TOKEN
+    github-backup WhiteHouse --token $ACCESS_TOKEN --organization --output-directory /tmp/white-house --repositories
+
+Backup a single organization repository with everything else (wiki, pull requests, comments, issues etc)::
+
+    export ACCESS_TOKEN=SOME-GITHUB-TOKEN
+    ORGANIZATION=docker
+    REPO=cli
+    # e.g. git@github.com:docker/cli.git
+    github-backup $ORGANIZATION -P -t $ACCESS_TOKEN -o . --all -O -R $REPO
+
+.. |PyPI| image:: https://img.shields.io/pypi/v/github-backup.svg
+   :target: https://pypi.python.org/pypi/github-backup/
+.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/github-backup.svg
+   :target: https://github.com/albertyw/github-backup

bin/github-backup

@@ -1,12 +1,14 @@
 #!/usr/bin/env python
 
 from __future__ import print_function
+import socket
 
 import argparse
 import base64
 import calendar
 import codecs
 import errno
+import getpass
 import json
 import logging
 import os
@@ -15,8 +17,29 @@ import select
 import subprocess
 import sys
 import time
-import urllib
+import platform
-import urllib2
+PY2 = False
+try:
+    # python 3
+    from urllib.parse import urlparse
+    from urllib.parse import quote as urlquote
+    from urllib.parse import urlencode
+    from urllib.error import HTTPError, URLError
+    from urllib.request import urlopen
+    from urllib.request import Request
+    from urllib.request import HTTPRedirectHandler
+    from urllib.request import build_opener
+except ImportError:
+    # python 2
+    PY2 = True
+    from urlparse import urlparse
+    from urllib import quote as urlquote
+    from urllib import urlencode
+    from urllib2 import HTTPError, URLError
+    from urllib2 import urlopen
+    from urllib2 import Request
+    from urllib2 import HTTPRedirectHandler
+    from urllib2 import build_opener
 
 from github_backup import __version__
 
@@ -24,16 +47,17 @@ FNULL = open(os.devnull, 'w')
 
 
 def log_error(message):
-    if type(message) == str:
+    """
-        message = [message]
+    Log message (str) or messages (List[str]) to stderr and exit with status 1
-
+    """
-    for msg in message:
+    log_warning(message)
-        sys.stderr.write("{0}\n".format(msg))
-
     sys.exit(1)
 
 
 def log_info(message):
+    """
+    Log message (str) or messages (List[str]) to stdout
+    """
     if type(message) == str:
         message = [message]
 
@@ -41,6 +65,17 @@ def log_info(message):
         sys.stdout.write("{0}\n".format(msg))
 
 
+def log_warning(message):
+    """
+    Log message (str) or messages (List[str]) to stderr
+    """
+    if type(message) == str:
+        message = [message]
+
+    for msg in message:
+        sys.stderr.write("{0}\n".format(msg))
+
+
 def logging_subprocess(popenargs,
                        logger,
                        stdout_log_level=logging.DEBUG,
@@ -53,11 +88,15 @@ def logging_subprocess(popenargs,
     """
     child = subprocess.Popen(popenargs, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE, **kwargs)
+    if sys.platform == 'win32':
+        log_info("Windows operating system detected - no subprocess logging will be returned")
 
     log_level = {child.stdout: stdout_log_level,
                  child.stderr: stderr_log_level}
 
     def check_io():
+        if sys.platform == 'win32':
+            return
         ready_to_read = select.select([child.stdout, child.stderr],
                                       [],
                                       [],
@@ -78,8 +117,8 @@ def logging_subprocess(popenargs,
     rc = child.wait()
 
     if rc != 0:
-        print(u'{} returned {}:'.format(popenargs[0], rc), file=sys.stderr)
+        print('{} returned {}:'.format(popenargs[0], rc), file=sys.stderr)
-        print('\t', u' '.join(popenargs), file=sys.stderr)
+        print('\t', ' '.join(popenargs), file=sys.stderr)
 
     return rc
 
@@ -95,9 +134,19 @@ def mkdir_p(*args):
                 raise
 
 
+def mask_password(url, secret='*****'):
+    parsed = urlparse(url)
+
+    if not parsed.password:
+        return url
+    elif parsed.password == 'x-oauth-basic':
+        return url.replace(parsed.username, secret)
+
+    return url.replace(parsed.password, secret)
+
+
 def parse_args():
-    parser = argparse.ArgumentParser(description='Backup a github account',
+    parser = argparse.ArgumentParser(description='Backup a github account')
-                                     prog='Github Backup')
     parser.add_argument('user',
                         metavar='USER',
                         type=str,
@@ -109,28 +158,47 @@ def parse_args():
     parser.add_argument('-p',
                         '--password',
                         dest='password',
-                        help='password for basic auth')
+                        help='password for basic auth. '
+                             'If a username is given but not a password, the '
+                             'password will be prompted for.')
     parser.add_argument('-t',
                         '--token',
                         dest='token',
-                        help='personal access or OAuth token')
+                        help='personal access or OAuth token, or path to token (file://...)')  # noqa
     parser.add_argument('-o',
                         '--output-directory',
                         default='.',
                         dest='output_directory',
                         help='directory at which to backup the repositories')
+    parser.add_argument('-i',
+                        '--incremental',
+                        action='store_true',
+                        dest='incremental',
+                        help='incremental backup')
     parser.add_argument('--starred',
                         action='store_true',
                         dest='include_starred',
-                        help='include starred repositories in backup')
+                        help='include JSON output of starred repositories in backup')
+    parser.add_argument('--all-starred',
+                        action='store_true',
+                        dest='all_starred',
+                        help='include starred repositories in backup [*]')
     parser.add_argument('--watched',
                         action='store_true',
                         dest='include_watched',
-                        help='include watched repositories in backup')
+                        help='include JSON output of watched repositories in backup')
+    parser.add_argument('--followers',
+                        action='store_true',
+                        dest='include_followers',
+                        help='include JSON output of followers in backup')
+    parser.add_argument('--following',
+                        action='store_true',
+                        dest='include_following',
+                        help='include JSON output of following users in backup')
     parser.add_argument('--all',
                         action='store_true',
                         dest='include_everything',
-                        help='include everything in backup')
+                        help='include everything in backup (not including [*])')
     parser.add_argument('--issues',
                         action='store_true',
                         dest='include_issues',
@@ -155,6 +223,10 @@ def parse_args():
                         action='store_true',
                         dest='include_pull_commits',
                         help='include pull request commits in backup')
+    parser.add_argument('--pull-details',
+                        action='store_true',
+                        dest='include_pull_details',
+                        help='include more pull request details in backup [*]')
     parser.add_argument('--labels',
                         action='store_true',
                         dest='include_labels',
@@ -162,7 +234,7 @@ def parse_args():
     parser.add_argument('--hooks',
                         action='store_true',
                         dest='include_hooks',
-                        help='include hooks in backup (works only when authenticated)')
+                        help='include hooks in backup (works only when authenticated)')  # noqa
     parser.add_argument('--milestones',
                         action='store_true',
                         dest='include_milestones',
@@ -171,10 +243,26 @@ def parse_args():
                         action='store_true',
                         dest='include_repository',
                         help='include repository clone in backup')
+    parser.add_argument('--bare',
+                        action='store_true',
+                        dest='bare_clone',
+                        help='clone bare repositories')
+    parser.add_argument('--lfs',
+                        action='store_true',
+                        dest='lfs_clone',
+                        help='clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com) [*]')
     parser.add_argument('--wikis',
                         action='store_true',
                         dest='include_wiki',
                         help='include wiki clone in backup')
+    parser.add_argument('--gists',
+                        action='store_true',
+                        dest='include_gists',
+                        help='include gists in backup [*]')
+    parser.add_argument('--starred-gists',
+                        action='store_true',
+                        dest='include_starred_gists',
+                        help='include starred gists in backup [*]')
     parser.add_argument('--skip-existing',
                         action='store_true',
                         dest='skip_existing',
@@ -204,32 +292,80 @@ def parse_args():
     parser.add_argument('-P', '--private',
                         action='store_true',
                         dest='private',
-                        help='include private repositories')
+                        help='include private repositories [*]')
     parser.add_argument('-F', '--fork',
                         action='store_true',
                         dest='fork',
-                        help='include forked repositories')
+                        help='include forked repositories [*]')
     parser.add_argument('--prefer-ssh',
                         action='store_true',
                         help='Clone repositories using SSH instead of HTTPS')
     parser.add_argument('-v', '--version',
                         action='version',
                         version='%(prog)s ' + __version__)
+    parser.add_argument('--keychain-name',
+                        dest='osx_keychain_item_name',
+                        help='OSX ONLY: name field of password item in OSX keychain that holds the personal access or OAuth token')
+    parser.add_argument('--keychain-account',
+                        dest='osx_keychain_item_account',
+                        help='OSX ONLY: account field of password item in OSX keychain that holds the personal access or OAuth token')
+    parser.add_argument('--releases',
+                        action='store_true',
+                        dest='include_releases',
+                        help='include release information, not including assets or binaries'
+                        )
+    parser.add_argument('--assets',
+                        action='store_true',
+                        dest='include_assets',
+                        help='include assets alongside release information; only applies if including releases')
     return parser.parse_args()
 
 
-def get_auth(args):
+def get_auth(args, encode=True):
     auth = None
-    if args.token:
+
-        auth = base64.b64encode(args.token + ':' + 'x-oauth-basic')
+    if args.osx_keychain_item_name:
-    elif args.username and args.password:
+        if not args.osx_keychain_item_account:
-        auth = base64.b64encode(args.username + ':' + args.password)
+            log_error('You must specify both name and account fields for osx keychain password items')
-    elif args.username and not args.password:
+        else:
-        log_error('You must specify a password for basic auth')
+            if platform.system() != 'Darwin':
-    elif args.password and not args.username:
+                log_error("Keychain arguments are only supported on Mac OSX")
+            try:
+                with open(os.devnull, 'w') as devnull:
+                    token = (subprocess.check_output([
+                        'security', 'find-generic-password',
+                        '-s', args.osx_keychain_item_name,
+                        '-a', args.osx_keychain_item_account,
+                        '-w'], stderr=devnull).strip())
+                auth = token + ':' + 'x-oauth-basic'
+            except:
+                log_error('No password item matching the provided name and account could be found in the osx keychain.')
+    elif args.osx_keychain_item_account:
+        log_error('You must specify both name and account fields for osx keychain password items')
+    elif args.token:
+        _path_specifier = 'file://'
+        if args.token.startswith(_path_specifier):
+            args.token = open(args.token[len(_path_specifier):],
+                              'rt').readline().strip()
+        auth = args.token + ':' + 'x-oauth-basic'
+    elif args.username:
+        if not args.password:
+            args.password = getpass.getpass()
+        if encode:
+            password = args.password
+        else:
+            password = urlquote(args.password)
+        auth = args.username + ':' + password
+    elif args.password:
         log_error('You must specify a username for basic auth')
 
-    return auth
+    if not auth:
+        return None
+
+    if not encode:
+        return auth
+
+    return base64.b64encode(auth.encode('ascii'))
 
 
 def get_github_api_host(args):
@@ -241,7 +377,7 @@ def get_github_api_host(args):
     return host
 
 
-def get_github_ssh_host(args):
+def get_github_host(args):
     if args.github_host:
         host = args.github_host
     else:
@@ -250,12 +386,31 @@ def get_github_ssh_host(args):
     return host
 
 
-def retrieve_data(args, template, query_args=None, single_request=False):
+def get_github_repo_url(args, repository):
+    if args.prefer_ssh:
+        return repository['ssh_url']
+
+    if repository.get('is_gist'):
+        return repository['git_pull_url']
+
+    auth = get_auth(args, False)
+    if auth:
+        repo_url = 'https://{0}@{1}/{2}/{3}.git'.format(
+            auth,
+            get_github_host(args),
+            repository['owner']['login'],
+            repository['name'])
+    else:
+        repo_url = repository['clone_url']
+
+    return repo_url
+
+
+def retrieve_data_gen(args, template, query_args=None, single_request=False):
     auth = get_auth(args)
     query_args = get_query_args(query_args)
     per_page = 100
     page = 0
-    data = []
 
     while True:
         page = page + 1
@@ -264,19 +419,30 @@ def retrieve_data(args, template, query_args=None, single_request=False):
 
         status_code = int(r.getcode())
 
+        retries = 0
+        while retries < 3 and status_code == 502:
+            print('API request returned HTTP 502: Bad Gateway. Retrying in 5 seconds')
+            retries += 1
+            time.sleep(5)
+            request = _construct_request(per_page, page, query_args, template, auth)  # noqa
+            r, errors = _get_response(request, auth, template)
+
+            status_code = int(r.getcode())
+
         if status_code != 200:
             template = 'API request returned HTTP {0}: {1}'
             errors.append(template.format(status_code, r.reason))
             log_error(errors)
 
-        response = json.loads(r.read())
+        response = json.loads(r.read().decode('utf-8'))
         if len(errors) == 0:
             if type(response) == list:
-                data.extend(response)
+                for resp in response:
+                    yield resp
                 if len(response) < per_page:
                     break
             elif type(response) == dict and single_request:
-                data.append(response)
+                yield response
 
         if len(errors) > 0:
             log_error(errors)
@@ -284,8 +450,8 @@ def retrieve_data(args, template, query_args=None, single_request=False):
         if single_request:
             break
 
-    return data
+def retrieve_data(args, template, query_args=None, single_request=False):
-
+    return list(retrieve_data_gen(args, template, query_args, single_request))
 
 def get_query_args(query_args=None):
     if not query_args:
@@ -301,11 +467,17 @@ def _get_response(request, auth, template):
     while True:
         should_continue = False
         try:
-            r = urllib2.urlopen(request)
+            r = urlopen(request)
-        except urllib2.HTTPError as exc:
+        except HTTPError as exc:
             errors, should_continue = _request_http_error(exc, auth, errors)  # noqa
             r = exc
-        except urllib2.URLError:
+        except URLError as e:
+            log_warning(e.reason)
+            should_continue = _request_url_error(template, retry_timeout)
+            if not should_continue:
+                raise
+        except socket.error as e:
+            log_warning(e.strerror)
             should_continue = _request_url_error(template, retry_timeout)
             if not should_continue:
                 raise
@@ -318,14 +490,15 @@ def _get_response(request, auth, template):
 
 
 def _construct_request(per_page, page, query_args, template, auth):
-    querystring = urllib.urlencode(dict({
+    querystring = urlencode(dict(list({
         'per_page': per_page,
         'page': page
-    }.items() + query_args.items()))
+    }.items()) + list(query_args.items())))
 
-    request = urllib2.Request(template + '?' + querystring)
+    request = Request(template + '?' + querystring)
     if auth is not None:
-        request.add_header('Authorization', 'Basic ' + auth)
+        request.add_header('Authorization', 'Basic '.encode('ascii') + auth)
+    log_info('Requesting {}?{}'.format(template, querystring))
     return request
 
 
@@ -352,10 +525,9 @@ def _request_http_error(exc, auth, errors):
         print('Exceeded rate limit of {} requests; waiting {} seconds to reset'.format(limit, delta),  # noqa
               file=sys.stderr)
 
-        ratelimit_error = 'No more requests remaining'
         if auth is None:
-            ratelimit_error += '; authenticate to raise your GitHub rate limit'  # noqa
+            print('Hint: Authenticate to raise your GitHub rate limit',
-        errors.append(ratelimit_error)
+                  file=sys.stderr)
 
         time.sleep(delta)
         should_continue = True
@@ -375,12 +547,65 @@ def _request_url_error(template, retry_timeout):
     return False
 
 
-def retrieve_repositories(args):
+class S3HTTPRedirectHandler(HTTPRedirectHandler):
+    """
+    A subclassed redirect handler for downloading Github assets from S3.
+
+    urllib will add the Authorization header to the redirected request to S3, which will result in a 400,
+    so we should remove said header on redirect.
+    """
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        if PY2:
+            # HTTPRedirectHandler is an old style class
+            request = HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
+        else:
+            request = super(S3HTTPRedirectHandler, self).redirect_request(req, fp, code, msg, headers, newurl)
+        del request.headers['Authorization']
+        return request
+
+
+def download_file(url, path, auth):
+    request = Request(url)
+    request.add_header('Accept', 'application/octet-stream')
+    request.add_header('Authorization', 'Basic '.encode('ascii') + auth)
+    opener = build_opener(S3HTTPRedirectHandler)
+    response = opener.open(request)
+
+    chunk_size = 16 * 1024
+    with open(path, 'wb') as f:
+        while True:
+            chunk = response.read(chunk_size)
+            if not chunk:
+                break
+            f.write(chunk)
+
+
+def get_authenticated_user(args):
+    template = 'https://{0}/user'.format(get_github_api_host(args))
+    data = retrieve_data(args, template, single_request=True)
+    return data[0]
+
+
+def check_git_lfs_install():
+    exit_code = subprocess.call(['git', 'lfs', 'version'])
+    if exit_code != 0:
+        log_error('The argument --lfs requires you to have Git LFS installed.\nYou can get it from https://git-lfs.github.com.')
+
+
+def retrieve_repositories(args, authenticated_user):
     log_info('Retrieving repositories')
     single_request = False
-    template = 'https://{0}/users/{1}/repos'.format(
+    if args.user == authenticated_user['login']:
-        get_github_api_host(args),
+        # we must use the /user/repos API to be able to access private repos
-        args.user)
+        template = 'https://{0}/user/repos'.format(
+            get_github_api_host(args))
+    else:
+        if args.private and not args.organization:
+            log_warning('Authenticated user is different from user being backed up, thus private repositories cannot be accessed')
+        template = 'https://{0}/users/{1}/repos'.format(
+            get_github_api_host(args),
+            args.user)
+
     if args.organization:
         template = 'https://{0}/orgs/{1}/repos'.format(
             get_github_api_host(args),
@@ -393,11 +618,45 @@ def retrieve_repositories(args):
             args.user,
             args.repository)
 
-    return retrieve_data(args, template, single_request=single_request)
+    repos = retrieve_data(args, template, single_request=single_request)
+
+    if args.all_starred:
+        starred_template = 'https://{0}/users/{1}/starred'.format(get_github_api_host(args), args.user)
+        starred_repos = retrieve_data(args, starred_template, single_request=False)
+        # flag each repo as starred for downstream processing
+        for item in starred_repos:
+            item.update({'is_starred': True})
+        repos.extend(starred_repos)
+
+    if args.include_gists:
+        gists_template = 'https://{0}/users/{1}/gists'.format(get_github_api_host(args), args.user)
+        gists = retrieve_data(args, gists_template, single_request=False)
+        # flag each repo as a gist for downstream processing
+        for item in gists:
+            item.update({'is_gist': True})
+        repos.extend(gists)
+
+    if args.include_starred_gists:
+        starred_gists_template = 'https://{0}/gists/starred'.format(get_github_api_host(args))
+        starred_gists = retrieve_data(args, starred_gists_template, single_request=False)
+        # flag each repo as a starred gist for downstream processing
+        for item in starred_gists:
+            item.update({'is_gist': True,
+                         'is_starred': True})
+        repos.extend(starred_gists)
+
+    return repos
 
 
-def filter_repositories(args, repositories):
+def filter_repositories(args, unfiltered_repositories):
     log_info('Filtering repositories')
+
+    repositories = []
+    for r in unfiltered_repositories:
+        # gists can be anonymous, so need to safely check owner
+        if r.get('owner', {}).get('login') == args.user or r.get('is_starred'):
+            repositories.append(r)
+
     name_regex = None
     if args.name_regex:
         name_regex = re.compile(args.name_regex)
@@ -407,11 +666,11 @@ def filter_repositories(args, repositories):
         languages = [x.lower() for x in args.languages]
 
     if not args.fork:
-        repositories = [r for r in repositories if not r['fork']]
+        repositories = [r for r in repositories if not r.get('fork')]
     if not args.private:
-        repositories = [r for r in repositories if not r['private']]
+        repositories = [r for r in repositories if not r.get('private') or r.get('public')]
     if languages:
-        repositories = [r for r in repositories if r['language'] and r['language'].lower() in languages]  # noqa
+        repositories = [r for r in repositories if r.get('language') and r.get('language').lower() in languages]  # noqa
     if name_regex:
         repositories = [r for r in repositories if name_regex.match(r['name'])]
 
@@ -422,28 +681,56 @@ def backup_repositories(args, output_directory, repositories):
     log_info('Backing up repositories')
     repos_template = 'https://{0}/repos'.format(get_github_api_host(args))
 
-    for repository in repositories:
+    if args.incremental:
-        backup_cwd = os.path.join(output_directory, 'repositories')
+        last_update = max(list(repository['updated_at'] for repository in repositories) or [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime())])  # noqa
-        repo_cwd = os.path.join(backup_cwd, repository['name'])
+        last_update_path = os.path.join(output_directory, 'last_update')
-        repo_dir = os.path.join(repo_cwd, 'repository')
+        if os.path.exists(last_update_path):
-
+            args.since = open(last_update_path).read().strip()
-        if args.prefer_ssh:
-            repo_url = repository['ssh_url']
         else:
-            repo_url = repository['clone_url']
+            args.since = None
+    else:
+        args.since = None
 
-        if args.include_repository or args.include_everything:
+    for repository in repositories:
-            fetch_repository(repository['name'],
+        if repository.get('is_gist'):
+            repo_cwd = os.path.join(output_directory, 'gists', repository['id'])
+        elif repository.get('is_starred'):
+            # put starred repos in -o/starred/${owner}/${repo} to prevent collision of
+            # any repositories with the same name
+            repo_cwd = os.path.join(output_directory, 'starred', repository['owner']['login'], repository['name'])
+        else:
+            repo_cwd = os.path.join(output_directory, 'repositories', repository['name'])
+
+        repo_dir = os.path.join(repo_cwd, 'repository')
+        repo_url = get_github_repo_url(args, repository)
+
+        include_gists = (args.include_gists or args.include_starred_gists)
+        if (args.include_repository or args.include_everything) \
+                or (include_gists and repository.get('is_gist')):
+            repo_name = repository.get('name') if not repository.get('is_gist') else repository.get('id')
+            fetch_repository(repo_name,
                              repo_url,
                              repo_dir,
-                             skip_existing=args.skip_existing)
+                             skip_existing=args.skip_existing,
+                             bare_clone=args.bare_clone,
+                             lfs_clone=args.lfs_clone)
+
+            if repository.get('is_gist'):
+                # dump gist information to a file as well
+                output_file = '{0}/gist.json'.format(repo_cwd)
+                with codecs.open(output_file, 'w', encoding='utf-8') as f:
+                    json_dump(repository, f)
+
+                continue  # don't try to back anything else for a gist; it doesn't exist
 
         download_wiki = (args.include_wiki or args.include_everything)
         if repository['has_wiki'] and download_wiki:
             fetch_repository(repository['name'],
                              repo_url.replace('.git', '.wiki.git'),
                              os.path.join(repo_cwd, 'wiki'),
-                             skip_existing=args.skip_existing)
+                             skip_existing=args.skip_existing,
+                             bare_clone=args.bare_clone,
+                             lfs_clone=args.lfs_clone)
 
         if args.include_issues or args.include_everything:
             backup_issues(args, repo_cwd, repository, repos_template)
@@ -460,6 +747,13 @@ def backup_repositories(args, output_directory, repositories):
         if args.include_hooks or args.include_everything:
             backup_hooks(args, repo_cwd, repository, repos_template)
 
+        if args.include_releases or args.include_everything:
+            backup_releases(args, repo_cwd, repository, repos_template,
+                            include_assets=args.include_assets or args.include_everything)
+
+    if args.incremental:
+        open(last_update_path, 'w').write(last_update)
+
 
 def backup_issues(args, repo_cwd, repository, repos_template):
     has_issues_dir = os.path.isdir('{0}/issues/.git'.format(repo_cwd))
@@ -471,26 +765,42 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     mkdir_p(repo_cwd, issue_cwd)
 
     issues = {}
+    issues_skipped = 0
+    issues_skipped_message = ''
     _issue_template = '{0}/{1}/issues'.format(repos_template,
                                               repository['full_name'])
 
+    should_include_pulls = args.include_pulls or args.include_everything
     issue_states = ['open', 'closed']
     for issue_state in issue_states:
         query_args = {
             'filter': 'all',
             'state': issue_state
         }
+        if args.since:
+            query_args['since'] = args.since
 
         _issues = retrieve_data(args,
                                 _issue_template,
                                 query_args=query_args)
         for issue in _issues:
+            # skip pull requests which are also returned as issues
+            # if retrieving pull requests is requested as well
+            if 'pull_request' in issue and should_include_pulls:
+                issues_skipped += 1
+                continue
+
             issues[issue['number']] = issue
 
-    log_info('Saving {0} issues to disk'.format(len(issues.keys())))
+    if issues_skipped:
+        issues_skipped_message = ' (skipped {0} pull requests)'.format(
+            issues_skipped)
+
+    log_info('Saving {0} issues to disk{1}'.format(
+        len(list(issues.keys())), issues_skipped_message))
     comments_template = _issue_template + '/{0}/comments'
     events_template = _issue_template + '/{0}/events'
-    for number, issue in issues.iteritems():
+    for number, issue in list(issues.items()):
         if args.include_issue_comments or args.include_everything:
             template = comments_template.format(number)
             issues[number]['comment_data'] = retrieve_data(args, template)
@@ -515,24 +825,44 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
     pulls = {}
     _pulls_template = '{0}/{1}/pulls'.format(repos_template,
                                              repository['full_name'])
+    query_args = {
+        'filter': 'all',
+        'state': 'all',
+        'sort': 'updated',
+        'direction': 'desc',
+    }
 
-    pull_states = ['open', 'closed']
+    if not args.include_pull_details:
-    for pull_state in pull_states:
+        pull_states = ['open', 'closed']
-        query_args = {
+        for pull_state in pull_states:
-            'filter': 'all',
+            query_args['state'] = pull_state
-            'state': pull_state
+            _pulls = retrieve_data_gen(args,
-        }
+                                   _pulls_template,
-
+                                   query_args=query_args)
-        _pulls = retrieve_data(args,
+            for pull in _pulls:
+                if args.since and pull['updated_at'] < args.since:
+                    break
+                if not args.since or pull['updated_at'] >= args.since:
+                    pulls[pull['number']] = pull
+    else:
+        _pulls = retrieve_data_gen(args,
                                _pulls_template,
                                query_args=query_args)
         for pull in _pulls:
-            pulls[pull['number']] = pull
+            if args.since and pull['updated_at'] < args.since:
+                break
+            if not args.since or pull['updated_at'] >= args.since:
+                pulls[pull['number']] = retrieve_data(
+                    args,
+                    _pulls_template + '/{}'.format(pull['number']),
+                    single_request=True
+                )[0]
 
-    log_info('Saving {0} pull requests to disk'.format(len(pulls.keys())))
+    log_info('Saving {0} pull requests to disk'.format(
+        len(list(pulls.keys()))))
     comments_template = _pulls_template + '/{0}/comments'
     commits_template = _pulls_template + '/{0}/commits'
-    for number, pull in pulls.iteritems():
+    for number, pull in list(pulls.items()):
         if args.include_pull_comments or args.include_everything:
             template = comments_template.format(number)
             pulls[number]['comment_data'] = retrieve_data(args, template)
@@ -566,8 +896,9 @@ def backup_milestones(args, repo_cwd, repository, repos_template):
     for milestone in _milestones:
         milestones[milestone['number']] = milestone
 
-    log_info('Saving {0} milestones to disk'.format(len(milestones.keys())))
+    log_info('Saving {0} milestones to disk'.format(
-    for number, milestone in milestones.iteritems():
+        len(list(milestones.keys()))))
+    for number, milestone in list(milestones.items()):
         milestone_file = '{0}/{1}.json'.format(milestone_cwd, number)
         with codecs.open(milestone_file, 'w', encoding='utf-8') as f:
             json_dump(milestone, f)
@@ -593,41 +924,111 @@ def backup_hooks(args, repo_cwd, repository, repos_template):
     hook_cwd = os.path.join(repo_cwd, 'hooks')
     output_file = '{0}/hooks.json'.format(hook_cwd)
     template = '{0}/{1}/hooks'.format(repos_template,
-                                       repository['full_name'])
+                                      repository['full_name'])
-    _backup_data(args,
+    try:
-                 'hooks',
+        _backup_data(args,
-                 template,
+                     'hooks',
-                 output_file,
+                     template,
-                 hook_cwd)
+                     output_file,
+                     hook_cwd)
+    except SystemExit:
+        log_info("Unable to read hooks, skipping")
 
 
-def fetch_repository(name, remote_url, local_dir, skip_existing=False):
+def backup_releases(args, repo_cwd, repository, repos_template, include_assets=False):
-    clone_exists = os.path.exists(os.path.join(local_dir, '.git'))
+    repository_fullname = repository['full_name']
+
+    # give release files somewhere to live & log intent
+    release_cwd = os.path.join(repo_cwd, 'releases')
+    log_info('Retrieving {0} releases'.format(repository_fullname))
+    mkdir_p(repo_cwd, release_cwd)
+
+    query_args = {}
+
+    release_template = '{0}/{1}/releases'.format(repos_template, repository_fullname)
+    releases = retrieve_data(args, release_template, query_args=query_args)
+
+    # for each release, store it
+    log_info('Saving {0} releases to disk'.format(len(releases)))
+    for release in releases:
+        release_name = release['tag_name']
+        output_filepath = os.path.join(release_cwd, '{0}.json'.format(release_name))
+        with codecs.open(output_filepath, 'w+', encoding='utf-8') as f:
+            json_dump(release, f)
+
+        if include_assets:
+            assets = retrieve_data(args, release['assets_url'])
+            for asset in assets:
+                download_file(asset['url'], os.path.join(release_cwd, asset['name']), get_auth(args))
+
+
+def fetch_repository(name,
+                     remote_url,
+                     local_dir,
+                     skip_existing=False,
+                     bare_clone=False,
+                     lfs_clone=False):
+    if bare_clone:
+        if os.path.exists(local_dir):
+            clone_exists = subprocess.check_output(['git',
+                                                    'rev-parse',
+                                                    '--is-bare-repository'],
+                                                   cwd=local_dir) == b"true\n"
+        else:
+            clone_exists = False
+    else:
+        clone_exists = os.path.exists(os.path.join(local_dir, '.git'))
 
     if clone_exists and skip_existing:
         return
 
-    initalized = subprocess.call('git ls-remote ' + remote_url,
+    masked_remote_url = mask_password(remote_url)
-                                 stdout=FNULL,
+
-                                 stderr=FNULL,
+    initialized = subprocess.call('git ls-remote ' + remote_url,
-                                 shell=True)
+                                  stdout=FNULL,
-    if initalized == 128:
+                                  stderr=FNULL,
-        log_info("Skipping {0} ({1}) since it's not initalized".format(name, remote_url))
+                                  shell=True)
+    if initialized == 128:
+        log_info("Skipping {0} ({1}) since it's not initialized".format(
+            name, masked_remote_url))
         return
 
     if clone_exists:
         log_info('Updating {0} in {1}'.format(name, local_dir))
-        git_command = ['git', 'remote', 'rm', 'origin']
+
-        logging_subprocess(git_command, None, cwd=local_dir)
+        remotes = subprocess.check_output(['git', 'remote', 'show'],
-        git_command = ['git', 'remote', 'add', 'origin', remote_url]
+                                          cwd=local_dir)
-        logging_subprocess(git_command, None, cwd=local_dir)
+        remotes = [i.strip() for i in remotes.decode('utf-8').splitlines()]
-        git_command = ['git', 'fetch', '--all', '--tags', '--prune']
+
+        if 'origin' not in remotes:
+            git_command = ['git', 'remote', 'rm', 'origin']
+            logging_subprocess(git_command, None, cwd=local_dir)
+            git_command = ['git', 'remote', 'add', 'origin', remote_url]
+            logging_subprocess(git_command, None, cwd=local_dir)
+        else:
+            git_command = ['git', 'remote', 'set-url', 'origin', remote_url]
+            logging_subprocess(git_command, None, cwd=local_dir)
+
+        if lfs_clone:
+            git_command = ['git', 'lfs', 'fetch', '--all', '--force', '--tags', '--prune']
+        else:
+            git_command = ['git', 'fetch', '--all', '--force', '--tags', '--prune']
         logging_subprocess(git_command, None, cwd=local_dir)
     else:
-        log_info('Cloning {0} repository from {1} to {2}'.format(name,
+        log_info('Cloning {0} repository from {1} to {2}'.format(
-                                                                 remote_url,
+            name,
-                                                                 local_dir))
+            masked_remote_url,
-        git_command = ['git', 'clone', remote_url, local_dir]
+            local_dir))
+        if bare_clone:
+            if lfs_clone:
+                git_command = ['git', 'lfs', 'clone', '--mirror', remote_url, local_dir]
+            else:
+                git_command = ['git', 'clone', '--mirror', remote_url, local_dir]
+        else:
+            if lfs_clone:
+                git_command = ['git', 'lfs', 'clone', remote_url, local_dir]
+            else:
+                git_command = ['git', 'clone', remote_url, local_dir]
         logging_subprocess(git_command, None)
 
 
@@ -635,21 +1036,37 @@ def backup_account(args, output_directory):
     account_cwd = os.path.join(output_directory, 'account')
 
     if args.include_starred or args.include_everything:
-        output_file = '{0}/starred.json'.format(account_cwd)
+        output_file = "{0}/starred.json".format(account_cwd)
-        template = "https://{0}/users/{1}/starred"
+        template = "https://{0}/users/{1}/starred".format(get_github_api_host(args), args.user)
-        template = template.format(get_github_api_host(args), args.user)
         _backup_data(args,
-                     'starred repositories',
+                     "starred repositories",
                      template,
                      output_file,
                      account_cwd)
 
     if args.include_watched or args.include_everything:
-        output_file = '{0}/watched.json'.format(account_cwd)
+        output_file = "{0}/watched.json".format(account_cwd)
-        template = "https://{0}/users/{1}/subscriptions"
+        template = "https://{0}/users/{1}/subscriptions".format(get_github_api_host(args), args.user)
-        template = template.format(get_github_api_host(args), args.user)
         _backup_data(args,
-                     'watched repositories',
+                     "watched repositories",
+                     template,
+                     output_file,
+                     account_cwd)
+
+    if args.include_followers or args.include_everything:
+        output_file = "{0}/followers.json".format(account_cwd)
+        template = "https://{0}/users/{1}/followers".format(get_github_api_host(args), args.user)
+        _backup_data(args,
+                     "followers",
+                     template,
+                     output_file,
+                     account_cwd)
+
+    if args.include_following or args.include_everything:
+        output_file = "{0}/following.json".format(account_cwd)
+        template = "https://{0}/users/{1}/following".format(get_github_api_host(args), args.user)
+        _backup_data(args,
+                     "following",
                      template,
                      output_file,
                      account_cwd)
@@ -684,9 +1101,13 @@ def main():
         log_info('Create output directory {0}'.format(output_directory))
         mkdir_p(output_directory)
 
+    if args.lfs_clone:
+        check_git_lfs_install()
+
     log_info('Backing up user {0} to {1}'.format(args.user, output_directory))
 
-    repositories = retrieve_repositories(args)
+    authenticated_user = get_authenticated_user(args)
+    repositories = retrieve_repositories(args, authenticated_user)
     repositories = filter_repositories(args, repositories)
     backup_repositories(args, output_directory, repositories)
     backup_account(args, output_directory)

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = '0.6.0'
+__version__ = '0.25.0'

release

@@ -1,8 +1,13 @@
 #!/usr/bin/env bash
 set -eo pipefail; [[ $RELEASE_TRACE ]] && set -x
 
-PACKAGE_NAME='github-backup'
+if [[ ! -f setup.py ]]; then
-INIT_PACKAGE_NAME='github_backup'
+  echo -e "${RED}WARNING: Missing setup.py${COLOR_OFF}\n"
+  exit 1
+fi
+
+PACKAGE_NAME="$(cat setup.py | grep "name='" | head | cut -d "'" -f2)"
+INIT_PACKAGE_NAME="$(echo "${PACKAGE_NAME//-/_}")"
 PUBLIC="true"
 
 # Colors
@@ -34,7 +39,7 @@ fi
 echo -e "\n${GREEN}STARTING RELEASE PROCESS${COLOR_OFF}\n"
 
 set +e;
-git status | grep "working directory clean" &> /dev/null
+git status | grep -Eo "working (directory|tree) clean" &> /dev/null
 if [ ! $? -eq 0 ]; then # working directory is NOT clean
     echo -e "${RED}WARNING: You have uncomitted changes, you may have forgotten something${COLOR_OFF}\n"
     exit 1

setup.py

@@ -37,8 +37,9 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Topic :: System :: Archiving :: Backup',
         'License :: OSI Approved :: MIT License',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
     ],
     description='backup a github user or organization',
     long_description=open_file('README.rst').read(),