Release version 0.16.0

Clone starred repos

CHANGES.rst

@@ -1,6 +1,71 @@
 Changelog
 =========
 
+0.16.0 (2018-01-22)
+-------------------
+
+- Change option to --all-starred. [W. Harrison Wright]
+
+- JK don't update documentation. [W. Harrison Wright]
+
+- Put starred clone repoistories under a new option. [W. Harrison
+  Wright]
+
+- Add comment. [W. Harrison Wright]
+
+- Add ability to clone starred repos. [W. Harrison Wright]
+
+0.14.1 (2017-10-11)
+-------------------
+
+- Fix arg not defined error. [Edward Pfremmer]
+
+  Ref: https://github.com/josegonzalez/python-github-backup/issues/69
+
+0.14.0 (2017-10-11)
+-------------------
+
+- Added a check to see if git-lfs is installed when doing an LFS clone.
+  [pieterclaerhout]
+
+- Added support for LFS clones. [pieterclaerhout]
+
+- Add pypi info to readme. [Albert Wang]
+
+- Explicitly support python 3 in package description. [Albert Wang]
+
+- Add couple examples to help new users. [Yusuf Tran]
+
+0.13.2 (2017-05-06)
+-------------------
+
+- Fix remotes while updating repository. [Dima Gerasimov]
+
+0.13.1 (2017-04-11)
+-------------------
+
+- Fix error when repository has no updated_at value. [Nicolai Ehemann]
+
+0.13.0 (2017-04-05)
+-------------------
+
+- Add OS check for OSX specific keychain args. [Martin O'Reilly]
+
+  Keychain arguments are only supported on Mac OSX.
+  Added check for operating system so we give a
+  "Keychain arguments are only supported on Mac OSX"
+  error message rather than a "No password item matching the
+  provided name and account could be found in the osx keychain"
+  error message
+
+
+- Add support for storing PAT in OSX keychain. [Martin O'Reilly]
+
+  Added additional optional arguments and README guidance for storing
+  and accessing a Github personal access token (PAT) in the OSX
+  keychain
+
+
 0.12.1 (2017-03-27)
 -------------------
 

README.rst

@@ -2,6 +2,8 @@
 github-backup
 =============
 
+|PyPI| |Python Versions|
+
 backup a github user or organization
 
 Requirements
@@ -29,11 +31,13 @@ CLI Usage is as follows::
                   [-o OUTPUT_DIRECTORY] [-i] [--starred] [--watched]
                   [--all] [--issues] [--issue-comments] [--issue-events]
                   [--pulls] [--pull-comments] [--pull-commits] [--labels]
-                  [--hooks] [--milestones] [--repositories] [--bare]
-                  [--wikis] [--skip-existing]
+                  [--hooks] [--milestones] [--repositories] [--bare] [--lfs]
+                  [--wikis] [--skip-existing] [--all-starred]
                   [-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
                   [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
                   [--prefer-ssh] [-v]
+                  [--keychain-name OSX_KEYCHAIN_ITEM_NAME]
+                  [--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT]
                   USER
 
     Backup a github account
@@ -53,7 +57,7 @@ CLI Usage is as follows::
       -o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                             directory at which to backup the repositories
       -i, --incremental     incremental backup
-      --starred             include starred repositories in backup
+      --starred             include JSON output of starred repositories in backup
       --watched             include watched repositories in backup
       --all                 include everything in backup
       --issues              include issues in backup
@@ -68,8 +72,10 @@ CLI Usage is as follows::
       --milestones          include milestones in backup
       --repositories        include repository clone in backup
       --bare                clone bare repositories
+      --lfs                 clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com)
       --wikis               include wiki clone in backup
       --skip-existing       skip project if a backup directory exists
+      --all-starred         include starred repositories in backup
       -L [LANGUAGES [LANGUAGES ...]], --languages [LANGUAGES [LANGUAGES ...]]
                             only allow these languages
       -N NAME_REGEX, --name-regex NAME_REGEX
@@ -83,6 +89,12 @@ CLI Usage is as follows::
       -F, --fork            include forked repositories
       --prefer-ssh          Clone repositories using SSH instead of HTTPS
       -v, --version         show program's version number and exit
+      --keychain-name OSX_KEYCHAIN_ITEM_NAME
+                            OSX ONLY: name field of password item in OSX keychain
+                            that holds the personal access or OAuth token
+      --keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT
+                            OSX ONLY: account field of password item in OSX
+                            keychain that holds the personal access or OAuth token
 
 
 The package can be used to backup an *entire* organization or repository, including issues and wikis in the most appropriate format (clones for wikis, json files for issues).
@@ -91,3 +103,46 @@ Authentication
 ==============
 
 Note: Password-based authentication will fail if you have two-factor authentication enabled.
+
+Using the Keychain on Mac OSX
+=============================
+Note: On Mac OSX the token can be stored securely in the user's keychain. To do this:
+
+1. Open Keychain from "Applications -> Utilities -> Keychain Access"
+2. Add a new password item using "File -> New Password Item"
+3. Enter a name in the "Keychain Item Name" box. You must provide this name to github-backup using the --keychain-name argument.
+4. Enter an account name in the "Account Name" box, enter your Github username as set above. You must provide this name to github-backup using the --keychain-account argument.
+5. Enter your Github personal access token in the "Password" box
+
+Note:  When you run github-backup, you will be asked whether you want to allow "security" to use your confidential information stored in your keychain. You have two options:
+
+1. **Allow:** In this case you will need to click "Allow" each time you run `github-backup`
+2. **Always Allow:** In this case, you will not be asked for permission when you run `github-backup` in future. This is less secure, but is required if you want to schedule `github-backup` to run automatically
+
+About Git LFS
+=============
+
+When you use the "--lfs" option, you will need to make sure you have Git LFS installed.
+
+Instructions on how to do this can be found on https://git-lfs.github.com.
+
+Examples
+========
+
+Backup all repositories::
+
+    export ACCESS_TOKEN=SOME-GITHUB-TOKEN
+    github-backup WhiteHouse --token $ACCESS_TOKEN --organization --output-directory /tmp/white-house --repositories
+
+Backup a single organization repository with everything else (wiki, pull requests, comments, issues etc)::
+
+    export ACCESS_TOKEN=SOME-GITHUB-TOKEN
+    ORGANIZATION=docker
+    REPO=cli
+    # e.g. git@github.com:docker/cli.git
+    github-backup $ORGANIZATION -P -t $ACCESS_TOKEN -o . --all -O -R $REPO
+
+.. |PyPI| image:: https://img.shields.io/pypi/v/github-backup.svg
+   :target: https://pypi.python.org/pypi/github-backup/
+.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/github-backup.svg
+   :target: https://github.com/albertyw/github-backup

bin/github-backup

@@ -16,6 +16,7 @@ import select
 import subprocess
 import sys
 import time
+import platform
 try:
     # python 3
     from urllib.parse import urlparse
@@ -68,11 +69,15 @@ def logging_subprocess(popenargs,
     """
     child = subprocess.Popen(popenargs, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE, **kwargs)
+    if sys.platform == 'win32':
+        log_info("Windows operating system detected - no subprocess logging will be returned")
 
     log_level = {child.stdout: stdout_log_level,
                  child.stderr: stderr_log_level}
 
     def check_io():
+        if sys.platform == 'win32':
+            return
         ready_to_read = select.select([child.stdout, child.stderr],
                                       [],
                                       [],
@@ -154,6 +159,10 @@ def parse_args():
     parser.add_argument('--starred',
                         action='store_true',
                         dest='include_starred',
+                        help='include JSON output of starred repositories in backup')
+    parser.add_argument('--all-starred',
+                        action='store_true',
+                        dest='all_starred',
                         help='include starred repositories in backup')
     parser.add_argument('--watched',
                         action='store_true',
@@ -207,6 +216,10 @@ def parse_args():
                         action='store_true',
                         dest='bare_clone',
                         help='clone bare repositories')
+    parser.add_argument('--lfs',
+                        action='store_true',
+                        dest='lfs_clone',
+                        help='clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com)')
     parser.add_argument('--wikis',
                         action='store_true',
                         dest='include_wiki',
@@ -251,13 +264,37 @@ def parse_args():
     parser.add_argument('-v', '--version',
                         action='version',
                         version='%(prog)s ' + __version__)
+    parser.add_argument('--keychain-name',
+                        dest='osx_keychain_item_name',
+                        help='OSX ONLY: name field of password item in OSX keychain that holds the personal access or OAuth token')
+    parser.add_argument('--keychain-account',
+                        dest='osx_keychain_item_account',
+                        help='OSX ONLY: account field of password item in OSX keychain that holds the personal access or OAuth token')
     return parser.parse_args()
 
 
 def get_auth(args, encode=True):
     auth = None
 
-    if args.token:
+    if args.osx_keychain_item_name:
+        if not args.osx_keychain_item_account:
+            log_error('You must specify both name and account fields for osx keychain password items')
+        else:
+            if platform.system() != 'Darwin':
+                log_error("Keychain arguments are only supported on Mac OSX")
+            try:
+                with open(os.devnull,'w') as devnull:
+                    token = (subprocess.check_output([
+                        'security','find-generic-password',
+                        '-s',args.osx_keychain_item_name,
+                        '-a',args.osx_keychain_item_account,
+                        '-w' ], stderr=devnull).strip())
+                auth = token + ':' + 'x-oauth-basic'
+            except:
+                log_error('No password item matching the provided name and account could be found in the osx keychain.')
+    elif args.osx_keychain_item_account:
+        log_error('You must specify both name and account fields for osx keychain password items')
+    elif args.token:
         _path_specifier = 'file://'
         if args.token.startswith(_path_specifier):
             args.token = open(args.token[len(_path_specifier):],
@@ -310,7 +347,7 @@ def get_github_repo_url(args, repository):
         repo_url = 'https://{0}@{1}/{2}/{3}.git'.format(
             auth,
             get_github_host(args),
-            args.user,
+            repository['owner']['login'],
             repository['name'])
     else:
         repo_url = repository['clone_url']
@@ -442,6 +479,13 @@ def _request_url_error(template, retry_timeout):
     return False
 
 
+def check_git_lfs_install():
+    exit_code = subprocess.call(['git', 'lfs', 'version'])
+    if exit_code != 0:
+        log_error('The argument --lfs requires you to have Git LFS installed.\nYou can get it from https://git-lfs.github.com.')
+        sys.exit(1)
+
+
 def retrieve_repositories(args):
     log_info('Retrieving repositories')
     single_request = False
@@ -459,7 +503,19 @@ def retrieve_repositories(args):
             args.user,
             args.repository)
 
-    return retrieve_data(args, template, single_request=single_request)
+    repos = retrieve_data(args, template, single_request=single_request)
+
+    if args.all_starred:
+        starred_template = 'https://{0}/user/starred'.format(
+            get_github_api_host(args))
+        starred_repos = retrieve_data(args, starred_template, single_request=False)
+        # we need to be able to determine this repo was retrieved as a starred repo
+        # later, so add a flag to each item
+        for item in starred_repos:
+            item.update({'is_starred': True})
+        repos.extend(starred_repos)
+
+    return repos
 
 
 def filter_repositories(args, unfiltered_repositories):
@@ -467,7 +523,7 @@ def filter_repositories(args, unfiltered_repositories):
 
     repositories = []
     for r in unfiltered_repositories:
-        if r['owner']['login'] == args.user:
+        if r['owner']['login'] == args.user or r.get('is_starred'):
             repositories.append(r)
 
     name_regex = None
@@ -495,7 +551,7 @@ def backup_repositories(args, output_directory, repositories):
     repos_template = 'https://{0}/repos'.format(get_github_api_host(args))
 
     if args.incremental:
-        last_update = max(repository['updated_at'] for repository in repositories)  # noqa
+        last_update = max(list(repository['updated_at'] for repository in repositories) or [time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime())])  # noqa
         last_update_path = os.path.join(output_directory, 'last_update')
         if os.path.exists(last_update_path):
             args.since = open(last_update_path).read().strip()
@@ -507,6 +563,14 @@ def backup_repositories(args, output_directory, repositories):
     for repository in repositories:
         backup_cwd = os.path.join(output_directory, 'repositories')
         repo_cwd = os.path.join(backup_cwd, repository['name'])
+
+        # put starred repos in -o/starred/${owner}/${repo} to prevent collision of
+        # any repositories with the same name
+        if repository.get('is_starred'):
+            backup_cwd = os.path.join(output_directory, 'starred')
+            repo_cwd = os.path.join(backup_cwd, repository['owner']['login'],
+                                    repository['name'])
+
         repo_dir = os.path.join(repo_cwd, 'repository')
         repo_url = get_github_repo_url(args, repository)
 
@@ -515,7 +579,8 @@ def backup_repositories(args, output_directory, repositories):
                              repo_url,
                              repo_dir,
                              skip_existing=args.skip_existing,
-                             bare_clone=args.bare_clone)
+                             bare_clone=args.bare_clone,
+                             lfs_clone=args.lfs_clone)
 
         download_wiki = (args.include_wiki or args.include_everything)
         if repository['has_wiki'] and download_wiki:
@@ -523,7 +588,8 @@ def backup_repositories(args, output_directory, repositories):
                              repo_url.replace('.git', '.wiki.git'),
                              os.path.join(repo_cwd, 'wiki'),
                              skip_existing=args.skip_existing,
-                             bare_clone=args.bare_clone)
+                             bare_clone=args.bare_clone,
+                             lfs_clone=args.lfs_clone)
 
         if args.include_issues or args.include_everything:
             backup_issues(args, repo_cwd, repository, repos_template)
@@ -713,7 +779,8 @@ def fetch_repository(name,
                      remote_url,
                      local_dir,
                      skip_existing=False,
-                     bare_clone=False):
+                     bare_clone=False,
+                     lfs_clone=False):
     if bare_clone:
         if os.path.exists(local_dir):
             clone_exists = subprocess.check_output(['git',
@@ -744,7 +811,7 @@ def fetch_repository(name,
 
         remotes = subprocess.check_output(['git', 'remote', 'show'],
                                           cwd=local_dir)
-        remotes = [i.strip() for i in remotes.decode('utf-8')]
+        remotes = [i.strip() for i in remotes.decode('utf-8').splitlines()]
 
         if 'origin' not in remotes:
             git_command = ['git', 'remote', 'rm', 'origin']
@@ -755,6 +822,9 @@ def fetch_repository(name,
             git_command = ['git', 'remote', 'set-url', 'origin', remote_url]
             logging_subprocess(git_command, None, cwd=local_dir)
 
+        if lfs_clone:
+            git_command = ['git', 'lfs', 'fetch', '--all', '--force', '--tags', '--prune']
+        else:
             git_command = ['git', 'fetch', '--all', '--force', '--tags', '--prune']
         logging_subprocess(git_command, None, cwd=local_dir)
     else:
@@ -763,7 +833,13 @@ def fetch_repository(name,
             masked_remote_url,
             local_dir))
         if bare_clone:
+            if lfs_clone:
+                git_command = ['git', 'lfs', 'clone', '--mirror', remote_url, local_dir]
+            else:
                 git_command = ['git', 'clone', '--mirror', remote_url, local_dir]
+        else:
+            if lfs_clone:
+                git_command = ['git', 'lfs', 'clone', remote_url, local_dir]
             else:
                 git_command = ['git', 'clone', remote_url, local_dir]
         logging_subprocess(git_command, None)
@@ -822,6 +898,9 @@ def main():
         log_info('Create output directory {0}'.format(output_directory))
         mkdir_p(output_directory)
 
+    if args.lfs_clone:
+        check_git_lfs_install()
+
     log_info('Backing up user {0} to {1}'.format(args.user, output_directory))
 
     repositories = retrieve_repositories(args)

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = '0.12.1'
+__version__ = '0.16.0'

setup.py

@@ -39,6 +39,8 @@ setup(
         'License :: OSI Approved :: MIT License',
         'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
     ],
     description='backup a github user or organization',
     long_description=open_file('README.rst').read(),