Compare commits

..

57 Commits

Author SHA1 Message Date
Jose Diaz-Gonzalez
38bf438d2f Release version 0.21.0 2018-11-28 01:59:03 -05:00
Jose Diaz-Gonzalez
899cf42b57 Merge pull request #97 from whwright/94-fix-user-repos
Correctly download repos when user arg != authenticated user
2018-11-28 01:58:37 -05:00
W. Harrison Wright
b5972aaaf0 Correctly download repos when user arg != authenticated user 2018-11-11 19:40:46 -06:00
Jose Diaz-Gonzalez
d860f369e9 Release version 0.20.1 2018-09-29 00:16:48 -04:00
Jose Diaz-Gonzalez
77ab1bda15 Merge pull request #92 from whwright/87-fix-starred-bug
Clone the specified user's starred repos/gists, not the authenticated user
2018-09-29 00:16:32 -04:00
W. Harrison Wright
4a4a317331 Clone the specified user's gists, not the authenticated user 2018-09-28 21:59:50 -05:00
W. Harrison Wright
5a8e1ac275 Clone the specified user's starred repos, not the authenticated user 2018-09-28 21:46:28 -05:00
Jose Diaz-Gonzalez
0de341eab4 Release version 0.20.0 2018-03-24 15:00:26 -04:00
Jose Diaz-Gonzalez
b0130fdf94 chore: drop Python 2.6 2018-03-24 15:00:05 -04:00
Jose Diaz-Gonzalez
b49f399037 feat: simplify release script 2018-03-24 14:59:56 -04:00
Jose Diaz-Gonzalez
321414d352 Release version 0.19.2 2018-03-24 01:16:34 -04:00
Jose Diaz-Gonzalez
413d4381cc fix: cleanup pep8 violations 2018-03-24 01:16:28 -04:00
Jose Diaz-Gonzalez
0110ea40ed Release version 0.19.1 2018-03-24 01:04:35 -04:00
Jose Diaz-Gonzalez
8d2ef2f528 Release version 0.19.0 2018-03-24 00:54:34 -04:00
Jose Diaz-Gonzalez
1a79f755a5 Merge pull request #77 from mayflower/pull-details
Pull Details
2018-03-23 23:40:22 -04:00
Jose Diaz-Gonzalez
abf45d5b54 Merge pull request #84 from johbo/fix-python36-skip-existing
Mark string as binary in comparison for skip_existing
2018-02-26 10:44:12 -05:00
Johannes Bornhold
fd33037b1c Mark string as binary in comparison for skip_existing
Found out that the flag "--skip-existing" did not work out as expected on Python
3.6. Tracked it down to the comparison which has to be against a string of bytes
in Python3.
2018-02-26 11:21:25 +01:00
Jose Diaz-Gonzalez
87dab293ed Release version 0.18.0 2018-02-22 12:13:06 -05:00
Jose Diaz-Gonzalez
0244af4e05 Merge pull request #82 from sgreene570/add-followers
Add option to fetch followers/following JSON data
2018-02-22 12:11:48 -05:00
Stephen Greene
eca9f0f7df Add option to fetch followers/following JSON data 2018-02-21 19:29:59 -08:00
Jose Diaz-Gonzalez
afa2a6d587 Release version 0.17.0 2018-02-20 13:06:48 -05:00
Jose Diaz-Gonzalez
b77ea48d74 Merge pull request #81 from whwright/gists
Add ability to back up gists
2018-02-19 15:29:20 -05:00
W. Harrison Wright
f378254188 Short circuit gists backup process 2018-02-07 21:46:59 -06:00
W. Harrison Wright
83128e986a Formatting 2018-02-07 21:30:55 -06:00
W. Harrison Wright
17e4f9a125 Add ability to backup gists 2018-02-07 21:29:49 -06:00
Jose Diaz-Gonzalez
e59d1e3a68 Release version 0.16.0 2018-01-22 12:49:31 -05:00
Jose Diaz-Gonzalez
de860ee5a9 Merge pull request #78 from whwright/clone-starred-repos
Clone starred repos
2018-01-22 12:36:42 -05:00
Jose Diaz-Gonzalez
cb054c2631 Update README.rst 2018-01-22 12:36:32 -05:00
W. Harrison Wright
c142707a90 Update documentation 2018-01-22 11:34:27 -06:00
W. Harrison Wright
7cccd42ec9 Change option to --all-starred 2018-01-14 10:22:10 -06:00
W. Harrison Wright
9a539b1d6b JK don't update documentation 2018-01-14 10:18:51 -06:00
W. Harrison Wright
cd2372183e Update documentation 2018-01-13 17:44:09 -06:00
W. Harrison Wright
bd346de898 Put starred clone repoistories under a new option 2018-01-13 17:43:00 -06:00
W. Harrison Wright
6e3cbe841a Add comment 2018-01-13 14:12:26 -06:00
W. Harrison Wright
8b95f187ad Add ability to clone starred repos 2018-01-13 14:08:36 -06:00
Robin Gloster
ef88248c41 Add additional output for the current request
This is useful to have some progress indication for huge repositories.
2017-12-29 23:33:53 +01:00
Robin Gloster
0a4decfb3b Add option to backup additional PR details
Some payload is only included when requesting a single pull request
2017-12-29 21:39:59 +01:00
Jose Diaz-Gonzalez
2b9549ffde Release version 0.15.0 2017-12-11 11:46:16 -05:00
Jose Diaz-Gonzalez
fb2c3ca921 Merge pull request #75 from slibby/slibby-patch-windows
update check_io() to allow scripts to run on Windows
2017-12-11 11:45:58 -05:00
Sam Libby
4f4785085d update logging_subprocess function
1. added newline for return
2. added one-time warning (once per subprocess)
2017-12-11 09:25:49 -07:00
Sam Libby
76895dcf69 update check_io() to allow scripts to run on Windows 2017-12-10 21:44:26 -07:00
Jose Diaz-Gonzalez
1d50a4038b Release version 0.14.1 2017-10-11 16:18:21 -04:00
Jose Diaz-Gonzalez
9d31ccfba9 Merge pull request #70 from epfremmer/patch-1
Fix arg not defined error
2017-10-11 16:17:58 -04:00
Edward Pfremmer
27a1ba2d04 Fix arg not defined error
Ref: https://github.com/josegonzalez/python-github-backup/issues/69
2017-10-11 15:12:34 -05:00
Jose Diaz-Gonzalez
f157ea107f Release version 0.14.0 2017-10-11 11:52:16 -04:00
Jose Diaz-Gonzalez
a129cc759a Merge pull request #68 from pieterclaerhout/master
Added support for LFS clones
2017-10-11 11:51:57 -04:00
pieterclaerhout
bb551a83f4 Updated the readme 2017-10-11 15:14:13 +02:00
pieterclaerhout
9b1b4a9ebc Added a check to see if git-lfs is installed when doing an LFS clone 2017-10-11 15:11:14 +02:00
pieterclaerhout
e6b6eb8bef Added support for LFS clones 2017-10-10 19:52:07 +02:00
Jose Diaz-Gonzalez
0b3f120e2b Merge pull request #66 from albertyw/python3
Explicitly support python 3
2017-09-30 21:01:27 -04:00
Albert Wang
990249b80b Add pypi info to readme 2017-09-30 17:16:38 -07:00
Albert Wang
cefb226545 Explicitly support python 3 in package description 2017-09-30 17:13:47 -07:00
Jose Diaz-Gonzalez
ea22ffdf26 Merge pull request #65 from mumblez/master
add couple examples to help new users
2017-05-30 11:58:42 -06:00
Yusuf Tran
0f21d7b8a4 add couple examples to help new users 2017-05-30 18:52:11 +01:00
Jose Diaz-Gonzalez
cb33b9bab7 Release version 0.13.2 2017-05-06 14:14:08 -06:00
Jose Diaz-Gonzalez
68c48cb0b3 Merge pull request #64 from karlicoss/fix-remotes
Fix remotes while updating repository
2017-05-06 14:13:46 -06:00
Dima Gerasimov
922a3c5a6e Fix remotes while updating repository 2017-05-06 12:58:42 +01:00
6 changed files with 355 additions and 57 deletions

View File

@@ -1,6 +1,111 @@
Changelog
=========
0.21.0 (2018-11-28)
-------------------
- Correctly download repos when user arg != authenticated user. [W.
Harrison Wright]
0.20.1 (2018-09-29)
-------------------
- Clone the specified user's gists, not the authenticated user. [W.
Harrison Wright]
- Clone the specified user's starred repos, not the authenticated user.
[W. Harrison Wright]
0.20.0 (2018-03-24)
-------------------
- Chore: drop Python 2.6. [Jose Diaz-Gonzalez]
- Feat: simplify release script. [Jose Diaz-Gonzalez]
0.19.2 (2018-03-24)
-------------------
Fix
~~~
- Cleanup pep8 violations. [Jose Diaz-Gonzalez]
0.19.0 (2018-03-24)
-------------------
- Add additional output for the current request. [Robin Gloster]
This is useful to have some progress indication for huge repositories.
- Add option to backup additional PR details. [Robin Gloster]
Some payload is only included when requesting a single pull request
- Mark string as binary in comparison for skip_existing. [Johannes
Bornhold]
Found out that the flag "--skip-existing" did not work out as expected on Python
3.6. Tracked it down to the comparison which has to be against a string of bytes
in Python3.
0.18.0 (2018-02-22)
-------------------
- Add option to fetch followers/following JSON data. [Stephen Greene]
0.17.0 (2018-02-20)
-------------------
- Short circuit gists backup process. [W. Harrison Wright]
- Formatting. [W. Harrison Wright]
- Add ability to backup gists. [W. Harrison Wright]
0.16.0 (2018-01-22)
-------------------
- Change option to --all-starred. [W. Harrison Wright]
- JK don't update documentation. [W. Harrison Wright]
- Put starred clone repoistories under a new option. [W. Harrison
Wright]
- Add comment. [W. Harrison Wright]
- Add ability to clone starred repos. [W. Harrison Wright]
0.14.1 (2017-10-11)
-------------------
- Fix arg not defined error. [Edward Pfremmer]
Ref: https://github.com/josegonzalez/python-github-backup/issues/69
0.14.0 (2017-10-11)
-------------------
- Added a check to see if git-lfs is installed when doing an LFS clone.
[pieterclaerhout]
- Added support for LFS clones. [pieterclaerhout]
- Add pypi info to readme. [Albert Wang]
- Explicitly support python 3 in package description. [Albert Wang]
- Add couple examples to help new users. [Yusuf Tran]
0.13.2 (2017-05-06)
-------------------
- Fix remotes while updating repository. [Dima Gerasimov]
0.13.1 (2017-04-11)
-------------------

View File

@@ -2,6 +2,8 @@
github-backup
=============
|PyPI| |Python Versions|
backup a github user or organization
Requirements
@@ -26,11 +28,12 @@ Usage
CLI Usage is as follows::
github-backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN]
[-o OUTPUT_DIRECTORY] [-i] [--starred] [--watched]
[--all] [--issues] [--issue-comments] [--issue-events]
[--pulls] [--pull-comments] [--pull-commits] [--labels]
[--hooks] [--milestones] [--repositories] [--bare]
[--wikis] [--skip-existing]
[-o OUTPUT_DIRECTORY] [-i] [--starred] [--all-starred]
[--watched] [--followers] [--following] [--all]
[--issues] [--issue-comments] [--issue-events] [--pulls]
[--pull-comments] [--pull-commits] [--labels] [--hooks]
[--milestones] [--repositories] [--bare] [--lfs]
[--wikis] [--gists] [--starred-gists] [--skip-existing]
[-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
[-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
[--prefer-ssh] [-v]
@@ -51,12 +54,16 @@ CLI Usage is as follows::
password for basic auth. If a username is given but
not a password, the password will be prompted for.
-t TOKEN, --token TOKEN
personal access or OAuth token
personal access or OAuth token, or path to token
(file://...)
-o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
directory at which to backup the repositories
-i, --incremental incremental backup
--starred include starred repositories in backup
--starred include JSON output of starred repositories in backup
--all-starred include starred repositories in backup
--watched include watched repositories in backup
--followers include JSON output of followers in backup
--following include JSON output of following users in backup
--all include everything in backup
--issues include issues in backup
--issue-comments include issue comments in backup
@@ -70,7 +77,11 @@ CLI Usage is as follows::
--milestones include milestones in backup
--repositories include repository clone in backup
--bare clone bare repositories
--lfs clone LFS repositories (requires Git LFS to be
installed, https://git-lfs.github.com)
--wikis include wiki clone in backup
--gists include gists in backup
--starred-gists include starred gists in backup
--skip-existing skip project if a backup directory exists
-L [LANGUAGES [LANGUAGES ...]], --languages [LANGUAGES [LANGUAGES ...]]
only allow these languages
@@ -114,3 +125,31 @@ Note: When you run github-backup, you will be asked whether you want to allow "
1. **Allow:** In this case you will need to click "Allow" each time you run `github-backup`
2. **Always Allow:** In this case, you will not be asked for permission when you run `github-backup` in future. This is less secure, but is required if you want to schedule `github-backup` to run automatically
About Git LFS
=============
When you use the "--lfs" option, you will need to make sure you have Git LFS installed.
Instructions on how to do this can be found on https://git-lfs.github.com.
Examples
========
Backup all repositories::
export ACCESS_TOKEN=SOME-GITHUB-TOKEN
github-backup WhiteHouse --token $ACCESS_TOKEN --organization --output-directory /tmp/white-house --repositories
Backup a single organization repository with everything else (wiki, pull requests, comments, issues etc)::
export ACCESS_TOKEN=SOME-GITHUB-TOKEN
ORGANIZATION=docker
REPO=cli
# e.g. git@github.com:docker/cli.git
github-backup $ORGANIZATION -P -t $ACCESS_TOKEN -o . --all -O -R $REPO
.. |PyPI| image:: https://img.shields.io/pypi/v/github-backup.svg
:target: https://pypi.python.org/pypi/github-backup/
.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/github-backup.svg
:target: https://github.com/albertyw/github-backup

View File

@@ -69,11 +69,15 @@ def logging_subprocess(popenargs,
"""
child = subprocess.Popen(popenargs, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, **kwargs)
if sys.platform == 'win32':
log_info("Windows operating system detected - no subprocess logging will be returned")
log_level = {child.stdout: stdout_log_level,
child.stderr: stderr_log_level}
def check_io():
if sys.platform == 'win32':
return
ready_to_read = select.select([child.stdout, child.stderr],
[],
[],
@@ -155,11 +159,23 @@ def parse_args():
parser.add_argument('--starred',
action='store_true',
dest='include_starred',
help='include JSON output of starred repositories in backup')
parser.add_argument('--all-starred',
action='store_true',
dest='all_starred',
help='include starred repositories in backup')
parser.add_argument('--watched',
action='store_true',
dest='include_watched',
help='include watched repositories in backup')
parser.add_argument('--followers',
action='store_true',
dest='include_followers',
help='include JSON output of followers in backup')
parser.add_argument('--following',
action='store_true',
dest='include_following',
help='include JSON output of following users in backup')
parser.add_argument('--all',
action='store_true',
dest='include_everything',
@@ -188,6 +204,10 @@ def parse_args():
action='store_true',
dest='include_pull_commits',
help='include pull request commits in backup')
parser.add_argument('--pull-details',
action='store_true',
dest='include_pull_details',
help='include more pull request details in backup')
parser.add_argument('--labels',
action='store_true',
dest='include_labels',
@@ -208,10 +228,22 @@ def parse_args():
action='store_true',
dest='bare_clone',
help='clone bare repositories')
parser.add_argument('--lfs',
action='store_true',
dest='lfs_clone',
help='clone LFS repositories (requires Git LFS to be installed, https://git-lfs.github.com)')
parser.add_argument('--wikis',
action='store_true',
dest='include_wiki',
help='include wiki clone in backup')
parser.add_argument('--gists',
action='store_true',
dest='include_gists',
help='include gists in backup')
parser.add_argument('--starred-gists',
action='store_true',
dest='include_starred_gists',
help='include starred gists in backup')
parser.add_argument('--skip-existing',
action='store_true',
dest='skip_existing',
@@ -271,12 +303,12 @@ def get_auth(args, encode=True):
if platform.system() != 'Darwin':
log_error("Keychain arguments are only supported on Mac OSX")
try:
with open(os.devnull,'w') as devnull:
with open(os.devnull, 'w') as devnull:
token = (subprocess.check_output([
'security','find-generic-password',
'-s',args.osx_keychain_item_name,
'-a',args.osx_keychain_item_account,
'-w' ], stderr=devnull).strip())
'security', 'find-generic-password',
'-s', args.osx_keychain_item_name,
'-a', args.osx_keychain_item_account,
'-w'], stderr=devnull).strip())
auth = token + ':' + 'x-oauth-basic'
except:
log_error('No password item matching the provided name and account could be found in the osx keychain.')
@@ -330,12 +362,15 @@ def get_github_repo_url(args, repository):
if args.prefer_ssh:
return repository['ssh_url']
if repository.get('is_gist'):
return repository['git_pull_url']
auth = get_auth(args, False)
if auth:
repo_url = 'https://{0}@{1}/{2}/{3}.git'.format(
auth,
get_github_host(args),
args.user,
repository['owner']['login'],
repository['name'])
else:
repo_url = repository['clone_url']
@@ -419,6 +454,7 @@ def _construct_request(per_page, page, query_args, template, auth):
request = Request(template + '?' + querystring)
if auth is not None:
request.add_header('Authorization', 'Basic '.encode('ascii') + auth)
log_info('Requesting {}?{}'.format(template, querystring))
return request
@@ -467,11 +503,33 @@ def _request_url_error(template, retry_timeout):
return False
def retrieve_repositories(args):
def get_authenticated_user(args):
template = 'https://{0}/user'.format(get_github_api_host(args))
data = retrieve_data(args, template, single_request=True)
return data[0]
def check_git_lfs_install():
exit_code = subprocess.call(['git', 'lfs', 'version'])
if exit_code != 0:
log_error('The argument --lfs requires you to have Git LFS installed.\nYou can get it from https://git-lfs.github.com.')
sys.exit(1)
def retrieve_repositories(args, authenticated_user):
log_info('Retrieving repositories')
single_request = False
template = 'https://{0}/user/repos'.format(
get_github_api_host(args))
if args.user == authenticated_user['login']:
# we must use the /user/repos API to be able to access private repos
template = 'https://{0}/user/repos'.format(
get_github_api_host(args))
else:
if args.private:
log_error('Authenticated user is different from user being backed up, thus private repositories cannot be accessed')
template = 'https://{0}/users/{1}/repos'.format(
get_github_api_host(args),
args.user)
if args.organization:
template = 'https://{0}/orgs/{1}/repos'.format(
get_github_api_host(args),
@@ -484,7 +542,34 @@ def retrieve_repositories(args):
args.user,
args.repository)
return retrieve_data(args, template, single_request=single_request)
repos = retrieve_data(args, template, single_request=single_request)
if args.all_starred:
starred_template = 'https://{0}/users/{1}/starred'.format(get_github_api_host(args), args.user)
starred_repos = retrieve_data(args, starred_template, single_request=False)
# flag each repo as starred for downstream processing
for item in starred_repos:
item.update({'is_starred': True})
repos.extend(starred_repos)
if args.include_gists:
gists_template = 'https://{0}/users/{1}/gists'.format(get_github_api_host(args), args.user)
gists = retrieve_data(args, gists_template, single_request=False)
# flag each repo as a gist for downstream processing
for item in gists:
item.update({'is_gist': True})
repos.extend(gists)
if args.include_starred_gists:
starred_gists_template = 'https://{0}/gists/starred'.format(get_github_api_host(args))
starred_gists = retrieve_data(args, starred_gists_template, single_request=False)
# flag each repo as a starred gist for downstream processing
for item in starred_gists:
item.update({'is_gist': True,
'is_starred': True})
repos.extend(starred_gists)
return repos
def filter_repositories(args, unfiltered_repositories):
@@ -492,7 +577,8 @@ def filter_repositories(args, unfiltered_repositories):
repositories = []
for r in unfiltered_repositories:
if r['owner']['login'] == args.user:
# gists can be anonymous, so need to safely check owner
if r.get('owner', {}).get('login') == args.user or r.get('is_starred'):
repositories.append(r)
name_regex = None
@@ -504,11 +590,11 @@ def filter_repositories(args, unfiltered_repositories):
languages = [x.lower() for x in args.languages]
if not args.fork:
repositories = [r for r in repositories if not r['fork']]
repositories = [r for r in repositories if not r.get('fork')]
if not args.private:
repositories = [r for r in repositories if not r['private']]
repositories = [r for r in repositories if not r.get('private') or r.get('public')]
if languages:
repositories = [r for r in repositories if r['language'] and r['language'].lower() in languages] # noqa
repositories = [r for r in repositories if r.get('language') and r.get('language').lower() in languages] # noqa
if name_regex:
repositories = [r for r in repositories if name_regex.match(r['name'])]
@@ -530,17 +616,36 @@ def backup_repositories(args, output_directory, repositories):
args.since = None
for repository in repositories:
backup_cwd = os.path.join(output_directory, 'repositories')
repo_cwd = os.path.join(backup_cwd, repository['name'])
if repository.get('is_gist'):
repo_cwd = os.path.join(output_directory, 'gists', repository['id'])
elif repository.get('is_starred'):
# put starred repos in -o/starred/${owner}/${repo} to prevent collision of
# any repositories with the same name
repo_cwd = os.path.join(output_directory, 'starred', repository['owner']['login'], repository['name'])
else:
repo_cwd = os.path.join(output_directory, 'repositories', repository['name'])
repo_dir = os.path.join(repo_cwd, 'repository')
repo_url = get_github_repo_url(args, repository)
if args.include_repository or args.include_everything:
fetch_repository(repository['name'],
include_gists = (args.include_gists or args.include_starred_gists)
if (args.include_repository or args.include_everything) \
or (include_gists and repository.get('is_gist')):
repo_name = repository.get('name') if not repository.get('is_gist') else repository.get('id')
fetch_repository(repo_name,
repo_url,
repo_dir,
skip_existing=args.skip_existing,
bare_clone=args.bare_clone)
bare_clone=args.bare_clone,
lfs_clone=args.lfs_clone)
if repository.get('is_gist'):
# dump gist information to a file as well
output_file = '{0}/gist.json'.format(repo_cwd)
with codecs.open(output_file, 'w', encoding='utf-8') as f:
json_dump(repository, f)
continue # don't try to back anything else for a gist; it doesn't exist
download_wiki = (args.include_wiki or args.include_everything)
if repository['has_wiki'] and download_wiki:
@@ -548,7 +653,8 @@ def backup_repositories(args, output_directory, repositories):
repo_url.replace('.git', '.wiki.git'),
os.path.join(repo_cwd, 'wiki'),
skip_existing=args.skip_existing,
bare_clone=args.bare_clone)
bare_clone=args.bare_clone,
lfs_clone=args.lfs_clone)
if args.include_issues or args.include_everything:
backup_issues(args, repo_cwd, repository, repos_template)
@@ -639,23 +745,35 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
pulls = {}
_pulls_template = '{0}/{1}/pulls'.format(repos_template,
repository['full_name'])
query_args = {
'filter': 'all',
'state': 'all',
'sort': 'updated',
'direction': 'desc',
}
pull_states = ['open', 'closed']
for pull_state in pull_states:
query_args = {
'filter': 'all',
'state': pull_state,
'sort': 'updated',
'direction': 'desc',
}
# It'd be nice to be able to apply the args.since filter here...
if not args.include_pull_details:
pull_states = ['open', 'closed']
for pull_state in pull_states:
query_args['state'] = pull_state
# It'd be nice to be able to apply the args.since filter here...
_pulls = retrieve_data(args,
_pulls_template,
query_args=query_args)
for pull in _pulls:
if not args.since or pull['updated_at'] >= args.since:
pulls[pull['number']] = pull
else:
_pulls = retrieve_data(args,
_pulls_template,
query_args=query_args)
for pull in _pulls:
if not args.since or pull['updated_at'] >= args.since:
pulls[pull['number']] = pull
pulls[pull['number']] = retrieve_data(
args,
_pulls_template + '/{}'.format(pull['number']),
single_request=True
)
log_info('Saving {0} pull requests to disk'.format(
len(list(pulls.keys()))))
@@ -738,13 +856,14 @@ def fetch_repository(name,
remote_url,
local_dir,
skip_existing=False,
bare_clone=False):
bare_clone=False,
lfs_clone=False):
if bare_clone:
if os.path.exists(local_dir):
clone_exists = subprocess.check_output(['git',
'rev-parse',
'--is-bare-repository'],
cwd=local_dir) == "true\n"
cwd=local_dir) == b"true\n"
else:
clone_exists = False
else:
@@ -769,7 +888,7 @@ def fetch_repository(name,
remotes = subprocess.check_output(['git', 'remote', 'show'],
cwd=local_dir)
remotes = [i.strip() for i in remotes.decode('utf-8')]
remotes = [i.strip() for i in remotes.decode('utf-8').splitlines()]
if 'origin' not in remotes:
git_command = ['git', 'remote', 'rm', 'origin']
@@ -780,7 +899,10 @@ def fetch_repository(name,
git_command = ['git', 'remote', 'set-url', 'origin', remote_url]
logging_subprocess(git_command, None, cwd=local_dir)
git_command = ['git', 'fetch', '--all', '--force', '--tags', '--prune']
if lfs_clone:
git_command = ['git', 'lfs', 'fetch', '--all', '--force', '--tags', '--prune']
else:
git_command = ['git', 'fetch', '--all', '--force', '--tags', '--prune']
logging_subprocess(git_command, None, cwd=local_dir)
else:
log_info('Cloning {0} repository from {1} to {2}'.format(
@@ -788,9 +910,15 @@ def fetch_repository(name,
masked_remote_url,
local_dir))
if bare_clone:
git_command = ['git', 'clone', '--mirror', remote_url, local_dir]
if lfs_clone:
git_command = ['git', 'lfs', 'clone', '--mirror', remote_url, local_dir]
else:
git_command = ['git', 'clone', '--mirror', remote_url, local_dir]
else:
git_command = ['git', 'clone', remote_url, local_dir]
if lfs_clone:
git_command = ['git', 'lfs', 'clone', remote_url, local_dir]
else:
git_command = ['git', 'clone', remote_url, local_dir]
logging_subprocess(git_command, None)
@@ -798,21 +926,37 @@ def backup_account(args, output_directory):
account_cwd = os.path.join(output_directory, 'account')
if args.include_starred or args.include_everything:
output_file = '{0}/starred.json'.format(account_cwd)
template = "https://{0}/users/{1}/starred"
template = template.format(get_github_api_host(args), args.user)
output_file = "{0}/starred.json".format(account_cwd)
template = "https://{0}/users/{1}/starred".format(get_github_api_host(args), args.user)
_backup_data(args,
'starred repositories',
"starred repositories",
template,
output_file,
account_cwd)
if args.include_watched or args.include_everything:
output_file = '{0}/watched.json'.format(account_cwd)
template = "https://{0}/users/{1}/subscriptions"
template = template.format(get_github_api_host(args), args.user)
output_file = "{0}/watched.json".format(account_cwd)
template = "https://{0}/users/{1}/subscriptions".format(get_github_api_host(args), args.user)
_backup_data(args,
'watched repositories',
"watched repositories",
template,
output_file,
account_cwd)
if args.include_followers or args.include_everything:
output_file = "{0}/followers.json".format(account_cwd)
template = "https://{0}/users/{1}/followers".format(get_github_api_host(args), args.user)
_backup_data(args,
"followers",
template,
output_file,
account_cwd)
if args.include_following or args.include_everything:
output_file = "{0}/following.json".format(account_cwd)
template = "https://{0}/users/{1}/following".format(get_github_api_host(args), args.user)
_backup_data(args,
"following",
template,
output_file,
account_cwd)
@@ -847,9 +991,13 @@ def main():
log_info('Create output directory {0}'.format(output_directory))
mkdir_p(output_directory)
if args.lfs_clone:
check_git_lfs_install()
log_info('Backing up user {0} to {1}'.format(args.user, output_directory))
repositories = retrieve_repositories(args)
authenticated_user = get_authenticated_user(args)
repositories = retrieve_repositories(args, authenticated_user)
repositories = filter_repositories(args, repositories)
backup_repositories(args, output_directory, repositories)
backup_account(args, output_directory)

View File

@@ -1 +1 @@
__version__ = '0.13.1'
__version__ = '0.21.0'

View File

@@ -1,8 +1,13 @@
#!/usr/bin/env bash
set -eo pipefail; [[ $RELEASE_TRACE ]] && set -x
PACKAGE_NAME='github-backup'
INIT_PACKAGE_NAME='github_backup'
if [[ ! -f setup.py ]]; then
echo -e "${RED}WARNING: Missing setup.py${COLOR_OFF}\n"
exit 1
fi
PACKAGE_NAME="$(cat setup.py | grep "name='" | head | cut -d "'" -f2)"
INIT_PACKAGE_NAME="$(echo "${PACKAGE_NAME//-/_}")"
PUBLIC="true"
# Colors

View File

@@ -37,8 +37,9 @@ setup(
'Development Status :: 5 - Production/Stable',
'Topic :: System :: Archiving :: Backup',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
description='backup a github user or organization',
long_description=open_file('README.rst').read(),