Release version 0.33.0

Add basic API request throttling

.gitignore

@@ -25,3 +25,12 @@ doc/_build
 
 # Generated man page
 doc/aws_hostname.1
+
+# Annoying macOS files
+.DS_Store
+._*
+
+# IDE configuration files
+.vscode
+.atom
+

CHANGES.rst

@@ -1,9 +1,84 @@
 Changelog
 =========
 
-0.26.0 (2019-09-23)
+0.33.0 (2020-04-13)
 -------------------
 ------------------------
+- Add basic API request throttling. [Enrico Tröger]
+
+  A simple approach to throttle API requests and so keep within the rate
+  limits of the API. Can be enabled with "--throttle-limit" to specify
+  when throttling should start.
+  "--throttle-pause" defines the time to sleep between further API
+  requests.
+
+
+0.32.0 (2020-04-13)
+-------------------
+- Add timestamp to log messages. [Enrico Tröger]
+
+
+0.31.0 (2020-02-25)
+-------------------
+- #123 update: changed --as-app 'help' description. [ethan]
+- #123: Support Authenticating As Github Application. [ethan]
+
+
+0.29.0 (2020-02-14)
+-------------------
+- #50 update: keep main() in bin. [ethan]
+- #50 - refactor for friendlier import. [ethan]
+
+
+0.28.0 (2020-02-03)
+-------------------
+- Remove deprecated (and removed) git lfs flags. [smiley]
+
+  "--tags" and "--force" were removed at some point from "git lfs fetch". This broke our backup script.
+
+
+0.27.0 (2020-01-22)
+-------------------
+- Fixed script fails if not installed from pip. [Ben Baron]
+
+  At the top of the script, the line from github_backup import __version__ gets the script's version number to use if the script is called with the -v or --version flags. The problem is that if the script hasn't been installed via pip (for example I cloned the repo directly to my backup server), the script will fail due to an import exception.
+
+  Also presumably it will always use the version number from pip even if running a modified version from git or a fork or something, though this does not fix that as I have no idea how to check if it's running the pip installed version or not. But at least the script will now work fine if cloned from git or just copied to another machine.
+
+  closes https://github.com/josegonzalez/python-github-backup/issues/141
+- Fixed macOS keychain access when using Python 3. [Ben Baron]
+
+  Python 3 is returning bytes rather than a string, so the string concatenation to create the auth variable was throwing an exception which the script was interpreting to mean it couldn't find the password. Adding a conversion to string first fixed the issue.
+- Public repos no longer include the auth token. [Ben Baron]
+
+  When backing up repositories using an auth token and https, the GitHub personal auth token is leaked in each backed up repository. It is included in the URL of each repository's git remote url.
+
+  This is not needed as they are public and can be accessed without the token and can cause issues in the future if the token is ever changed, so I think it makes more sense not to have the token stored in each repo backup. I think the token should only be "leaked" like this out of necessity, e.g. it's a private repository and the --prefer-ssh option was not chosen so https with auth token was required to perform the clone.
+- Fixed comment typo. [Ben Baron]
+- Switched log_info to log_warning in download_file. [Ben Baron]
+- Crash when an release asset doesn't exist. [Ben Baron]
+
+  Currently, the script crashes whenever a release asset is unable to download (for example a 404 response). This change instead logs the failure and allows the script to continue. No retry logic is enabled, but at least it prevents the crash and allows the backup to complete. Retry logic can be implemented later if wanted.
+
+  closes https://github.com/josegonzalez/python-github-backup/issues/129
+- Moved asset downloading loop inside the if block. [Ben Baron]
+- Separate release assets and skip re-downloading. [Ben Baron]
+
+  Currently the script puts all release assets into the same folder called `releases`. So any time 2 release files have the same name, only the last one downloaded is actually saved. A particularly bad example of this is MacDownApp/macdown where all of their releases are named `MacDown.app.zip`. So even though they have 36 releases and all 36 are downloaded, only the last one is actually saved.
+
+  With this change, each releases' assets are now stored in a fubfolder inside `releases` named after the release name. There could still be edge cases if two releases have the same name, but this is still much safer tha the previous behavior.
+
+  This change also now checks if the asset file already exists on disk and skips downloading it. This drastically speeds up addiotnal syncs as it no longer downloads every single release every single time. It will now only download new releases which I believe is the expected behavior.
+
+  closes https://github.com/josegonzalez/python-github-backup/issues/126
+- Added newline to end of file. [Ben Baron]
+- Improved gitignore, macOS files and IDE configs. [Ben Baron]
+
+  Ignores the annoying hidden macOS files .DS_Store and ._* as well as the IDE configuration folders for contributors using the popular Visual Studio Code and Atom IDEs (more can be added later as needed).
+
+
+0.26.0 (2019-09-23)
+-------------------
 - Workaround gist clone in `--prefer-ssh` mode. [Vladislav Yarmak]
 - Create PULL_REQUEST.md. [Jose Diaz-Gonzalez]
 - Create ISSUE_TEMPLATE.md. [Jose Diaz-Gonzalez]

ISSUE_TEMPLATE.md

@@ -6,7 +6,7 @@ If pull requests implementing bug fixes or enhancements are pushed, I am happy t
 
 If you wish to have a bug fixed, you have a few options:
 
-- Fix it yourself.
+- Fix it yourself and file a pull request.
 - File a bug and hope someone else fixes it for you.
 - Pay me to fix it (my rate is $200 an hour, minimum 1 hour, contact me via my [github email address](https://github.com/josegonzalez) if you want to go this route).
 

README.rst

@@ -29,19 +29,19 @@ Usage
 
 CLI Usage is as follows::
 
-    github-backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN]
+    github-backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN] [--as-app]
                   [-o OUTPUT_DIRECTORY] [-i] [--starred] [--all-starred]
                   [--watched] [--followers] [--following] [--all]
                   [--issues] [--issue-comments] [--issue-events] [--pulls]
-                  [--pull-comments] [--pull-commits] [--labels] [--hooks]
+                  [--pull-comments] [--pull-commits] [--pull-details]
-                  [--milestones] [--repositories] [--releases] [--assets]
+                  [--labels] [--hooks] [--milestones] [--repositories]
                   [--bare] [--lfs] [--wikis] [--gists] [--starred-gists]
-                  [--skip-existing]
+                  [--skip-existing] [-L [LANGUAGES [LANGUAGES ...]]]
-                  [-L [LANGUAGES [LANGUAGES ...]]] [-N NAME_REGEX]
+                  [-N NAME_REGEX] [-H GITHUB_HOST] [-O] [-R REPOSITORY]
-                  [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-P] [-F]
+                  [-P] [-F] [--prefer-ssh] [-v]
-                  [--prefer-ssh] [-v]
                   [--keychain-name OSX_KEYCHAIN_ITEM_NAME]
                   [--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT]
+                  [--releases] [--assets]
                   USER
 
     Backup a github account
@@ -57,36 +57,36 @@ CLI Usage is as follows::
                             password for basic auth. If a username is given but
                             not a password, the password will be prompted for.
       -t TOKEN, --token TOKEN
-                            personal access or OAuth token, or path to token
+                            personal access, OAuth, or JSON Web token, or path to
-                            (file://...)
+                            token (file://...)
+      --as-app              authenticate as github app instead of as a user.
       -o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                             directory at which to backup the repositories
       -i, --incremental     incremental backup
       --starred             include JSON output of starred repositories in backup
-      --all-starred         include starred repositories in backup
+      --all-starred         include starred repositories in backup [*]
-      --watched             include watched repositories in backup
+      --watched             include JSON output of watched repositories in backup
       --followers           include JSON output of followers in backup
       --following           include JSON output of following users in backup
-      --all                 include everything in backup
+      --all                 include everything in backup (not including [*])
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-commits        include pull request commits in backup
+      --pull-details        include more pull request details in backup [*]
       --labels              include labels in backup
       --hooks               include hooks in backup (works only when
                             authenticated)
       --milestones          include milestones in backup
       --repositories        include repository clone in backup
-      --releases            include repository releases' information without assets or binaries
-      --assets              include assets alongside release information; only applies if including releases
       --bare                clone bare repositories
       --lfs                 clone LFS repositories (requires Git LFS to be
-                            installed, https://git-lfs.github.com)
+                            installed, https://git-lfs.github.com) [*]
       --wikis               include wiki clone in backup
-      --gists               include gists in backup
+      --gists               include gists in backup [*]
-      --starred-gists       include starred gists in backup
+      --starred-gists       include starred gists in backup [*]
       --skip-existing       skip project if a backup directory exists
       -L [LANGUAGES [LANGUAGES ...]], --languages [LANGUAGES [LANGUAGES ...]]
                             only allow these languages
@@ -97,8 +97,8 @@ CLI Usage is as follows::
       -O, --organization    whether or not this is an organization user
       -R REPOSITORY, --repository REPOSITORY
                             name of repository to limit backup to
-      -P, --private         include private repositories
+      -P, --private         include private repositories [*]
-      -F, --fork            include forked repositories
+      -F, --fork            include forked repositories [*]
       --prefer-ssh          Clone repositories using SSH instead of HTTPS
       -v, --version         show program's version number and exit
       --keychain-name OSX_KEYCHAIN_ITEM_NAME
@@ -107,6 +107,10 @@ CLI Usage is as follows::
       --keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT
                             OSX ONLY: account field of password item in OSX
                             keychain that holds the personal access or OAuth token
+      --releases            include release information, not including assets or
+                            binaries
+      --assets              include assets alongside release information; only
+                            applies if including releases
 
 
 The package can be used to backup an *entire* organization or repository, including issues and wikis in the most appropriate format (clones for wikis, json files for issues).

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = '0.26.0'
+__version__ = '0.33.0'