Compare commits

..

48 Commits

Author SHA1 Message Date
GitHub Action
6780d3ad6c Release version 0.61.1 2026-01-13 23:10:05 +00:00
Jose Diaz-Gonzalez
65bacc27f0 Merge pull request #478 from Iamrodos/fix-477-fine-grained-pat-attachments
Fix 477 fine grained pat attachments
2026-01-13 18:09:27 -05:00
Rodos
ab0eebb175 Refactor test fixtures to use shared create_args helper
Uses the real parse_args() function to get CLI defaults, so when
new arguments are added they're automatically available to all tests.

Changes:
- Add tests/conftest.py with create_args fixture
- Update 8 test files to use shared fixture
- Remove duplicate _create_mock_args methods
- Remove redundant @pytest.fixture mock_args definitions

This eliminates the need to update multiple test files when
adding new CLI arguments.
2026-01-13 13:47:33 +11:00
Rodos
fce4abb74a Fix fine-grained PAT attachment downloads for private repos (#477)
Fine-grained personal access tokens cannot download attachments from
private repositories directly due to a GitHub platform limitation.

This adds a workaround for image attachments (/assets/ URLs) using
GitHub's Markdown API to convert URLs to JWT-signed URLs that can be
downloaded without authentication.

Changes:
- Add get_jwt_signed_url_via_markdown_api() function
- Detect fine-grained token + private repo + /assets/ URL upfront
- Use JWT workaround for those cases, mark success with jwt_workaround flag
- Skip download with skipped_at when workaround fails
- Add startup warning when using --attachments with fine-grained tokens
- Document limitation in README (file attachments still fail)
- Add 6 unit tests for JWT workaround logic
2026-01-13 13:15:38 +11:00
GitHub Action
c63fb37d30 Release version 0.61.0 2026-01-12 16:30:28 +00:00
Jose Diaz-Gonzalez
94b08d06c9 Merge pull request #476 from lukasbestle/patch-1
docs: Add missing `--retries` argument to README
2026-01-12 11:29:56 -05:00
Jose Diaz-Gonzalez
54a9872e47 Merge pull request #475 from lukasbestle/feat/security-advisories
feat: Backup of repository security advisories
2026-01-11 14:26:39 -05:00
Lukas Bestle
b3d35f9d9f docs: Add missing --retries argument to README 2026-01-10 15:44:37 +01:00
Lukas Bestle
a175ac3ed9 test: Adapt tests to new argument 2026-01-10 11:12:42 +01:00
Lukas Bestle
9a6f0b4c21 feat: Backup of repository security advisories 2026-01-09 21:04:21 +01:00
GitHub Action
858731ebbd Release version 0.60.0 2025-12-24 00:45:01 +00:00
Jose Diaz-Gonzalez
2e999d0d3c Merge pull request #474 from mwtzzz/retry_logic
update retry logic and logging
2025-12-23 19:44:32 -05:00
michaelmartinez
44b0003ec9 updates to the tests, and fixes to the retry 2025-12-23 14:07:38 -08:00
michaelmartinez
5ab3852476 rm max_retries.py 2025-12-23 08:57:57 -08:00
michaelmartinez
8b21e2501c readme 2025-12-23 08:55:52 -08:00
michaelmartinez
f9827da342 don't use a global variable, pass the args instead 2025-12-23 08:53:54 -08:00
michaelmartinez
1f2ec016d5 readme, simplify the logic a bit 2025-12-22 16:13:12 -08:00
michaelmartinez
8b1b632d89 max_retries 5 2025-12-22 14:47:26 -08:00
michaelmartinez
89502c326d update retry logic and logging
### What
1. configureable retry count
2. additional logging

### Why
1. pass retry count as a command line arg; default 5
2. show details when api requests fail

### Testing before merge
compiles cleanly

### Validation after merge
compile and test

### Issue addressed by this PR
https://github.com/stellar/ops/issues/2039
2025-12-22 14:23:02 -08:00
GitHub Action
81a72ac8af Release version 0.59.0 2025-12-21 23:48:36 +00:00
Jose Diaz-Gonzalez
3edbfc777c Merge pull request #472 from Iamrodos/feature/108-starred-skip-size-over
Add --starred-skip-size-over flag to limit starred repo size (#108)
2025-12-21 18:47:58 -05:00
Rodos
3c43e0f481 Add --starred-skip-size-over flag to limit starred repo size (#108)
Allow users to skip starred repositories exceeding a size threshold
when using --all-starred. Size is specified in MB and checked against
the GitHub API's repository size field.

- Only affects starred repos; user's own repos always included
- Logs each skipped repo with name and size

Closes #108
2025-12-21 22:18:09 +11:00
Jose Diaz-Gonzalez
875f09eeaf Merge pull request #473 from Iamrodos/chore/remove-password-auth
chore: remove deprecated -u/-p password authentication options
2025-12-21 01:36:35 -05:00
Rodos
db36c3c137 chore: remove deprecated -u/-p password authentication options 2025-12-20 19:16:11 +11:00
GitHub Action
c70cc43f57 Release version 0.58.0 2025-12-16 15:17:23 +00:00
Jose Diaz-Gonzalez
27d3fcdafa Merge pull request #471 from Iamrodos/fix/retry-logic
Fix retry logic for HTTP 5xx errors and network failures
2025-12-16 10:16:48 -05:00
Rodos
46140b0ff1 Fix retry logic for HTTP 5xx errors and network failures
Refactors error handling to retry all 5xx errors (not just 502), network errors (URLError, socket.error, IncompleteRead), and JSON parse errors with exponential backoff and jitter. Respects retry-after and rate limit headers per GitHub API requirements. Consolidates retry logic into make_request_with_retry() wrapper and adds clear logging for retry attempts and failures. Removes dead code from 2016 (errors list, _request_http_error, _request_url_error) that was intentionally disabled in commit 1e5a9048 to fix #29.

Fixes #140, #110, #138
2025-12-16 21:55:47 +11:00
Jose Diaz-Gonzalez
02dd902b67 Merge pull request #470 from Iamrodos/chore/cleanup-release-requirements
chore: remove transitive deps from release-requirements.txt
2025-12-12 21:51:24 -05:00
Rodos
241949137d chore: remove transitive deps from release-requirements.txt 2025-12-13 11:22:53 +11:00
Jose Diaz-Gonzalez
1155da849d Merge pull request #469 from josegonzalez/dependabot/pip/python-packages-3c63e8caab
chore(deps): bump urllib3 from 2.6.1 to 2.6.2 in the python-packages group
2025-12-12 16:39:50 -05:00
dependabot[bot]
59a70ff11a chore(deps): bump urllib3 in the python-packages group
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).


Updates `urllib3` from 2.6.1 to 2.6.2
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.1...2.6.2)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-packages
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-12 13:09:29 +00:00
GitHub Action
ba852b5830 Release version 0.57.0 2025-12-12 11:07:14 +00:00
Jose Diaz-Gonzalez
934ee4b14b Merge pull request #467 from Iamrodos/docs/187-189-auth-docs
Add GitHub Apps documentation and stdin token example
2025-12-12 06:06:30 -05:00
Jose Diaz-Gonzalez
37a0c5c123 Merge pull request #468 from Iamrodos/feature/135-skip-assets-on
Add --skip-assets-on flag to skip release asset downloads (#135)
2025-12-12 06:05:47 -05:00
Rodos
f6e2f40b09 Add --skip-assets-on flag to skip release asset downloads (#135)
Allow users to skip downloading release assets for specific repositories
while still backing up release metadata. Useful for starred repos with
large assets (e.g. syncthing with 27GB+).

Usage: --skip-assets-on repo1 repo2 owner/repo3

Features:
- Space-separated repos (consistent with --exclude)
- Case-insensitive matching
- Supports both repo name and owner/repo format
2025-12-12 16:21:52 +11:00
Rodos
ef990483e2 Add GitHub Apps documentation and remove outdated header
- Add GitHub Apps authentication section with setup steps
  and CI/CD workflow example using actions/create-github-app-token
- Remove outdated machine-man-preview header (graduated 2020)

Closes #189
2025-12-12 10:25:49 +11:00
Rodos
3a513b6646 docs: add stdin token example to README
Add example showing how to pipe a token from stdin using
file:///dev/stdin to avoid storing tokens in environment
variables or command history.

Closes #187
2025-12-12 09:55:13 +11:00
GitHub Action
2bb83d6d8b Release version 0.56.0 2025-12-11 16:50:28 +00:00
Jose Diaz-Gonzalez
8fcc142621 Merge pull request #465 from Iamrodos/fix/379-lfs-clone-deprecated
fix: replace deprecated git lfs clone with git clone + git lfs fetch --all
2025-12-11 11:49:53 -05:00
Jose Diaz-Gonzalez
7615ce6102 Merge pull request #464 from Iamrodos/fix/246-restore-docs
docs: clarify no inbuilt restore and GitHub API limitations
2025-12-11 11:49:39 -05:00
Jose Diaz-Gonzalez
3f1ef821c3 Merge pull request #466 from Iamrodos/fix/112-windows-support
fix: add Windows support with entry_points and os.replace
2025-12-11 11:48:59 -05:00
Rodos
3684756eaa fix: add Windows support with entry_points and os.replace
- Replace os.rename() with os.replace() for atomic file operations
  on Windows (os.rename fails if destination exists on Windows)
- Add entry_points console_scripts for proper .exe generation on Windows
- Create github_backup/cli.py with main() entry point
- Add github_backup/__main__.py for python -m github_backup support
- Keep bin/github-backup as thin wrapper for backwards compatibility

Closes #112
2025-12-11 22:03:45 +11:00
Rodos
e745b55755 fix: replace deprecated git lfs clone with git clone + git lfs fetch --all
git lfs clone is deprecated - modern git clone handles LFS automatically.
Using git lfs fetch --all ensures all LFS objects across all refs are
backed up, matching the existing bare clone behavior and providing
complete LFS backups.

Closes #379
2025-12-11 20:55:38 +11:00
Rodos
75e6f56773 docs: add "Restoring from Backup" section to README
Clarifies that this tool is backup-only with no inbuilt restore.
Documents that git repos can be pushed back, but issues/PRs have
GitHub API limitations affecting all backup tools.

Closes #246
2025-12-11 20:35:08 +11:00
Jose Diaz-Gonzalez
b991c363a0 Merge pull request #463 from josegonzalez/dependabot/pip/python-packages-9e0978b55f
chore(deps): bump urllib3 from 2.6.0 to 2.6.1 in the python-packages group
2025-12-10 09:39:07 -05:00
dependabot[bot]
6d74af9126 chore(deps): bump urllib3 in the python-packages group
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).


Updates `urllib3` from 2.6.0 to 2.6.1
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.1)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-packages
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-09 13:10:12 +00:00
Jose Diaz-Gonzalez
381d67af96 Merge pull request #462 from josegonzalez/dependabot/pip/python-packages-3a01b12ef5
chore(deps): bump the python-packages group with 3 updates
2025-12-08 16:00:24 -05:00
dependabot[bot]
2fbe8d272c chore(deps): bump the python-packages group with 3 updates
Bumps the python-packages group with 3 updates: [black](https://github.com/psf/black), [pytest](https://github.com/pytest-dev/pytest) and [platformdirs](https://github.com/tox-dev/platformdirs).


Updates `black` from 25.11.0 to 25.12.0
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/25.11.0...25.12.0)

Updates `pytest` from 9.0.1 to 9.0.2
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.1...9.0.2)

Updates `platformdirs` from 4.5.0 to 4.5.1
- [Release notes](https://github.com/tox-dev/platformdirs/releases)
- [Changelog](https://github.com/tox-dev/platformdirs/blob/main/CHANGES.rst)
- [Commits](https://github.com/tox-dev/platformdirs/compare/4.5.0...4.5.1)

---
updated-dependencies:
- dependency-name: black
  dependency-version: 25.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-packages
- dependency-name: pytest
  dependency-version: 9.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-packages
- dependency-name: platformdirs
  dependency-version: 4.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-packages
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-08 13:09:32 +00:00
18 changed files with 2104 additions and 647 deletions

View File

@@ -1,9 +1,231 @@
Changelog Changelog
========= =========
0.55.0 (2025-12-07) 0.61.1 (2026-01-13)
------------------- -------------------
------------------------ ------------------------
- Refactor test fixtures to use shared create_args helper. [Rodos]
Uses the real parse_args() function to get CLI defaults, so when
new arguments are added they're automatically available to all tests.
Changes:
- Add tests/conftest.py with create_args fixture
- Update 8 test files to use shared fixture
- Remove duplicate _create_mock_args methods
- Remove redundant @pytest.fixture mock_args definitions
This eliminates the need to update multiple test files when
adding new CLI arguments.
- Fix fine-grained PAT attachment downloads for private repos (#477)
[Rodos]
Fine-grained personal access tokens cannot download attachments from
private repositories directly due to a GitHub platform limitation.
This adds a workaround for image attachments (/assets/ URLs) using
GitHub's Markdown API to convert URLs to JWT-signed URLs that can be
downloaded without authentication.
Changes:
- Add get_jwt_signed_url_via_markdown_api() function
- Detect fine-grained token + private repo + /assets/ URL upfront
- Use JWT workaround for those cases, mark success with jwt_workaround flag
- Skip download with skipped_at when workaround fails
- Add startup warning when using --attachments with fine-grained tokens
- Document limitation in README (file attachments still fail)
- Add 6 unit tests for JWT workaround logic
0.61.0 (2026-01-12)
-------------------
- Docs: Add missing `--retries` argument to README. [Lukas Bestle]
- Test: Adapt tests to new argument. [Lukas Bestle]
- Feat: Backup of repository security advisories. [Lukas Bestle]
0.60.0 (2025-12-24)
-------------------
- Rm max_retries.py. [michaelmartinez]
- Readme. [michaelmartinez]
- Don't use a global variable, pass the args instead. [michaelmartinez]
- Readme, simplify the logic a bit. [michaelmartinez]
- Max_retries 5. [michaelmartinez]
0.59.0 (2025-12-21)
-------------------
- Add --starred-skip-size-over flag to limit starred repo size (#108)
[Rodos]
Allow users to skip starred repositories exceeding a size threshold
when using --all-starred. Size is specified in MB and checked against
the GitHub API's repository size field.
- Only affects starred repos; user's own repos always included
- Logs each skipped repo with name and size
Closes #108
- Chore: remove deprecated -u/-p password authentication options.
[Rodos]
0.58.0 (2025-12-16)
-------------------
- Fix retry logic for HTTP 5xx errors and network failures. [Rodos]
Refactors error handling to retry all 5xx errors (not just 502), network errors (URLError, socket.error, IncompleteRead), and JSON parse errors with exponential backoff and jitter. Respects retry-after and rate limit headers per GitHub API requirements. Consolidates retry logic into make_request_with_retry() wrapper and adds clear logging for retry attempts and failures. Removes dead code from 2016 (errors list, _request_http_error, _request_url_error) that was intentionally disabled in commit 1e5a9048 to fix #29.
Fixes #140, #110, #138
- Chore: remove transitive deps from release-requirements.txt. [Rodos]
- Chore(deps): bump urllib3 in the python-packages group.
[dependabot[bot]]
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).
Updates `urllib3` from 2.6.1 to 2.6.2
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.1...2.6.2)
---
updated-dependencies:
- dependency-name: urllib3
dependency-version: 2.6.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: python-packages
...
0.57.0 (2025-12-12)
-------------------
- Add GitHub Apps documentation and remove outdated header. [Rodos]
- Add GitHub Apps authentication section with setup steps
and CI/CD workflow example using actions/create-github-app-token
- Remove outdated machine-man-preview header (graduated 2020)
Closes #189
- Docs: add stdin token example to README. [Rodos]
Add example showing how to pipe a token from stdin using
file:///dev/stdin to avoid storing tokens in environment
variables or command history.
Closes #187
- Add --skip-assets-on flag to skip release asset downloads (#135)
[Rodos]
Allow users to skip downloading release assets for specific repositories
while still backing up release metadata. Useful for starred repos with
large assets (e.g. syncthing with 27GB+).
Usage: --skip-assets-on repo1 repo2 owner/repo3
Features:
- Space-separated repos (consistent with --exclude)
- Case-insensitive matching
- Supports both repo name and owner/repo format
0.56.0 (2025-12-11)
-------------------
Fix
~~~
- Replace deprecated git lfs clone with git clone + git lfs fetch --all.
[Rodos]
git lfs clone is deprecated - modern git clone handles LFS automatically.
Using git lfs fetch --all ensures all LFS objects across all refs are
backed up, matching the existing bare clone behavior and providing
complete LFS backups.
Closes #379
- Add Windows support with entry_points and os.replace. [Rodos]
- Replace os.rename() with os.replace() for atomic file operations
on Windows (os.rename fails if destination exists on Windows)
- Add entry_points console_scripts for proper .exe generation on Windows
- Create github_backup/cli.py with main() entry point
- Add github_backup/__main__.py for python -m github_backup support
- Keep bin/github-backup as thin wrapper for backwards compatibility
Closes #112
Other
~~~~~
- Docs: add "Restoring from Backup" section to README. [Rodos]
Clarifies that this tool is backup-only with no inbuilt restore.
Documents that git repos can be pushed back, but issues/PRs have
GitHub API limitations affecting all backup tools.
Closes #246
- Chore(deps): bump urllib3 in the python-packages group.
[dependabot[bot]]
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).
Updates `urllib3` from 2.6.0 to 2.6.1
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.1)
---
updated-dependencies:
- dependency-name: urllib3
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: python-packages
...
- Chore(deps): bump the python-packages group with 3 updates.
[dependabot[bot]]
Bumps the python-packages group with 3 updates: [black](https://github.com/psf/black), [pytest](https://github.com/pytest-dev/pytest) and [platformdirs](https://github.com/tox-dev/platformdirs).
Updates `black` from 25.11.0 to 25.12.0
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/25.11.0...25.12.0)
Updates `pytest` from 9.0.1 to 9.0.2
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.1...9.0.2)
Updates `platformdirs` from 4.5.0 to 4.5.1
- [Release notes](https://github.com/tox-dev/platformdirs/releases)
- [Changelog](https://github.com/tox-dev/platformdirs/blob/main/CHANGES.rst)
- [Commits](https://github.com/tox-dev/platformdirs/compare/4.5.0...4.5.1)
---
updated-dependencies:
- dependency-name: black
dependency-version: 25.12.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: python-packages
- dependency-name: pytest
dependency-version: 9.0.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: python-packages
- dependency-name: platformdirs
dependency-version: 4.5.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: python-packages
...
0.55.0 (2025-12-07)
-------------------
Fix Fix
~~~ ~~~

View File

@@ -36,23 +36,26 @@ Show the CLI help output::
CLI Help output:: CLI Help output::
github-backup [-h] [-u USERNAME] [-p PASSWORD] [-t TOKEN_CLASSIC] github-backup [-h] [-t TOKEN_CLASSIC] [-f TOKEN_FINE] [-q] [--as-app]
[-f TOKEN_FINE] [--as-app] [-o OUTPUT_DIRECTORY] [-o OUTPUT_DIRECTORY] [-l LOG_LEVEL] [-i]
[-l LOG_LEVEL] [-i] [--starred] [--all-starred] [--incremental-by-files]
[--watched] [--followers] [--following] [--all] [--issues] [--starred] [--all-starred] [--starred-skip-size-over MB]
[--issue-comments] [--issue-events] [--pulls] [--watched] [--followers] [--following] [--all]
[--issues] [--issue-comments] [--issue-events] [--pulls]
[--pull-comments] [--pull-commits] [--pull-details] [--pull-comments] [--pull-commits] [--pull-details]
[--labels] [--hooks] [--milestones] [--repositories] [--labels] [--hooks] [--milestones] [--security-advisories]
[--bare] [--lfs] [--wikis] [--gists] [--starred-gists] [--repositories] [--bare] [--no-prune] [--lfs] [--wikis]
[--skip-archived] [--skip-existing] [-L [LANGUAGES ...]] [--gists] [--starred-gists] [--skip-archived] [--skip-existing]
[-N NAME_REGEX] [-H GITHUB_HOST] [-O] [-R REPOSITORY] [-L [LANGUAGES ...]] [-N NAME_REGEX] [-H GITHUB_HOST]
[-P] [-F] [--prefer-ssh] [-v] [-O] [-R REPOSITORY] [-P] [-F] [--prefer-ssh] [-v]
[--keychain-name OSX_KEYCHAIN_ITEM_NAME] [--keychain-name OSX_KEYCHAIN_ITEM_NAME]
[--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT] [--keychain-account OSX_KEYCHAIN_ITEM_ACCOUNT]
[--releases] [--latest-releases NUMBER_OF_LATEST_RELEASES] [--releases] [--latest-releases NUMBER_OF_LATEST_RELEASES]
[--skip-prerelease] [--assets] [--attachments] [--skip-prerelease] [--assets]
[--exclude [REPOSITORY [REPOSITORY ...]] [--skip-assets-on [SKIP_ASSETS_ON ...]] [--attachments]
[--throttle-limit THROTTLE_LIMIT] [--throttle-pause THROTTLE_PAUSE] [--throttle-limit THROTTLE_LIMIT]
[--throttle-pause THROTTLE_PAUSE]
[--exclude [EXCLUDE ...]] [--retries MAX_RETRIES]
USER USER
Backup a github account Backup a github account
@@ -60,29 +63,29 @@ CLI Help output::
positional arguments: positional arguments:
USER github username USER github username
optional arguments: options:
-h, --help show this help message and exit -h, --help show this help message and exit
-u USERNAME, --username USERNAME -t, --token TOKEN_CLASSIC
username for basic auth
-p PASSWORD, --password PASSWORD
password for basic auth. If a username is given but
not a password, the password will be prompted for.
-f TOKEN_FINE, --token-fine TOKEN_FINE
fine-grained personal access token or path to token
(file://...)
-t TOKEN_CLASSIC, --token TOKEN_CLASSIC
personal access, OAuth, or JSON Web token, or path to personal access, OAuth, or JSON Web token, or path to
token (file://...) token (file://...)
-f, --token-fine TOKEN_FINE
fine-grained personal access token (github_pat_....),
or path to token (file://...)
-q, --quiet supress log messages less severe than warning, e.g.
info
--as-app authenticate as github app instead of as a user. --as-app authenticate as github app instead of as a user.
-o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY -o, --output-directory OUTPUT_DIRECTORY
directory at which to backup the repositories directory at which to backup the repositories
-l LOG_LEVEL, --log-level LOG_LEVEL -l, --log-level LOG_LEVEL
log level to use (default: info, possible levels: log level to use (default: info, possible levels:
debug, info, warning, error, critical) debug, info, warning, error, critical)
-i, --incremental incremental backup -i, --incremental incremental backup
--incremental-by-files incremental backup using modified time of files --incremental-by-files
incremental backup based on modification date of files
--starred include JSON output of starred repositories in backup --starred include JSON output of starred repositories in backup
--all-starred include starred repositories in backup [*] --all-starred include starred repositories in backup [*]
--starred-skip-size-over MB
skip starred repositories larger than this size in MB
--watched include JSON output of watched repositories in backup --watched include JSON output of watched repositories in backup
--followers include JSON output of followers in backup --followers include JSON output of followers in backup
--following include JSON output of following users in backup --following include JSON output of following users in backup
@@ -98,22 +101,26 @@ CLI Help output::
--hooks include hooks in backup (works only when --hooks include hooks in backup (works only when
authenticated) authenticated)
--milestones include milestones in backup --milestones include milestones in backup
--security-advisories
include security advisories in backup
--repositories include repository clone in backup --repositories include repository clone in backup
--bare clone bare repositories --bare clone bare repositories
--no-prune disable prune option for git fetch
--lfs clone LFS repositories (requires Git LFS to be --lfs clone LFS repositories (requires Git LFS to be
installed, https://git-lfs.github.com) [*] installed, https://git-lfs.github.com) [*]
--wikis include wiki clone in backup --wikis include wiki clone in backup
--gists include gists in backup [*] --gists include gists in backup [*]
--starred-gists include starred gists in backup [*] --starred-gists include starred gists in backup [*]
--skip-archived skip project if it is archived
--skip-existing skip project if a backup directory exists --skip-existing skip project if a backup directory exists
-L [LANGUAGES [LANGUAGES ...]], --languages [LANGUAGES [LANGUAGES ...]] -L, --languages [LANGUAGES ...]
only allow these languages only allow these languages
-N NAME_REGEX, --name-regex NAME_REGEX -N, --name-regex NAME_REGEX
python regex to match names against python regex to match names against
-H GITHUB_HOST, --github-host GITHUB_HOST -H, --github-host GITHUB_HOST
GitHub Enterprise hostname GitHub Enterprise hostname
-O, --organization whether or not this is an organization user -O, --organization whether or not this is an organization user
-R REPOSITORY, --repository REPOSITORY -R, --repository REPOSITORY
name of repository to limit backup to name of repository to limit backup to
-P, --private include private repositories [*] -P, --private include private repositories [*]
-F, --fork include forked repositories [*] -F, --fork include forked repositories [*]
@@ -128,16 +135,16 @@ CLI Help output::
--releases include release information, not including assets or --releases include release information, not including assets or
binaries binaries
--latest-releases NUMBER_OF_LATEST_RELEASES --latest-releases NUMBER_OF_LATEST_RELEASES
include certain number of the latest releases; include certain number of the latest releases; only
only applies if including releases applies if including releases
--skip-prerelease skip prerelease and draft versions; only applies if including releases --skip-prerelease skip prerelease and draft versions; only applies if
including releases
--assets include assets alongside release information; only --assets include assets alongside release information; only
applies if including releases applies if including releases
--attachments download user-attachments from issues and pull requests --skip-assets-on [SKIP_ASSETS_ON ...]
to issues/attachments/{issue_number}/ and skip asset downloads for these repositories
pulls/attachments/{pull_number}/ directories --attachments download user-attachments from issues and pull
--exclude [REPOSITORY [REPOSITORY ...]] requests
names of repositories to exclude from backup.
--throttle-limit THROTTLE_LIMIT --throttle-limit THROTTLE_LIMIT
start throttling of GitHub API requests after this start throttling of GitHub API requests after this
amount of API requests remain amount of API requests remain
@@ -145,7 +152,10 @@ CLI Help output::
wait this amount of seconds when API request wait this amount of seconds when API request
throttling is active (default: 30.0, requires throttling is active (default: 30.0, requires
--throttle-limit to be set) --throttle-limit to be set)
--exclude [EXCLUDE ...]
names of repositories to exclude
--retries MAX_RETRIES
maximum number of retries for API calls (default: 5)
Usage Details Usage Details
============= =============
@@ -153,13 +163,13 @@ Usage Details
Authentication Authentication
-------------- --------------
**Password-based authentication** will fail if you have two-factor authentication enabled, and will `be deprecated <https://github.blog/2023-03-09-raising-the-bar-for-software-security-github-2fa-begins-march-13/>`_ by 2023 EOY. GitHub requires token-based authentication for API access. Password authentication was `removed in November 2020 <https://developer.github.com/changes/2020-02-14-deprecating-password-auth/>`_.
``--username`` is used for basic password authentication and separate from the positional argument ``USER``, which specifies the user account you wish to back up. The positional argument ``USER`` specifies the user or organization account you wish to back up.
**Classic tokens** are `slightly less secure <https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic>`_ as they provide very coarse-grained permissions. **Fine-grained tokens** (``-f TOKEN_FINE``) are recommended for most use cases, especially long-running backups (e.g. cron jobs), as they provide precise permission control.
If you need authentication for long-running backups (e.g. for a cron job) it is recommended to use **fine-grained personal access token** ``-f TOKEN_FINE``. **Classic tokens** (``-t TOKEN``) are `slightly less secure <https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic>`_ as they provide very coarse-grained permissions.
Fine Tokens Fine Tokens
@@ -174,6 +184,37 @@ Customise the permissions for your use case, but for a personal account full bac
**Repository permissions**: Read access to contents, issues, metadata, pull requests, and webhooks. **Repository permissions**: Read access to contents, issues, metadata, pull requests, and webhooks.
GitHub Apps
~~~~~~~~~~~
GitHub Apps are ideal for organization backups in CI/CD. Tokens are scoped to specific repositories and expire after 1 hour.
**One-time setup:**
1. Create a GitHub App at *Settings -> Developer Settings -> GitHub Apps -> New GitHub App*
2. Set a name and homepage URL (can be any URL)
3. Uncheck "Webhook > Active" (not needed for backups)
4. Set permissions (same as fine-grained tokens above)
5. Click "Create GitHub App", then note the **App ID** shown on the next page
6. Under "Private keys", click "Generate a private key" and save the downloaded file
7. Go to *Install App* in your app's settings
8. Select the account/organization and which repositories to back up
**CI/CD usage with GitHub Actions:**
Store the App ID as a repository variable and the private key contents as a secret, then use ``actions/create-github-app-token``::
- uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ vars.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- run: github-backup myorg -t ${{ steps.app-token.outputs.token }} --as-app -o ./backup --all
Note: Installation tokens expire after 1 hour. For long-running backups, use a fine-grained personal access token instead.
Prefer SSH Prefer SSH
~~~~~~~~~~ ~~~~~~~~~~
@@ -215,6 +256,8 @@ When you use the ``--lfs`` option, you will need to make sure you have Git LFS i
Instructions on how to do this can be found on https://git-lfs.github.com. Instructions on how to do this can be found on https://git-lfs.github.com.
LFS objects are fetched for all refs, not just the current checkout, ensuring a complete backup of all LFS content across all branches and history.
About Attachments About Attachments
----------------- -----------------
@@ -238,6 +281,8 @@ The tool automatically extracts file extensions from HTTP headers to ensure file
**Repository filtering** for repo files/assets handles renamed and transferred repositories gracefully. URLs are included if they either match the current repository name directly, or redirect to it (e.g., ``willmcgugan/rich`` redirects to ``Textualize/rich`` after transfer). **Repository filtering** for repo files/assets handles renamed and transferred repositories gracefully. URLs are included if they either match the current repository name directly, or redirect to it (e.g., ``willmcgugan/rich`` redirects to ``Textualize/rich`` after transfer).
**Fine-grained token limitation:** Due to a GitHub platform limitation, fine-grained personal access tokens (``github_pat_...``) cannot download attachments from private repositories directly. This affects both ``/assets/`` (images) and ``/files/`` (documents) URLs. The tool implements a workaround for image attachments using GitHub's Markdown API, which converts URLs to temporary JWT-signed URLs that can be downloaded. However, this workaround only works for images - document attachments (PDFs, text files, etc.) will fail with 404 errors when using fine-grained tokens on private repos. For full attachment support on private repositories, use a classic token (``-t``) instead of a fine-grained token (``-f``). See `#477 <https://github.com/josegonzalez/python-github-backup/issues/477>`_ for details.
Run in Docker container Run in Docker container
----------------------- -----------------------
@@ -254,10 +299,20 @@ All is not everything
The ``--all`` argument does not include: cloning private repos (``-P, --private``), cloning forks (``-F, --fork``), cloning starred repositories (``--all-starred``), ``--pull-details``, cloning LFS repositories (``--lfs``), cloning gists (``--gists``) or cloning starred gist repos (``--starred-gists``). See examples for more. The ``--all`` argument does not include: cloning private repos (``-P, --private``), cloning forks (``-F, --fork``), cloning starred repositories (``--all-starred``), ``--pull-details``, cloning LFS repositories (``--lfs``), cloning gists (``--gists``) or cloning starred gist repos (``--starred-gists``). See examples for more.
Cloning all starred size Starred repository size
------------------------ -----------------------
Using the ``--all-starred`` argument to clone all starred repositories may use a large amount of storage space, especially if ``--all`` or more arguments are used. e.g. commonly starred repos can have tens of thousands of issues, many large assets and the repo itself etc. Consider just storing links to starred repos in JSON format with ``--starred``. Using the ``--all-starred`` argument to clone all starred repositories may use a large amount of storage space.
To see your starred repositories sorted by size (requires `GitHub CLI <https://cli.github.com>`_)::
gh api user/starred --paginate --jq 'sort_by(-.size)[]|"\(.full_name) \(.size/1024|round)MB"'
To limit which starred repositories are cloned, use ``--starred-skip-size-over SIZE`` where SIZE is in MB. For example, ``--starred-skip-size-over 500`` will skip any starred repository where the git repository size (code and history) exceeds 500 MB. Note that this size limit only applies to the repository itself, not issues, release assets or other metadata. This filter only affects starred repositories; your own repositories are always included regardless of size.
For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-assets-on`` for specific repositories with large release binaries.
Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
Incremental Backup Incremental Backup
------------------ ------------------
@@ -350,7 +405,7 @@ Quietly and incrementally backup useful Github user data (public and private rep
export FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN export FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
GH_USER=YOUR-GITHUB-USER GH_USER=YOUR-GITHUB-USER
github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --security-advisories --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. :: Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
@@ -359,6 +414,28 @@ Debug an error/block or incomplete backup into a temporary directory. Omit "incr
github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
my-secret-manager get github-token | github-backup user -t file:///dev/stdin -o /backup --repositories
Restoring from Backup
=====================
This tool creates backups only, there is no inbuilt restore command.
**Git repositories, wikis, and gists** can be restored by pushing them back to GitHub as you would any git repository. For example, to restore a bare repository backup::
cd /tmp/white-house/repositories/petitions/repository
git push --mirror git@github.com:WhiteHouse/petitions.git
**Issues, pull requests, comments, and other metadata** are saved as JSON files for archival purposes. The GitHub API does not support recreating this data faithfully, creating issues via the API has limitations:
- New issue/PR numbers are assigned (original numbers cannot be set)
- Timestamps reflect creation time (original dates cannot be set)
- The API caller becomes the author (original authors cannot be set)
- Cross-references between issues and PRs will break
These are GitHub API limitations that affect all backup and migration tools, not just this one. Recreating issues with these limitations via the GitHub API is an exercise for the reader. The JSON backups remain useful for searching, auditing, or manual reference.
Development Development

View File

@@ -1,76 +1,18 @@
#!/usr/bin/env python #!/usr/bin/env python
"""
Backwards-compatible wrapper script.
The recommended way to run github-backup is via the installed command
(pip install github-backup) or python -m github_backup.
This script is kept for backwards compatibility with existing installations
that may reference this path directly.
"""
import logging
import os
import sys import sys
from github_backup.github_backup import ( from github_backup.cli import main
backup_account, from github_backup.github_backup import logger
backup_repositories,
check_git_lfs_install,
filter_repositories,
get_auth,
get_authenticated_user,
logger,
mkdir_p,
parse_args,
retrieve_repositories,
)
# INFO and DEBUG go to stdout, WARNING and above go to stderr
log_format = logging.Formatter(
fmt="%(asctime)s.%(msecs)03d: %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
)
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
stdout_handler.setFormatter(log_format)
stderr_handler = logging.StreamHandler(sys.stderr)
stderr_handler.setLevel(logging.WARNING)
stderr_handler.setFormatter(log_format)
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
def main():
args = parse_args()
if args.private and not get_auth(args):
logger.warning(
"The --private flag has no effect without authentication. "
"Use -t/--token, -f/--token-fine, or -u/--username to authenticate."
)
if args.quiet:
logger.setLevel(logging.WARNING)
output_directory = os.path.realpath(args.output_directory)
if not os.path.isdir(output_directory):
logger.info("Create output directory {0}".format(output_directory))
mkdir_p(output_directory)
if args.lfs_clone:
check_git_lfs_install()
if args.log_level:
log_level = logging.getLevelName(args.log_level.upper())
if isinstance(log_level, int):
logger.root.setLevel(log_level)
if not args.as_app:
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
authenticated_user = get_authenticated_user(args)
else:
authenticated_user = {"login": None}
repositories = retrieve_repositories(args, authenticated_user)
repositories = filter_repositories(args, repositories)
backup_repositories(args, output_directory, repositories)
backup_account(args, output_directory)
if __name__ == "__main__": if __name__ == "__main__":
try: try:

View File

@@ -1 +1 @@
__version__ = "0.55.0" __version__ = "0.61.1"

13
github_backup/__main__.py Normal file
View File

@@ -0,0 +1,13 @@
"""Allow running as: python -m github_backup"""
import sys
from github_backup.cli import main
from github_backup.github_backup import logger
if __name__ == "__main__":
try:
main()
except Exception as e:
logger.error(str(e))
sys.exit(1)

92
github_backup/cli.py Normal file
View File

@@ -0,0 +1,92 @@
#!/usr/bin/env python
"""Command-line interface for github-backup."""
import logging
import os
import sys
from github_backup.github_backup import (
backup_account,
backup_repositories,
check_git_lfs_install,
filter_repositories,
get_auth,
get_authenticated_user,
logger,
mkdir_p,
parse_args,
retrieve_repositories,
)
# INFO and DEBUG go to stdout, WARNING and above go to stderr
log_format = logging.Formatter(
fmt="%(asctime)s.%(msecs)03d: %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
)
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
stdout_handler.setFormatter(log_format)
stderr_handler = logging.StreamHandler(sys.stderr)
stderr_handler.setLevel(logging.WARNING)
stderr_handler.setFormatter(log_format)
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
def main():
"""Main entry point for github-backup CLI."""
args = parse_args()
if args.private and not get_auth(args):
logger.warning(
"The --private flag has no effect without authentication. "
"Use -t/--token or -f/--token-fine to authenticate."
)
# Issue #477: Fine-grained PATs cannot download all attachment types from
# private repos. Image attachments will be retried via Markdown API workaround.
if args.include_attachments and args.token_fine:
logger.warning(
"Using --attachments with fine-grained token. Due to GitHub platform "
"limitations, file attachments (PDFs, etc.) from private repos may fail. "
"Image attachments will be retried via workaround. For full attachment "
"support, use --token-classic instead."
)
if args.quiet:
logger.setLevel(logging.WARNING)
output_directory = os.path.realpath(args.output_directory)
if not os.path.isdir(output_directory):
logger.info("Create output directory {0}".format(output_directory))
mkdir_p(output_directory)
if args.lfs_clone:
check_git_lfs_install()
if args.log_level:
log_level = logging.getLevelName(args.log_level.upper())
if isinstance(log_level, int):
logger.root.setLevel(log_level)
if not args.as_app:
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
authenticated_user = get_authenticated_user(args)
else:
authenticated_user = {"login": None}
repositories = retrieve_repositories(args, authenticated_user)
repositories = filter_repositories(args, repositories)
backup_repositories(args, output_directory, repositories)
backup_account(args, output_directory)
if __name__ == "__main__":
try:
main()
except Exception as e:
logger.error(str(e))
sys.exit(1)

File diff suppressed because it is too large Load Diff

View File

@@ -1,40 +1,15 @@
# Linting & Formatting
autopep8==2.3.2 autopep8==2.3.2
black==25.11.0 black==25.12.0
bleach==6.3.0
certifi==2025.11.12
charset-normalizer==3.4.4
click==8.3.1
colorama==0.4.6
docutils==0.22.3
flake8==7.3.0 flake8==7.3.0
gitchangelog==3.0.4
pytest==9.0.1 # Testing
idna==3.11 pytest==9.0.2
importlib-metadata==8.7.0
jaraco.classes==3.4.0 # Release & Publishing
keyring==25.7.0
markdown-it-py==4.0.0
mccabe==0.7.0
mdurl==0.1.2
more-itertools==10.8.0
mypy-extensions==1.1.0
packaging==25.0
pathspec==0.12.1
pkginfo==1.12.1.2
platformdirs==4.5.0
pycodestyle==2.14.0
pyflakes==3.4.0
Pygments==2.19.2
readme-renderer==44.0
requests==2.32.5
requests-toolbelt==1.0.0
restructuredtext-lint==2.0.2
rfc3986==2.0.0
rich==14.2.0
setuptools==80.9.0
six==1.17.0
tqdm==4.67.1
twine==6.2.0 twine==6.2.0
urllib3==2.6.0 gitchangelog==3.0.4
webencodings==0.5.1 setuptools==80.9.0
zipp==3.23.0
# Documentation
restructuredtext-lint==2.0.2

View File

@@ -33,7 +33,11 @@ setup(
author="Jose Diaz-Gonzalez", author="Jose Diaz-Gonzalez",
author_email="github-backup@josediazgonzalez.com", author_email="github-backup@josediazgonzalez.com",
packages=["github_backup"], packages=["github_backup"],
scripts=["bin/github-backup"], entry_points={
"console_scripts": [
"github-backup=github_backup.cli:main",
],
},
url="http://github.com/josegonzalez/python-github-backup", url="http://github.com/josegonzalez/python-github-backup",
license="MIT", license="MIT",
classifiers=[ classifiers=[

25
tests/conftest.py Normal file
View File

@@ -0,0 +1,25 @@
"""Shared pytest fixtures for github-backup tests."""
import pytest
from github_backup.github_backup import parse_args
@pytest.fixture
def create_args():
"""Factory fixture that creates args with real CLI defaults.
Uses the actual argument parser so new CLI args are automatically
available with their defaults - no test updates needed.
Usage:
def test_something(self, create_args):
args = create_args(include_releases=True, user="myuser")
"""
def _create(**overrides):
# Use real parser to get actual defaults
args = parse_args(["testuser"])
for key, value in overrides.items():
setattr(args, key, value)
return args
return _create

View File

@@ -1,7 +1,7 @@
"""Tests for --all-starred flag behavior (issue #225).""" """Tests for --all-starred flag behavior (issue #225)."""
import pytest import pytest
from unittest.mock import Mock, patch from unittest.mock import patch
from github_backup import github_backup from github_backup import github_backup
@@ -12,59 +12,14 @@ class TestAllStarredCloning:
Issue #225: --all-starred should clone starred repos without requiring --repositories. Issue #225: --all-starred should clone starred repos without requiring --repositories.
""" """
def _create_mock_args(self, **overrides):
"""Create a mock args object with sensible defaults."""
args = Mock()
args.user = "testuser"
args.output_directory = "/tmp/backup"
args.include_repository = False
args.include_everything = False
args.include_gists = False
args.include_starred_gists = False
args.all_starred = False
args.skip_existing = False
args.bare_clone = False
args.lfs_clone = False
args.no_prune = False
args.include_wiki = False
args.include_issues = False
args.include_issue_comments = False
args.include_issue_events = False
args.include_pulls = False
args.include_pull_comments = False
args.include_pull_commits = False
args.include_pull_details = False
args.include_labels = False
args.include_hooks = False
args.include_milestones = False
args.include_releases = False
args.include_assets = False
args.include_attachments = False
args.incremental = False
args.incremental_by_files = False
args.github_host = None
args.prefer_ssh = False
args.token_classic = None
args.token_fine = None
args.username = None
args.password = None
args.as_app = False
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
for key, value in overrides.items():
setattr(args, key, value)
return args
@patch('github_backup.github_backup.fetch_repository') @patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url') @patch('github_backup.github_backup.get_github_repo_url')
def test_all_starred_clones_without_repositories_flag(self, mock_get_url, mock_fetch): def test_all_starred_clones_without_repositories_flag(self, mock_get_url, mock_fetch, create_args):
"""--all-starred should clone starred repos without --repositories flag. """--all-starred should clone starred repos without --repositories flag.
This is the core fix for issue #225. This is the core fix for issue #225.
""" """
args = self._create_mock_args(all_starred=True) args = create_args(all_starred=True)
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git" mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
# A starred repository (is_starred flag set by retrieve_repositories) # A starred repository (is_starred flag set by retrieve_repositories)
@@ -89,9 +44,9 @@ class TestAllStarredCloning:
@patch('github_backup.github_backup.fetch_repository') @patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url') @patch('github_backup.github_backup.get_github_repo_url')
def test_starred_repo_not_cloned_without_all_starred_flag(self, mock_get_url, mock_fetch): def test_starred_repo_not_cloned_without_all_starred_flag(self, mock_get_url, mock_fetch, create_args):
"""Starred repos should NOT be cloned if --all-starred is not set.""" """Starred repos should NOT be cloned if --all-starred is not set."""
args = self._create_mock_args(all_starred=False) args = create_args(all_starred=False)
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git" mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
starred_repo = { starred_repo = {
@@ -112,9 +67,9 @@ class TestAllStarredCloning:
@patch('github_backup.github_backup.fetch_repository') @patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url') @patch('github_backup.github_backup.get_github_repo_url')
def test_non_starred_repo_not_cloned_with_only_all_starred(self, mock_get_url, mock_fetch): def test_non_starred_repo_not_cloned_with_only_all_starred(self, mock_get_url, mock_fetch, create_args):
"""Non-starred repos should NOT be cloned when only --all-starred is set.""" """Non-starred repos should NOT be cloned when only --all-starred is set."""
args = self._create_mock_args(all_starred=True) args = create_args(all_starred=True)
mock_get_url.return_value = "https://github.com/testuser/my-project.git" mock_get_url.return_value = "https://github.com/testuser/my-project.git"
# A regular (non-starred) repository # A regular (non-starred) repository
@@ -136,9 +91,9 @@ class TestAllStarredCloning:
@patch('github_backup.github_backup.fetch_repository') @patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url') @patch('github_backup.github_backup.get_github_repo_url')
def test_repositories_flag_still_works(self, mock_get_url, mock_fetch): def test_repositories_flag_still_works(self, mock_get_url, mock_fetch, create_args):
"""--repositories flag should still clone repos as before.""" """--repositories flag should still clone repos as before."""
args = self._create_mock_args(include_repository=True) args = create_args(include_repository=True)
mock_get_url.return_value = "https://github.com/testuser/my-project.git" mock_get_url.return_value = "https://github.com/testuser/my-project.git"
regular_repo = { regular_repo = {

View File

@@ -4,7 +4,7 @@ import json
import os import os
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from unittest.mock import Mock from unittest.mock import Mock, patch
import pytest import pytest
@@ -12,24 +12,13 @@ from github_backup import github_backup
@pytest.fixture @pytest.fixture
def attachment_test_setup(tmp_path): def attachment_test_setup(tmp_path, create_args):
"""Fixture providing setup and helper for attachment download tests.""" """Fixture providing setup and helper for attachment download tests."""
from unittest.mock import patch
issue_cwd = tmp_path / "issues" issue_cwd = tmp_path / "issues"
issue_cwd.mkdir() issue_cwd.mkdir()
# Mock args # Create args using shared fixture
args = Mock() args = create_args(user="testuser", repository="testrepo")
args.as_app = False
args.token_fine = None
args.token_classic = None
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.user = "testuser"
args.repository = "testrepo"
repository = {"full_name": "testuser/testrepo"} repository = {"full_name": "testuser/testrepo"}
@@ -351,3 +340,146 @@ class TestManifestDuplicatePrevention:
downloaded_urls[0] downloaded_urls[0]
== "https://github.com/user-attachments/assets/unavailable" == "https://github.com/user-attachments/assets/unavailable"
) )
class TestJWTWorkaround:
"""Test JWT workaround for fine-grained tokens on private repos (issue #477)."""
def test_markdown_api_extracts_jwt_url(self):
"""Markdown API response with JWT URL is extracted correctly."""
html_response = (
'<p><a href="https://private-user-images.githubusercontent.com'
'/123/abc.png?jwt=eyJhbGciOiJ"><img src="https://private-user-'
'images.githubusercontent.com/123/abc.png?jwt=eyJhbGciOiJ" '
'alt="img"></a></p>'
)
mock_response = Mock()
mock_response.read.return_value = html_response.encode("utf-8")
with patch("github_backup.github_backup.urlopen", return_value=mock_response):
result = github_backup.get_jwt_signed_url_via_markdown_api(
"https://github.com/user-attachments/assets/abc123",
"github_pat_token",
"owner/repo"
)
expected = (
"https://private-user-images.githubusercontent.com"
"/123/abc.png?jwt=eyJhbGciOiJ"
)
assert result == expected
def test_markdown_api_returns_none_on_http_error(self):
"""HTTP errors return None."""
from urllib.error import HTTPError
error = HTTPError("http://test", 403, "Forbidden", {}, None)
with patch("github_backup.github_backup.urlopen", side_effect=error):
result = github_backup.get_jwt_signed_url_via_markdown_api(
"https://github.com/user-attachments/assets/abc123",
"github_pat_token",
"owner/repo"
)
assert result is None
def test_markdown_api_returns_none_when_no_jwt_url(self):
"""Response without JWT URL returns None."""
mock_response = Mock()
mock_response.read.return_value = b"<p>No image here</p>"
with patch("github_backup.github_backup.urlopen", return_value=mock_response):
result = github_backup.get_jwt_signed_url_via_markdown_api(
"https://github.com/user-attachments/assets/abc123",
"github_pat_token",
"owner/repo"
)
assert result is None
def test_needs_jwt_only_for_fine_grained_private_assets(self):
"""needs_jwt is True only for fine-grained + private + /assets/ URL."""
assets_url = "https://github.com/user-attachments/assets/abc123"
files_url = "https://github.com/user-attachments/files/123/doc.pdf"
token_fine = "github_pat_test"
private = True
public = False
# Fine-grained + private + assets = True
needs_jwt = (
token_fine is not None
and private
and "github.com/user-attachments/assets/" in assets_url
)
assert needs_jwt is True
# Fine-grained + private + files = False
needs_jwt = (
token_fine is not None
and private
and "github.com/user-attachments/assets/" in files_url
)
assert needs_jwt is False
# Fine-grained + public + assets = False
needs_jwt = (
token_fine is not None
and public
and "github.com/user-attachments/assets/" in assets_url
)
assert needs_jwt is False
def test_jwt_workaround_sets_manifest_flag(self, attachment_test_setup):
"""Successful JWT workaround sets jwt_workaround flag in manifest."""
setup = attachment_test_setup
setup["args"].token_fine = "github_pat_test"
setup["repository"]["private"] = True
issue_data = {"body": "https://github.com/user-attachments/assets/abc123"}
jwt_url = "https://private-user-images.githubusercontent.com/123/abc.png?jwt=token"
with patch(
"github_backup.github_backup.get_jwt_signed_url_via_markdown_api",
return_value=jwt_url
), patch(
"github_backup.github_backup.download_attachment_file",
return_value={"success": True, "http_status": 200, "url": jwt_url}
):
github_backup.download_attachments(
setup["args"], setup["issue_cwd"], issue_data, 123, setup["repository"]
)
manifest_path = os.path.join(setup["issue_cwd"], "attachments", "123", "manifest.json")
with open(manifest_path) as f:
manifest = json.load(f)
assert manifest["attachments"][0]["jwt_workaround"] is True
assert manifest["attachments"][0]["url"] == "https://github.com/user-attachments/assets/abc123"
def test_jwt_workaround_failure_uses_skipped_at(self, attachment_test_setup):
"""Failed JWT workaround uses skipped_at instead of downloaded_at."""
setup = attachment_test_setup
setup["args"].token_fine = "github_pat_test"
setup["repository"]["private"] = True
issue_data = {"body": "https://github.com/user-attachments/assets/abc123"}
with patch(
"github_backup.github_backup.get_jwt_signed_url_via_markdown_api",
return_value=None # Markdown API failed
):
github_backup.download_attachments(
setup["args"], setup["issue_cwd"], issue_data, 123, setup["repository"]
)
manifest_path = os.path.join(setup["issue_cwd"], "attachments", "123", "manifest.json")
with open(manifest_path) as f:
manifest = json.load(f)
attachment = manifest["attachments"][0]
assert attachment["success"] is False
assert "skipped_at" in attachment
assert "downloaded_at" not in attachment
assert "Use --token-classic" in attachment["error"]

View File

@@ -1,7 +1,6 @@
"""Tests for case-insensitive username/organization filtering.""" """Tests for case-insensitive username/organization filtering."""
import pytest import pytest
from unittest.mock import Mock
from github_backup import github_backup from github_backup import github_backup
@@ -9,23 +8,14 @@ from github_backup import github_backup
class TestCaseSensitivity: class TestCaseSensitivity:
"""Test suite for case-insensitive username matching in filter_repositories.""" """Test suite for case-insensitive username matching in filter_repositories."""
def test_filter_repositories_case_insensitive_user(self): def test_filter_repositories_case_insensitive_user(self, create_args):
"""Should filter repositories case-insensitively for usernames. """Should filter repositories case-insensitively for usernames.
Reproduces issue #198 where typing 'iamrodos' fails to match Reproduces issue #198 where typing 'iamrodos' fails to match
repositories with owner.login='Iamrodos' (the canonical case from GitHub API). repositories with owner.login='Iamrodos' (the canonical case from GitHub API).
""" """
# Simulate user typing lowercase username # Simulate user typing lowercase username
args = Mock() args = create_args(user="iamrodos")
args.user = "iamrodos" # lowercase (what user typed)
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
# Simulate GitHub API returning canonical case # Simulate GitHub API returning canonical case
repos = [ repos = [
@@ -50,21 +40,12 @@ class TestCaseSensitivity:
assert filtered[0]["name"] == "repo1" assert filtered[0]["name"] == "repo1"
assert filtered[1]["name"] == "repo2" assert filtered[1]["name"] == "repo2"
def test_filter_repositories_case_insensitive_org(self): def test_filter_repositories_case_insensitive_org(self, create_args):
"""Should filter repositories case-insensitively for organizations. """Should filter repositories case-insensitively for organizations.
Tests the example from issue #198 where 'prai-org' doesn't match 'PRAI-Org'. Tests the example from issue #198 where 'prai-org' doesn't match 'PRAI-Org'.
""" """
args = Mock() args = create_args(user="prai-org")
args.user = "prai-org" # lowercase (what user typed)
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
repos = [ repos = [
{ {
@@ -81,18 +62,9 @@ class TestCaseSensitivity:
assert len(filtered) == 1 assert len(filtered) == 1
assert filtered[0]["name"] == "repo1" assert filtered[0]["name"] == "repo1"
def test_filter_repositories_case_variations(self): def test_filter_repositories_case_variations(self, create_args):
"""Should handle various case combinations correctly.""" """Should handle various case combinations correctly."""
args = Mock() args = create_args(user="TeSt-UsEr")
args.user = "TeSt-UsEr" # Mixed case
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
repos = [ repos = [
{"name": "repo1", "owner": {"login": "test-user"}, "private": False, "fork": False}, {"name": "repo1", "owner": {"login": "test-user"}, "private": False, "fork": False},

View File

@@ -11,21 +11,10 @@ from github_backup import github_backup
class TestHTTP451Exception: class TestHTTP451Exception:
"""Test suite for HTTP 451 DMCA takedown exception handling.""" """Test suite for HTTP 451 DMCA takedown exception handling."""
def test_repository_unavailable_error_raised(self): def test_repository_unavailable_error_raised(self, create_args):
"""HTTP 451 should raise RepositoryUnavailableError with DMCA URL.""" """HTTP 451 should raise RepositoryUnavailableError with DMCA URL."""
# Create mock args args = create_args()
args = Mock()
args.as_app = False
args.token_fine = None
args.token_classic = None
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.throttle_limit = None
args.throttle_pause = 0
# Mock HTTPError 451 response
mock_response = Mock() mock_response = Mock()
mock_response.getcode.return_value = 451 mock_response.getcode.return_value = 451
@@ -34,36 +23,31 @@ class TestHTTP451Exception:
"block": { "block": {
"reason": "dmca", "reason": "dmca",
"created_at": "2024-11-12T14:38:04Z", "created_at": "2024-11-12T14:38:04Z",
"html_url": "https://github.com/github/dmca/blob/master/2024/11/2024-11-04-source-code.md" "html_url": "https://github.com/github/dmca/blob/master/2024/11/2024-11-04-source-code.md",
} },
} }
mock_response.read.return_value = json.dumps(dmca_data).encode("utf-8") mock_response.read.return_value = json.dumps(dmca_data).encode("utf-8")
mock_response.headers = {"x-ratelimit-remaining": "5000"} mock_response.headers = {"x-ratelimit-remaining": "5000"}
mock_response.reason = "Unavailable For Legal Reasons" mock_response.reason = "Unavailable For Legal Reasons"
def mock_get_response(request, auth, template): with patch(
return mock_response, [] "github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
with patch("github_backup.github_backup._get_response", side_effect=mock_get_response): ):
with pytest.raises(github_backup.RepositoryUnavailableError) as exc_info: with pytest.raises(github_backup.RepositoryUnavailableError) as exc_info:
list(github_backup.retrieve_data_gen(args, "https://api.github.com/repos/test/dmca/issues")) github_backup.retrieve_data(
args, "https://api.github.com/repos/test/dmca/issues"
)
# Check exception has DMCA URL assert (
assert exc_info.value.dmca_url == "https://github.com/github/dmca/blob/master/2024/11/2024-11-04-source-code.md" exc_info.value.dmca_url
== "https://github.com/github/dmca/blob/master/2024/11/2024-11-04-source-code.md"
)
assert "451" in str(exc_info.value) assert "451" in str(exc_info.value)
def test_repository_unavailable_error_without_dmca_url(self): def test_repository_unavailable_error_without_dmca_url(self, create_args):
"""HTTP 451 without DMCA details should still raise exception.""" """HTTP 451 without DMCA details should still raise exception."""
args = Mock() args = create_args()
args.as_app = False
args.token_fine = None
args.token_classic = None
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.throttle_limit = None
args.throttle_pause = 0
mock_response = Mock() mock_response = Mock()
mock_response.getcode.return_value = 451 mock_response.getcode.return_value = 451
@@ -71,29 +55,21 @@ class TestHTTP451Exception:
mock_response.headers = {"x-ratelimit-remaining": "5000"} mock_response.headers = {"x-ratelimit-remaining": "5000"}
mock_response.reason = "Unavailable For Legal Reasons" mock_response.reason = "Unavailable For Legal Reasons"
def mock_get_response(request, auth, template): with patch(
return mock_response, [] "github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
with patch("github_backup.github_backup._get_response", side_effect=mock_get_response): ):
with pytest.raises(github_backup.RepositoryUnavailableError) as exc_info: with pytest.raises(github_backup.RepositoryUnavailableError) as exc_info:
list(github_backup.retrieve_data_gen(args, "https://api.github.com/repos/test/dmca/issues")) github_backup.retrieve_data(
args, "https://api.github.com/repos/test/dmca/issues"
)
# Exception raised even without DMCA URL
assert exc_info.value.dmca_url is None assert exc_info.value.dmca_url is None
assert "451" in str(exc_info.value) assert "451" in str(exc_info.value)
def test_repository_unavailable_error_with_malformed_json(self): def test_repository_unavailable_error_with_malformed_json(self, create_args):
"""HTTP 451 with malformed JSON should still raise exception.""" """HTTP 451 with malformed JSON should still raise exception."""
args = Mock() args = create_args()
args.as_app = False
args.token_fine = None
args.token_classic = None
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.throttle_limit = None
args.throttle_pause = 0
mock_response = Mock() mock_response = Mock()
mock_response.getcode.return_value = 451 mock_response.getcode.return_value = 451
@@ -101,42 +77,14 @@ class TestHTTP451Exception:
mock_response.headers = {"x-ratelimit-remaining": "5000"} mock_response.headers = {"x-ratelimit-remaining": "5000"}
mock_response.reason = "Unavailable For Legal Reasons" mock_response.reason = "Unavailable For Legal Reasons"
def mock_get_response(request, auth, template): with patch(
return mock_response, [] "github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
with patch("github_backup.github_backup._get_response", side_effect=mock_get_response): ):
with pytest.raises(github_backup.RepositoryUnavailableError): with pytest.raises(github_backup.RepositoryUnavailableError):
list(github_backup.retrieve_data_gen(args, "https://api.github.com/repos/test/dmca/issues")) github_backup.retrieve_data(
args, "https://api.github.com/repos/test/dmca/issues"
def test_other_http_errors_unchanged(self): )
"""Other HTTP errors should still raise generic Exception."""
args = Mock()
args.as_app = False
args.token_fine = None
args.token_classic = None
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.throttle_limit = None
args.throttle_pause = 0
mock_response = Mock()
mock_response.getcode.return_value = 404
mock_response.read.return_value = b'{"message": "Not Found"}'
mock_response.headers = {"x-ratelimit-remaining": "5000"}
mock_response.reason = "Not Found"
def mock_get_response(request, auth, template):
return mock_response, []
with patch("github_backup.github_backup._get_response", side_effect=mock_get_response):
# Should raise generic Exception, not RepositoryUnavailableError
with pytest.raises(Exception) as exc_info:
list(github_backup.retrieve_data_gen(args, "https://api.github.com/repos/test/notfound/issues"))
assert not isinstance(exc_info.value, github_backup.RepositoryUnavailableError)
assert "404" in str(exc_info.value)
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -1,9 +1,7 @@
"""Tests for Link header pagination handling.""" """Tests for Link header pagination handling."""
import json import json
from unittest.mock import Mock, patch from unittest.mock import patch
import pytest
from github_backup import github_backup from github_backup import github_backup
@@ -38,24 +36,9 @@ class MockHTTPResponse:
return headers return headers
@pytest.fixture def test_cursor_based_pagination(create_args):
def mock_args():
"""Mock args for retrieve_data_gen."""
args = Mock()
args.as_app = False
args.token_fine = None
args.token_classic = "fake_token"
args.username = None
args.password = None
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
args.throttle_limit = None
args.throttle_pause = 0
return args
def test_cursor_based_pagination(mock_args):
"""Link header with 'after' cursor parameter works correctly.""" """Link header with 'after' cursor parameter works correctly."""
args = create_args(token_classic="fake_token")
# Simulate issues endpoint behavior: returns cursor in Link header # Simulate issues endpoint behavior: returns cursor in Link header
responses = [ responses = [
@@ -77,10 +60,8 @@ def test_cursor_based_pagination(mock_args):
return responses[len(requests_made) - 1] return responses[len(requests_made) - 1]
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen): with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
results = list( results = github_backup.retrieve_data(
github_backup.retrieve_data_gen( args, "https://api.github.com/repos/owner/repo/issues"
mock_args, "https://api.github.com/repos/owner/repo/issues"
)
) )
# Verify all items retrieved and cursor was used in second request # Verify all items retrieved and cursor was used in second request
@@ -89,8 +70,9 @@ def test_cursor_based_pagination(mock_args):
assert "after=ABC123" in requests_made[1] assert "after=ABC123" in requests_made[1]
def test_page_based_pagination(mock_args): def test_page_based_pagination(create_args):
"""Link header with 'page' parameter works correctly.""" """Link header with 'page' parameter works correctly."""
args = create_args(token_classic="fake_token")
# Simulate pulls/repos endpoint behavior: returns page numbers in Link header # Simulate pulls/repos endpoint behavior: returns page numbers in Link header
responses = [ responses = [
@@ -112,10 +94,8 @@ def test_page_based_pagination(mock_args):
return responses[len(requests_made) - 1] return responses[len(requests_made) - 1]
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen): with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
results = list( results = github_backup.retrieve_data(
github_backup.retrieve_data_gen( args, "https://api.github.com/repos/owner/repo/pulls"
mock_args, "https://api.github.com/repos/owner/repo/pulls"
)
) )
# Verify all items retrieved and page parameter was used (not cursor) # Verify all items retrieved and page parameter was used (not cursor)
@@ -125,8 +105,9 @@ def test_page_based_pagination(mock_args):
assert "after" not in requests_made[1] assert "after" not in requests_made[1]
def test_no_link_header_stops_pagination(mock_args): def test_no_link_header_stops_pagination(create_args):
"""Pagination stops when Link header is absent.""" """Pagination stops when Link header is absent."""
args = create_args(token_classic="fake_token")
# Simulate endpoint with results that fit in a single page # Simulate endpoint with results that fit in a single page
responses = [ responses = [
@@ -142,10 +123,8 @@ def test_no_link_header_stops_pagination(mock_args):
return responses[len(requests_made) - 1] return responses[len(requests_made) - 1]
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen): with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
results = list( results = github_backup.retrieve_data(
github_backup.retrieve_data_gen( args, "https://api.github.com/repos/owner/repo/labels"
mock_args, "https://api.github.com/repos/owner/repo/labels"
)
) )
# Verify pagination stopped after first request # Verify pagination stopped after first request

479
tests/test_retrieve_data.py Normal file
View File

@@ -0,0 +1,479 @@
"""Tests for retrieve_data function."""
import json
import socket
from unittest.mock import Mock, patch
from urllib.error import HTTPError, URLError
import pytest
from github_backup import github_backup
from github_backup.github_backup import (
calculate_retry_delay,
make_request_with_retry,
)
# Default retry count used in tests (matches argparse default)
# With max_retries=5, total attempts = 6 (1 initial + 5 retries)
DEFAULT_MAX_RETRIES = 5
class TestCalculateRetryDelay:
def test_respects_retry_after_header(self):
headers = {"retry-after": "30"}
assert calculate_retry_delay(0, headers) == 30
def test_respects_rate_limit_reset(self):
import time
import calendar
# Set reset time 60 seconds in the future
future_reset = calendar.timegm(time.gmtime()) + 60
headers = {"x-ratelimit-remaining": "0", "x-ratelimit-reset": str(future_reset)}
delay = calculate_retry_delay(0, headers)
# Should be approximately 60 seconds (with some tolerance for execution time)
assert 55 <= delay <= 65
def test_exponential_backoff(self):
delay_0 = calculate_retry_delay(0, {})
delay_1 = calculate_retry_delay(1, {})
delay_2 = calculate_retry_delay(2, {})
# Base delay is 1s, so delays should be roughly 1, 2, 4 (plus jitter)
assert 0.9 <= delay_0 <= 1.2 # ~1s + up to 10% jitter
assert 1.8 <= delay_1 <= 2.4 # ~2s + up to 10% jitter
assert 3.6 <= delay_2 <= 4.8 # ~4s + up to 10% jitter
def test_max_delay_cap(self):
# Very high attempt number should not exceed 120s + jitter
delay = calculate_retry_delay(100, {})
assert delay <= 120 * 1.1 # 120s max + 10% jitter
def test_minimum_rate_limit_delay(self):
import time
import calendar
# Set reset time in the past (already reset)
past_reset = calendar.timegm(time.gmtime()) - 100
headers = {"x-ratelimit-remaining": "0", "x-ratelimit-reset": str(past_reset)}
delay = calculate_retry_delay(0, headers)
# Should be minimum 10 seconds even if reset time is in past
assert delay >= 10
class TestRetrieveDataRetry:
"""Tests for retry behavior in retrieve_data."""
def test_json_parse_error_retries_and_fails(self, create_args):
"""HTTP 200 with invalid JSON should retry and eventually fail."""
args = create_args(token_classic="fake_token")
mock_response = Mock()
mock_response.getcode.return_value = 200
mock_response.read.return_value = b"not valid json {"
mock_response.headers = {"x-ratelimit-remaining": "5000"}
call_count = 0
def mock_make_request(*a, **kw):
nonlocal call_count
call_count += 1
return mock_response
with patch(
"github_backup.github_backup.make_request_with_retry",
side_effect=mock_make_request,
):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
): # No delay in tests
with pytest.raises(Exception) as exc_info:
github_backup.retrieve_data(
args, "https://api.github.com/repos/test/repo/issues"
)
assert "Failed to read response after" in str(exc_info.value)
assert (
call_count == DEFAULT_MAX_RETRIES + 1
) # 1 initial + 5 retries = 6 attempts
def test_json_parse_error_recovers_on_retry(self, create_args):
"""HTTP 200 with invalid JSON should succeed if retry returns valid JSON."""
args = create_args(token_classic="fake_token")
bad_response = Mock()
bad_response.getcode.return_value = 200
bad_response.read.return_value = b"not valid json {"
bad_response.headers = {"x-ratelimit-remaining": "5000"}
good_response = Mock()
good_response.getcode.return_value = 200
good_response.read.return_value = json.dumps([{"id": 1}]).encode("utf-8")
good_response.headers = {"x-ratelimit-remaining": "5000", "Link": ""}
responses = [bad_response, bad_response, good_response]
call_count = 0
def mock_make_request(*a, **kw):
nonlocal call_count
result = responses[call_count]
call_count += 1
return result
with patch(
"github_backup.github_backup.make_request_with_retry",
side_effect=mock_make_request,
):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
result = github_backup.retrieve_data(
args, "https://api.github.com/repos/test/repo/issues"
)
assert result == [{"id": 1}]
assert call_count == 3 # Failed twice, succeeded on third
def test_http_error_raises_exception(self, create_args):
"""Non-success HTTP status codes should raise Exception."""
args = create_args(token_classic="fake_token")
mock_response = Mock()
mock_response.getcode.return_value = 404
mock_response.read.return_value = b'{"message": "Not Found"}'
mock_response.headers = {"x-ratelimit-remaining": "5000"}
mock_response.reason = "Not Found"
with patch(
"github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
):
with pytest.raises(Exception) as exc_info:
github_backup.retrieve_data(
args, "https://api.github.com/repos/test/notfound/issues"
)
assert not isinstance(
exc_info.value, github_backup.RepositoryUnavailableError
)
assert "404" in str(exc_info.value)
class TestMakeRequestWithRetry:
"""Tests for HTTP error retry behavior in make_request_with_retry."""
def test_502_error_retries_and_succeeds(self):
"""HTTP 502 should retry and succeed if subsequent request works."""
good_response = Mock()
good_response.read.return_value = b'{"ok": true}'
call_count = 0
fail_count = DEFAULT_MAX_RETRIES # Fail all retries, succeed on last attempt
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count <= fail_count:
raise HTTPError(
url="https://api.github.com/test",
code=502,
msg="Bad Gateway",
hdrs={"x-ratelimit-remaining": "5000"},
fp=None,
)
return good_response
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
result = make_request_with_retry(Mock(), None)
assert result == good_response
assert (
call_count == DEFAULT_MAX_RETRIES + 1
) # 1 initial + 5 retries = 6 attempts
def test_503_error_retries_until_exhausted(self):
"""HTTP 503 should make 1 initial + DEFAULT_MAX_RETRIES retry attempts then raise."""
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
raise HTTPError(
url="https://api.github.com/test",
code=503,
msg="Service Unavailable",
hdrs={"x-ratelimit-remaining": "5000"},
fp=None,
)
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
with pytest.raises(HTTPError) as exc_info:
make_request_with_retry(Mock(), None)
assert exc_info.value.code == 503
assert (
call_count == DEFAULT_MAX_RETRIES + 1
) # 1 initial + 5 retries = 6 attempts
def test_404_error_not_retried(self):
"""HTTP 404 should not be retried - raise immediately."""
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
raise HTTPError(
url="https://api.github.com/test",
code=404,
msg="Not Found",
hdrs={"x-ratelimit-remaining": "5000"},
fp=None,
)
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with pytest.raises(HTTPError) as exc_info:
make_request_with_retry(Mock(), None)
assert exc_info.value.code == 404
assert call_count == 1 # No retries
def test_rate_limit_403_retried_when_remaining_zero(self):
"""HTTP 403 with x-ratelimit-remaining=0 should retry."""
good_response = Mock()
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
raise HTTPError(
url="https://api.github.com/test",
code=403,
msg="Forbidden",
hdrs={"x-ratelimit-remaining": "0"},
fp=None,
)
return good_response
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
result = make_request_with_retry(Mock(), None)
assert result == good_response
assert call_count == 2
def test_403_not_retried_when_remaining_nonzero(self):
"""HTTP 403 with x-ratelimit-remaining>0 should not retry (permission error)."""
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
raise HTTPError(
url="https://api.github.com/test",
code=403,
msg="Forbidden",
hdrs={"x-ratelimit-remaining": "5000"},
fp=None,
)
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with pytest.raises(HTTPError) as exc_info:
make_request_with_retry(Mock(), None)
assert exc_info.value.code == 403
assert call_count == 1 # No retries
def test_connection_error_retries_and_succeeds(self):
"""URLError (connection error) should retry and succeed if subsequent request works."""
good_response = Mock()
call_count = 0
fail_count = DEFAULT_MAX_RETRIES # Fail all retries, succeed on last attempt
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count <= fail_count:
raise URLError("Connection refused")
return good_response
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
result = make_request_with_retry(Mock(), None)
assert result == good_response
assert (
call_count == DEFAULT_MAX_RETRIES + 1
) # 1 initial + 5 retries = 6 attempts
def test_socket_error_retries_until_exhausted(self):
"""socket.error should make 1 initial + DEFAULT_MAX_RETRIES retry attempts then raise."""
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
raise socket.error("Connection reset by peer")
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
with pytest.raises(socket.error):
make_request_with_retry(Mock(), None)
assert (
call_count == DEFAULT_MAX_RETRIES + 1
) # 1 initial + 5 retries = 6 attempts
class TestRetrieveDataThrottling:
"""Tests for throttling behavior in retrieve_data."""
def test_throttling_pauses_when_rate_limit_low(self, create_args):
"""Should pause when x-ratelimit-remaining is at or below throttle_limit."""
args = create_args(
token_classic="fake_token",
throttle_limit=10,
throttle_pause=5,
)
mock_response = Mock()
mock_response.getcode.return_value = 200
mock_response.read.return_value = json.dumps([{"id": 1}]).encode("utf-8")
mock_response.headers = {
"x-ratelimit-remaining": "5",
"Link": "",
} # Below throttle_limit
with patch(
"github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
):
with patch("github_backup.github_backup.time.sleep") as mock_sleep:
github_backup.retrieve_data(
args, "https://api.github.com/repos/test/repo/issues"
)
mock_sleep.assert_called_once_with(5) # throttle_pause value
class TestRetrieveDataSingleItem:
"""Tests for single item (dict) responses in retrieve_data."""
def test_dict_response_returned_as_list(self, create_args):
"""Single dict response should be returned as a list with one item."""
args = create_args(token_classic="fake_token")
mock_response = Mock()
mock_response.getcode.return_value = 200
mock_response.read.return_value = json.dumps(
{"login": "testuser", "id": 123}
).encode("utf-8")
mock_response.headers = {"x-ratelimit-remaining": "5000", "Link": ""}
with patch(
"github_backup.github_backup.make_request_with_retry",
return_value=mock_response,
):
result = github_backup.retrieve_data(
args, "https://api.github.com/user"
)
assert result == [{"login": "testuser", "id": 123}]
class TestRetriesCliArgument:
"""Tests for --retries CLI argument validation and behavior."""
def test_retries_argument_accepted(self):
"""--retries flag should be accepted and parsed correctly."""
args = github_backup.parse_args(["--retries", "3", "testuser"])
assert args.max_retries == 3
def test_retries_default_value(self):
"""--retries should default to 5 if not specified."""
args = github_backup.parse_args(["testuser"])
assert args.max_retries == 5
def test_retries_zero_is_valid(self):
"""--retries 0 should be valid and mean 1 attempt (no retries)."""
args = github_backup.parse_args(["--retries", "0", "testuser"])
assert args.max_retries == 0
def test_retries_negative_rejected(self):
"""--retries with negative value should be rejected by argparse."""
with pytest.raises(SystemExit):
github_backup.parse_args(["--retries", "-1", "testuser"])
def test_retries_non_integer_rejected(self):
"""--retries with non-integer value should be rejected by argparse."""
with pytest.raises(SystemExit):
github_backup.parse_args(["--retries", "abc", "testuser"])
def test_retries_one_with_transient_error_succeeds(self):
"""--retries 1 should allow one retry after initial failure."""
good_response = Mock()
good_response.read.return_value = b'{"ok": true}'
call_count = 0
def mock_urlopen(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
raise HTTPError(
url="https://api.github.com/test",
code=502,
msg="Bad Gateway",
hdrs={"x-ratelimit-remaining": "5000"},
fp=None,
)
return good_response
with patch("github_backup.github_backup.urlopen", side_effect=mock_urlopen):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
result = make_request_with_retry(Mock(), None, max_retries=1)
assert result == good_response
assert call_count == 2 # 1 initial + 1 retry = 2 attempts
def test_custom_retry_count_limits_attempts(self, create_args):
"""Custom --retries value should limit actual retry attempts."""
args = create_args(
token_classic="fake_token",
max_retries=2, # 2 retries = 3 total attempts (1 initial + 2 retries)
)
mock_response = Mock()
mock_response.getcode.return_value = 200
mock_response.read.return_value = b"not valid json {"
mock_response.headers = {"x-ratelimit-remaining": "5000"}
call_count = 0
def mock_make_request(*args, **kwargs):
nonlocal call_count
call_count += 1
return mock_response
with patch(
"github_backup.github_backup.make_request_with_retry",
side_effect=mock_make_request,
):
with patch(
"github_backup.github_backup.calculate_retry_delay", return_value=0
):
with pytest.raises(Exception) as exc_info:
github_backup.retrieve_data(
args, "https://api.github.com/repos/test/repo/issues"
)
assert "Failed to read response after 3 attempts" in str(exc_info.value)
assert call_count == 3 # 1 initial + 2 retries = 3 attempts

View File

@@ -0,0 +1,272 @@
"""Tests for --skip-assets-on flag behavior (issue #135)."""
import pytest
from unittest.mock import patch
from github_backup import github_backup
class TestSkipAssetsOn:
"""Test suite for --skip-assets-on flag.
Issue #135: Allow skipping asset downloads for specific repositories
while still backing up release metadata.
"""
def _create_mock_repository(self, name="test-repo", owner="testuser"):
"""Create a mock repository object."""
return {
"name": name,
"full_name": f"{owner}/{name}",
"owner": {"login": owner},
"private": False,
"fork": False,
"has_wiki": False,
}
def _create_mock_release(self, tag="v1.0.0"):
"""Create a mock release object."""
return {
"tag_name": tag,
"name": tag,
"prerelease": False,
"draft": False,
"assets_url": f"https://api.github.com/repos/testuser/test-repo/releases/{tag}/assets",
}
def _create_mock_asset(self, name="asset.zip"):
"""Create a mock asset object."""
return {
"name": name,
"url": f"https://api.github.com/repos/testuser/test-repo/releases/assets/{name}",
}
class TestSkipAssetsOnArgumentParsing(TestSkipAssetsOn):
"""Tests for --skip-assets-on argument parsing."""
def test_skip_assets_on_not_set_defaults_to_none(self):
"""When --skip-assets-on is not specified, it should default to None."""
args = github_backup.parse_args(["testuser"])
assert args.skip_assets_on is None
def test_skip_assets_on_single_repo(self):
"""Single --skip-assets-on should create list with one item."""
args = github_backup.parse_args(["testuser", "--skip-assets-on", "big-repo"])
assert args.skip_assets_on == ["big-repo"]
def test_skip_assets_on_multiple_repos(self):
"""Multiple repos can be specified space-separated (like --exclude)."""
args = github_backup.parse_args(
[
"testuser",
"--skip-assets-on",
"big-repo",
"another-repo",
"owner/third-repo",
]
)
assert args.skip_assets_on == ["big-repo", "another-repo", "owner/third-repo"]
class TestSkipAssetsOnBehavior(TestSkipAssetsOn):
"""Tests for --skip-assets-on behavior in backup_releases."""
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_assets_downloaded_when_not_skipped(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Assets should be downloaded when repo is not in skip list."""
args = create_args(skip_assets_on=[])
repository = self._create_mock_repository(name="normal-repo")
release = self._create_mock_release()
asset = self._create_mock_asset()
mock_json_dump.return_value = True
mock_retrieve.side_effect = [
[release], # First call: get releases
[asset], # Second call: get assets
]
with patch("os.path.join", side_effect=lambda *args: "/".join(args)):
github_backup.backup_releases(
args,
"/tmp/backup/repositories/normal-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file should have been called for the asset
mock_download.assert_called_once()
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_assets_skipped_when_repo_name_matches(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Assets should be skipped when repo name is in skip list."""
args = create_args(skip_assets_on=["big-repo"])
repository = self._create_mock_repository(name="big-repo")
release = self._create_mock_release()
mock_json_dump.return_value = True
mock_retrieve.return_value = [release]
github_backup.backup_releases(
args,
"/tmp/backup/repositories/big-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file should NOT have been called
mock_download.assert_not_called()
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_assets_skipped_when_full_name_matches(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Assets should be skipped when owner/repo format matches."""
args = create_args(skip_assets_on=["otheruser/big-repo"])
repository = self._create_mock_repository(name="big-repo", owner="otheruser")
release = self._create_mock_release()
mock_json_dump.return_value = True
mock_retrieve.return_value = [release]
github_backup.backup_releases(
args,
"/tmp/backup/repositories/big-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file should NOT have been called
mock_download.assert_not_called()
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_case_insensitive_matching(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Skip matching should be case-insensitive."""
# User types uppercase, repo name is lowercase
args = create_args(skip_assets_on=["BIG-REPO"])
repository = self._create_mock_repository(name="big-repo")
release = self._create_mock_release()
mock_json_dump.return_value = True
mock_retrieve.return_value = [release]
github_backup.backup_releases(
args,
"/tmp/backup/repositories/big-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file should NOT have been called (case-insensitive match)
assert not mock_download.called
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_multiple_skip_repos(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Multiple repos in skip list should all be skipped."""
args = create_args(skip_assets_on=["repo1", "repo2", "repo3"])
repository = self._create_mock_repository(name="repo2")
release = self._create_mock_release()
mock_json_dump.return_value = True
mock_retrieve.return_value = [release]
github_backup.backup_releases(
args,
"/tmp/backup/repositories/repo2",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file should NOT have been called
mock_download.assert_not_called()
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_release_metadata_still_saved_when_assets_skipped(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Release JSON should still be saved even when assets are skipped."""
args = create_args(skip_assets_on=["big-repo"])
repository = self._create_mock_repository(name="big-repo")
release = self._create_mock_release()
mock_json_dump.return_value = True
mock_retrieve.return_value = [release]
github_backup.backup_releases(
args,
"/tmp/backup/repositories/big-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# json_dump_if_changed should have been called for release metadata
mock_json_dump.assert_called_once()
# But download_file should NOT have been called
mock_download.assert_not_called()
@patch("github_backup.github_backup.download_file")
@patch("github_backup.github_backup.retrieve_data")
@patch("github_backup.github_backup.mkdir_p")
@patch("github_backup.github_backup.json_dump_if_changed")
def test_non_matching_repo_still_downloads_assets(
self, mock_json_dump, mock_mkdir, mock_retrieve, mock_download, create_args
):
"""Repos not in skip list should still download assets."""
args = create_args(skip_assets_on=["other-repo"])
repository = self._create_mock_repository(name="normal-repo")
release = self._create_mock_release()
asset = self._create_mock_asset()
mock_json_dump.return_value = True
mock_retrieve.side_effect = [
[release], # First call: get releases
[asset], # Second call: get assets
]
with patch("os.path.join", side_effect=lambda *args: "/".join(args)):
github_backup.backup_releases(
args,
"/tmp/backup/repositories/normal-repo",
repository,
"https://api.github.com/repos/{owner}/{repo}",
include_assets=True,
)
# download_file SHOULD have been called
mock_download.assert_called_once()
if __name__ == "__main__":
pytest.main([__file__, "-v"])

View File

@@ -0,0 +1,201 @@
"""Tests for --starred-skip-size-over flag behavior (issue #108)."""
import pytest
from github_backup import github_backup
class TestStarredSkipSizeOverArgumentParsing:
"""Tests for --starred-skip-size-over argument parsing."""
def test_starred_skip_size_over_not_set_defaults_to_none(self):
"""When --starred-skip-size-over is not specified, it should default to None."""
args = github_backup.parse_args(["testuser"])
assert args.starred_skip_size_over is None
def test_starred_skip_size_over_accepts_integer(self):
"""--starred-skip-size-over should accept an integer value."""
args = github_backup.parse_args(["testuser", "--starred-skip-size-over", "500"])
assert args.starred_skip_size_over == 500
def test_starred_skip_size_over_rejects_non_integer(self):
"""--starred-skip-size-over should reject non-integer values."""
with pytest.raises(SystemExit):
github_backup.parse_args(["testuser", "--starred-skip-size-over", "abc"])
class TestStarredSkipSizeOverFiltering:
"""Tests for --starred-skip-size-over filtering behavior.
Issue #108: Allow restricting size of starred repositories before cloning.
The size is based on the GitHub API's 'size' field (in KB), but the CLI
argument accepts MB for user convenience.
"""
def test_starred_repo_under_limit_is_kept(self, create_args):
"""Starred repos under the size limit should be kept."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "small-repo",
"owner": {"login": "otheruser"},
"size": 100 * 1024, # 100 MB in KB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
assert result[0]["name"] == "small-repo"
def test_starred_repo_over_limit_is_filtered(self, create_args):
"""Starred repos over the size limit should be filtered out."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "huge-repo",
"owner": {"login": "otheruser"},
"size": 600 * 1024, # 600 MB in KB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 0
def test_own_repo_over_limit_is_kept(self, create_args):
"""User's own repos should not be affected by the size limit."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "my-huge-repo",
"owner": {"login": "testuser"},
"size": 600 * 1024, # 600 MB in KB
# No is_starred flag - this is the user's own repo
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
assert result[0]["name"] == "my-huge-repo"
def test_starred_repo_at_exact_limit_is_kept(self, create_args):
"""Starred repos at exactly the size limit should be kept."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "exact-limit-repo",
"owner": {"login": "otheruser"},
"size": 500 * 1024, # Exactly 500 MB in KB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
assert result[0]["name"] == "exact-limit-repo"
def test_mixed_repos_filtered_correctly(self, create_args):
"""Mix of own and starred repos should be filtered correctly."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "my-huge-repo",
"owner": {"login": "testuser"},
"size": 1000 * 1024, # 1 GB - own repo, should be kept
},
{
"name": "starred-small",
"owner": {"login": "otheruser"},
"size": 100 * 1024, # 100 MB - under limit
"is_starred": True,
},
{
"name": "starred-huge",
"owner": {"login": "anotheruser"},
"size": 2000 * 1024, # 2 GB - over limit
"is_starred": True,
},
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 2
names = [r["name"] for r in result]
assert "my-huge-repo" in names
assert "starred-small" in names
assert "starred-huge" not in names
def test_no_size_limit_keeps_all_starred(self, create_args):
"""When no size limit is set, all starred repos should be kept."""
args = create_args(starred_skip_size_over=None)
repos = [
{
"name": "huge-starred-repo",
"owner": {"login": "otheruser"},
"size": 10000 * 1024, # 10 GB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
def test_repo_without_size_field_is_kept(self, create_args):
"""Repos without a size field should be kept (size defaults to 0)."""
args = create_args(starred_skip_size_over=500)
repos = [
{
"name": "no-size-repo",
"owner": {"login": "otheruser"},
"is_starred": True,
# No size field
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
def test_zero_value_warns_and_is_ignored(self, create_args, caplog):
"""Zero value should warn and keep all repos."""
args = create_args(starred_skip_size_over=0)
repos = [
{
"name": "huge-starred-repo",
"owner": {"login": "otheruser"},
"size": 10000 * 1024, # 10 GB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
assert "must be greater than 0" in caplog.text
def test_negative_value_warns_and_is_ignored(self, create_args, caplog):
"""Negative value should warn and keep all repos."""
args = create_args(starred_skip_size_over=-5)
repos = [
{
"name": "huge-starred-repo",
"owner": {"login": "otheruser"},
"size": 10000 * 1024, # 10 GB
"is_starred": True,
}
]
result = github_backup.filter_repositories(args, repos)
assert len(result) == 1
assert "must be greater than 0" in caplog.text
if __name__ == "__main__":
pytest.main([__file__, "-v"])