Commit Graph

78 Commits

Author SHA1 Message Date
Rodos
5739ac0745 Avoid rewriting unchanged JSON files for labels, milestones, releases, hooks, followers, and following
This change reduces unnecessary writes when backing up metadata that changes
infrequently. The implementation compares existing file content before writing
and skips the write if the content is identical, preserving file timestamps.

Key changes:
- Added json_dump_if_changed() helper that compares content before writing
- Uses atomic writes (temp file + rename) for all metadata files
- NOT applied to issues/pulls (they use incremental_by_files logic)
- Made log messages consistent and past tense ("Saved" instead of "Saving")
- Added informative logging showing skip counts

Fixes #133
2025-11-29 17:21:14 +11:00
Rodos
7840528fe2 Skip DMCA'd repos which return a 451 response
Log a warning and the link to the DMCA notice. Continue backing up
other repositories instead of crashing.

Closes #163
2025-11-29 09:52:02 +11:00
Helio Machado
7551829677 Use cursor based pagination 2025-11-17 02:09:29 +01:00
Jose Diaz-Gonzalez
3eae9d78ed Merge pull request #447 from Iamrodos/master
fix: Improve CA certificate detection with fallback chain
2025-11-16 18:54:58 -05:00
Rodos
90ba839c7d fix: Improve CA certificate detection with fallback chain
The previous implementation incorrectly assumed empty get_ca_certs()
meant broken SSL, causing false failures in GitHub Codespaces and other
directory-based cert systems where certificates exist but aren't pre-loaded.
It would then attempt to import certifi as a workaround, but certifi wasn't
listed in requirements.txt, causing the fallback to fail with ImportError
even though the system certificates would have worked fine.

This commit replaces the naive check with a layered fallback approach that
checks multiple certificate sources. First it checks for pre-loaded system
certs (file-based systems). Then it verifies system cert paths exist
(directory-based systems like Ubuntu/Debian/Codespaces). Finally it attempts
to use certifi as an optional fallback only if needed.

This approach eliminates hard dependencies (certifi is now optional), works
in GitHub Codespaces without any setup, and fails gracefully with clear hints
for resolution when SSL is actually broken rather than failing with
ModuleNotFoundError.

Fixes #444
2025-11-16 16:33:10 +11:00
Rodos
7a9455db88 fix: Prevent duplicate attachment downloads
Fixes bug where attachments were downloaded multiple times with
incremented filenames (file.mov, file_1.mov, file_2.mov) when
running backups without --skip-existing flag.

I should not have used the --skip-existing flag for attachments,
it did not do what I thought it did.

The correct approach is to always use the manifest to guide what
has already been downloaded and what now needs to be done.
2025-11-14 10:28:30 +11:00
Rodos
1ed3d66777 refactor: Add atomic writes for attachment files and manifests 2025-11-06 12:42:57 +11:00
Rodos
a194fa48ce feat: Add attachment download support for issues and pull requests
Adds new --attachments flag that downloads user-uploaded files from
issue and PR bodies and comments. Key features:

- Determines attachment URLs
- Tracks downloads in manifest.json with metadata
- Supports --skip-existing to avoid re-downloading
- Handles filename collisions with counter suffix
- Smart retry logic for transient vs permanent failures
- Uses Content-Disposition for correct file extensions
2025-11-06 12:42:57 +11:00
Eric Wheeler
d820dd994d Fix -R flag to allow backups of repositories not owned by user
Previously, using -R flag would show zero issues/PRs for repositories
not owned by the primary user due to incorrect pagination parameters
being added to single repository API calls.

- Remove pagination parameters for single repository requests
- Support owner/repo format in -R flag (e.g., -R owner/repo-name)
- Skip filtering when specific repository is requested
- Fix URL construction for requests without query parameters

This enables backing up any repository, not just those owned by the
primary user specified in -u flag.
2025-07-19 17:28:52 -07:00
Jose Diaz-Gonzalez
3d354beb24 chore: fix inline comments 2025-02-21 22:14:37 -05:00
Honza Maly
20e4d385a5 Convert timestamp to string, although maybe the other way around would be better ... 2025-01-17 07:28:49 +00:00
Honza Maly
a49322cf7d Implementing incremental by files, safer version of incremental backup. 2025-01-16 21:00:02 +00:00
Jose Diaz-Gonzalez
0e0197149e chore: reformat file to fix lint issues 2025-01-03 20:07:40 -05:00
Jose Diaz-Gonzalez
22d8f8e649 Merge pull request #344 from xloem/https_ctx
Detect empty HTTPS contexts.
2024-12-09 09:46:04 -05:00
John Doe
53f6650f61 KeyError fix with gists 2024-09-21 21:38:23 -04:00
John Doe
548a2ec405 Detect empty HTTPS contexts.
Some users are relying solely on the certifi package to provide their CA certs, as requests does this by default.

This patch detects this situation and emits a clear warning as well as importing certifi to work around the situation..

Fixes #162 .
2024-09-21 20:50:54 -04:00
Louis Parisot
04c70ce277 git fetch is required even when using lfs 2024-09-10 11:00:17 +02:00
Jarl Totland
ba46cb87e8 fix: do not double encode auth when retrieving release assets 2024-07-24 10:43:30 +02:00
dale-primer-e
9812988a4a Remove trailing whitespaces
That are triggering flake.
2024-03-15 08:27:09 +13:00
dale-primer-e
1eccebcb83 Fix error with as_app flag 2024-03-14 15:27:22 +13:00
dale-primer-e
122eb56aa1 Fix error downloading assets
When downloading assets using a fine grained token you will get a "can't
concat str to bytes" error. This is due to the fine grained token being
concatenated onto bytes in the line:

`request.add_header("Authorization", "Basic ".encode("ascii") + auth)`

This is better handled in the function `_construct_request` so I changed
the lines that construct the request in `download_file` to use the
function `_construct_request` and updated the function signature to
reflect that.
2024-03-14 15:02:49 +13:00
Jose Diaz-Gonzalez
585af4c4e3 Merge pull request #236 from BrOleg5/master
Add options to include certain number of latest release and skip prereleases
2024-01-29 07:51:35 -05:00
Moritz Federspiel
42b836f623 fix: Catch 404s for non-existing hooks. Fixes #176
Explanation: Repositories where no webhooks are defined return 404 errors. This breaks further script execution.
2024-01-16 15:17:48 +01:00
BrOleg5
09f4168db6 Fix code style. 2024-01-05 20:46:02 +08:00
BrOleg5
3e9a4fa0d8 Update CLI help output in README.
Rename destination of latest-releases argument.
2024-01-05 20:35:37 +08:00
BrOleg5
ab18e96ea8 Add option to skip prerelease versions.
Replace release sorting by tag with release sorting by creation date.
2024-01-05 20:26:08 +08:00
BrOleg5
eb88def888 Add option to include certain number of the latest releases. 2024-01-02 12:54:06 +08:00
Jose Diaz-Gonzalez
2724f02b0a chore: reformat file and update flake8 2023-12-09 00:30:44 -05:00
Jose Diaz-Gonzalez
dbc1619106 Merge pull request #218 from Ondkloss/fix/win32logger
Suggested modification to fix win32 logging failure
2023-10-09 20:38:38 -04:00
Halvor Holsten Strand
7437e3abb1 Merge pull request, while keeping -q --quiet flag.
Most changes were already included, only adjusted with black formatting.
2023-10-09 12:01:32 +02:00
Halvor Holsten Strand
6f3be3d0e8 Suggested modification to fix win32 logging failure, due to local variable scope.
Logger does not appear to have any utility within "logging_subprocess".
2023-10-07 19:02:52 +02:00
Halvor Holsten Strand
b277baa6ea Update github_backup.py 2023-10-02 09:14:40 +02:00
Halvor Holsten Strand
15de769d67 Simplified one if/elif scenario.
Extracted file reading of another if/elif scenario.
2023-10-01 22:22:15 +02:00
Halvor Holsten Strand
a9d35c0fd5 Ran black. 2023-09-29 14:40:16 +02:00
Halvor Holsten Strand
20f5fd7a86 Merge branch 'master' into feature/fine_grained
# Conflicts:
#	README.rst
#	github_backup/github_backup.py
2023-09-29 14:34:06 +02:00
Halvor Holsten Strand
f12b877509 Keep backwards compatability by going back to "--token" for classic.
Allow "file://" uri for "--token-fine".
2023-09-29 14:01:53 +02:00
ZhymabekRoman
a8a583bed1 fix: minor cosmetic changes 2023-06-25 10:41:48 +06:00
ZhymabekRoman
68e718010f fix: add forgotten variable formatting 2023-06-25 10:39:16 +06:00
ZhymabekRoman
a06c3e9fd3 fix: refactor logging
Based on #195
2023-06-25 10:38:31 +06:00
Jose Diaz-Gonzalez
8b1bfd433c chore: formatting 2023-05-29 18:34:22 -04:00
Jose Diaz-Gonzalez
cca8a851ad chore: run black 2023-05-29 18:32:39 -04:00
josegonzalez
b5d749ec46 Merge pull request #199 from eht16/add_log_level_argument
Add --log-level command line argument
2023-05-29 18:29:41 -04:00
froggleston
60cb484a19 Add support for fine-grained tokens 2023-03-22 14:53:07 +00:00
Ken Bailey
fbb977acf4 Check both updated_at and pushed_at properties
Check both updated_at and pushed_at dates to get the last_update to reduce data retrieved on incremental api calls using since.
2023-02-28 15:44:14 -07:00
josegonzalez
dcc90b747a Merge pull request #204 from Assimila/exclude_repositories
Add option to exclude repositories
2022-11-28 00:23:20 -05:00
josegonzalez
f414fac108 Merge pull request #202 from TheOneric/pull_backup-regular-comments
Backup regular pull request comments as well
2022-11-28 00:23:08 -05:00
josegonzalez
38692bc836 Merge pull request #201 from TRAdEWORKS/fix-bug-request-url-error-forever-retry
Fix a bug forever retry when request url error
2022-11-28 00:22:54 -05:00
npounder
753a26d0d6 add option to exclude repositories 2022-11-25 12:35:24 +00:00
Oneric
b629a865f4 Backup regular pull request comments as well
Before, only review comments were backed up;
regular comments need to be fetched via issue API.

Fixes: https://github.com/josegonzalez/python-github-backup/issues/150
2022-07-12 18:38:11 +02:00
kornpisey
75ec773a6f fix bug forever retry when request url error 2022-05-30 13:50:23 +09:00