Commit Graph

220 Commits

Author SHA1 Message Date
Changaco
17b79fcbef rename a function to match what it actually does 2026-04-30 14:43:08 +00:00
Changaco
f1fca0f9b7 don't leave files open 2026-04-30 14:43:08 +00:00
Changaco
ccc27b95f7 remove legacy code in mkdir_p function 2026-04-30 14:43:08 +00:00
Changaco
f3eabf0bfe don't pass stdin when doing so can't do any good
When the child process doesn't inherit stderr, it can't ask the user for input, so it shouldn't inherit stdin either.
2026-04-30 14:38:42 +00:00
Changaco
b92aee6f11 use subprocess.DEVNULL instead of emulating it 2026-04-30 14:38:42 +00:00
Jose Diaz-Gonzalez
4d1772319f Merge pull request #508 from Changaco/security-improvements
Security-related changes
2026-04-30 10:19:19 -04:00
Jose Diaz-Gonzalez
2c7fdab54e Merge pull request #507 from Changaco/output-cleanup
Hide output of `git lfs version`
2026-04-30 10:19:02 -04:00
Changaco
2f130ecd66 remove bad invocation of the system shell 2026-04-30 12:24:26 +00:00
Changaco
ddf7f82e65 add missing context argument to urlopen call 2026-04-30 12:24:26 +00:00
Changaco
ddf82f1115 suppress output of call to git lfs version 2026-04-30 12:22:57 +00:00
Changaco
0638666bc7 handle more network errors
```python-traceback
Traceback (most recent call last):
  File ".local/bin/github-backup", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File ".local/share/pipx/venvs/github-backup/lib/python3.14/site-packages/github_backup/cli.py", line 83, in main
    backup_repositories(args, output_directory, repositories)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/pipx/venvs/github-backup/lib/python3.14/site-packages/github_backup/github_backup.py", line 1845, in backup_repositories
    backup_pulls(args, repo_cwd, repository, repos_template)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/pipx/venvs/github-backup/lib/python3.14/site-packages/github_backup/github_backup.py", line 2019, in backup_pulls
    pulls[number]["commit_data"] = retrieve_data(args, template)
                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File ".local/share/pipx/venvs/github-backup/lib/python3.14/site-packages/github_backup/github_backup.py", line 766, in retrieve_data
    return list(fetch_all())
  File ".local/share/pipx/venvs/github-backup/lib/python3.14/site-packages/github_backup/github_backup.py", line 717, in fetch_all
    response = json.loads(http_response.read().decode("utf-8"))
                          ~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/http/client.py", line 500, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.14/http/client.py", line 648, in _safe_read
    data = self.fp.read(cursize)
  File "/usr/lib/python3.14/socket.py", line 725, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.14/ssl.py", line 1304, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/ssl.py", line 1138, in read
    return self._sslobj.read(len, buffer)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
```
2026-04-29 12:39:13 +00:00
GitHub Action
f8cdf55050 Release version 0.62.0 2026-04-29 12:10:11 +00:00
Duncan Ogilvie
014eff395a Skip checkpoint-equal incremental items 2026-04-29 12:56:24 +02:00
Duncan Ogilvie
9d0cfdb61d Avoid redundant release asset list requests 2026-04-29 12:56:23 +02:00
Duncan Ogilvie
6cd0ab3633 Reduce unnecessary pull requests with incremental fetching 2026-04-29 12:56:23 +02:00
Duncan Ogilvie
b3a8241c9a Implement per-resource last_update timestamps
Closes #62
2026-04-29 12:06:33 +02:00
Duncan Ogilvie
24b3fdb4f3 Add support for pull request reviews
Closes #124
2026-04-29 11:43:30 +02:00
Duncan Ogilvie
4d022d94d0 Add support for discussions
Closes #290
2026-04-28 14:32:27 +02:00
Duncan Ogilvie
f4117990b2 Add --token-from-gh authentication option 2026-04-27 15:52:55 +02:00
GitHub Action
68af1d406a Release version 0.61.5 2026-02-18 21:04:32 +00:00
Rodos
f54a5458f6 Fix empty repository crash due to None timestamp comparison (#489)
Empty repositories have None for pushed_at/updated_at, causing a
TypeError when compared to the last_update string. Use .get() with
truthiness check to skip None timestamps in incremental tracking.
2026-02-18 20:10:48 +11:00
GitHub Action
60067650b0 Release version 0.61.4 2026-02-16 05:46:39 +00:00
Rodos
0162f7ed46 Fix HTTP 451 DMCA and 403 TOS handling regression (#487)
The DMCA handling added in PR #454 had a bug: make_request_with_retry()
raises HTTPError before retrieve_data() could check the status code via
getcode(), making the case 451 handler dead code. This also affected
HTTP 403 TOS violations (e.g. jumoog/MagiskOnWSA).

Fix by catching HTTPError in retrieve_data() and converting 451 and
blocked 403 responses (identified by "block" key in response body) to
RepositoryUnavailableError. Non-block 403s (permissions, scopes) still
propagate as HTTPError. Also handle RepositoryUnavailableError in
retrieve_repositories() for the --repository case.

Rewrote tests to mock urlopen (not make_request_with_retry) to exercise
the real code path that was previously untested.

Closes #487
2026-02-16 10:16:33 +11:00
GitHub Action
9be6282719 Release version 0.61.3 2026-01-24 05:45:42 +00:00
Rodos
0d8a504b02 Fix KeyError: 'Private' when using --all flag (#481)
The repository dictionary uses lowercase "private" key. Use .get() with
the correct case to match the pattern used elsewhere in the codebase.

The bug only affects --all users since --security-advisories short-circuits
before the key access.
2026-01-21 21:12:03 +11:00
GitHub Action
712d22d124 Release version 0.61.2 2026-01-19 17:40:27 +00:00
Lukas Bestle
856ad5db41 fix: Skip security advisories for private repos unless explicitly requested 2026-01-14 21:10:12 +01:00
Lukas Bestle
c6fa8c7695 feat: Only make security advisory dir if successful
Avoids empty directories for private repos
2026-01-14 21:02:51 +01:00
Lukas Bestle
93e505c07d fix: Handle 404 errors on security advisories 2026-01-14 21:01:59 +01:00
GitHub Action
6780d3ad6c Release version 0.61.1 2026-01-13 23:10:05 +00:00
Rodos
fce4abb74a Fix fine-grained PAT attachment downloads for private repos (#477)
Fine-grained personal access tokens cannot download attachments from
private repositories directly due to a GitHub platform limitation.

This adds a workaround for image attachments (/assets/ URLs) using
GitHub's Markdown API to convert URLs to JWT-signed URLs that can be
downloaded without authentication.

Changes:
- Add get_jwt_signed_url_via_markdown_api() function
- Detect fine-grained token + private repo + /assets/ URL upfront
- Use JWT workaround for those cases, mark success with jwt_workaround flag
- Skip download with skipped_at when workaround fails
- Add startup warning when using --attachments with fine-grained tokens
- Document limitation in README (file attachments still fail)
- Add 6 unit tests for JWT workaround logic
2026-01-13 13:15:38 +11:00
GitHub Action
c63fb37d30 Release version 0.61.0 2026-01-12 16:30:28 +00:00
Lukas Bestle
9a6f0b4c21 feat: Backup of repository security advisories 2026-01-09 21:04:21 +01:00
GitHub Action
858731ebbd Release version 0.60.0 2025-12-24 00:45:01 +00:00
michaelmartinez
44b0003ec9 updates to the tests, and fixes to the retry 2025-12-23 14:07:38 -08:00
michaelmartinez
5ab3852476 rm max_retries.py 2025-12-23 08:57:57 -08:00
michaelmartinez
f9827da342 don't use a global variable, pass the args instead 2025-12-23 08:53:54 -08:00
michaelmartinez
1f2ec016d5 readme, simplify the logic a bit 2025-12-22 16:13:12 -08:00
michaelmartinez
8b1b632d89 max_retries 5 2025-12-22 14:47:26 -08:00
michaelmartinez
89502c326d update retry logic and logging
### What
1. configureable retry count
2. additional logging

### Why
1. pass retry count as a command line arg; default 5
2. show details when api requests fail

### Testing before merge
compiles cleanly

### Validation after merge
compile and test

### Issue addressed by this PR
https://github.com/stellar/ops/issues/2039
2025-12-22 14:23:02 -08:00
GitHub Action
81a72ac8af Release version 0.59.0 2025-12-21 23:48:36 +00:00
Rodos
3c43e0f481 Add --starred-skip-size-over flag to limit starred repo size (#108)
Allow users to skip starred repositories exceeding a size threshold
when using --all-starred. Size is specified in MB and checked against
the GitHub API's repository size field.

- Only affects starred repos; user's own repos always included
- Logs each skipped repo with name and size

Closes #108
2025-12-21 22:18:09 +11:00
Rodos
db36c3c137 chore: remove deprecated -u/-p password authentication options 2025-12-20 19:16:11 +11:00
GitHub Action
c70cc43f57 Release version 0.58.0 2025-12-16 15:17:23 +00:00
Rodos
46140b0ff1 Fix retry logic for HTTP 5xx errors and network failures
Refactors error handling to retry all 5xx errors (not just 502), network errors (URLError, socket.error, IncompleteRead), and JSON parse errors with exponential backoff and jitter. Respects retry-after and rate limit headers per GitHub API requirements. Consolidates retry logic into make_request_with_retry() wrapper and adds clear logging for retry attempts and failures. Removes dead code from 2016 (errors list, _request_http_error, _request_url_error) that was intentionally disabled in commit 1e5a9048 to fix #29.

Fixes #140, #110, #138
2025-12-16 21:55:47 +11:00
GitHub Action
ba852b5830 Release version 0.57.0 2025-12-12 11:07:14 +00:00
Jose Diaz-Gonzalez
934ee4b14b Merge pull request #467 from Iamrodos/docs/187-189-auth-docs
Add GitHub Apps documentation and stdin token example
2025-12-12 06:06:30 -05:00
Rodos
f6e2f40b09 Add --skip-assets-on flag to skip release asset downloads (#135)
Allow users to skip downloading release assets for specific repositories
while still backing up release metadata. Useful for starred repos with
large assets (e.g. syncthing with 27GB+).

Usage: --skip-assets-on repo1 repo2 owner/repo3

Features:
- Space-separated repos (consistent with --exclude)
- Case-insensitive matching
- Supports both repo name and owner/repo format
2025-12-12 16:21:52 +11:00
Rodos
ef990483e2 Add GitHub Apps documentation and remove outdated header
- Add GitHub Apps authentication section with setup steps
  and CI/CD workflow example using actions/create-github-app-token
- Remove outdated machine-man-preview header (graduated 2020)

Closes #189
2025-12-12 10:25:49 +11:00
GitHub Action
2bb83d6d8b Release version 0.56.0 2025-12-11 16:50:28 +00:00