mirror of
https://github.com/josegonzalez/python-github-backup.git
synced 2025-12-11 18:41:11 +01:00
Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2bb83d6d8b | ||
|
|
8fcc142621 | ||
|
|
7615ce6102 | ||
|
|
3f1ef821c3 | ||
|
|
3684756eaa | ||
|
|
e745b55755 | ||
|
|
75e6f56773 | ||
|
|
b991c363a0 | ||
|
|
6d74af9126 | ||
|
|
381d67af96 | ||
|
|
2fbe8d272c | ||
|
|
eb5779ac23 | ||
|
|
5b52931ebf | ||
|
|
1d6d474408 | ||
|
|
b80049e96e | ||
|
|
58ad1c2378 | ||
|
|
6e2a7e521c | ||
|
|
aba048a3e9 | ||
|
|
9f7c08166f | ||
|
|
fdfaaec1ba | ||
|
|
8f9cf7ff89 | ||
|
|
899ab5fdc2 |
135
CHANGES.rst
135
CHANGES.rst
@@ -1,10 +1,143 @@
|
|||||||
Changelog
|
Changelog
|
||||||
=========
|
=========
|
||||||
|
|
||||||
0.54.0 (2025-12-03)
|
0.56.0 (2025-12-11)
|
||||||
-------------------
|
-------------------
|
||||||
------------------------
|
------------------------
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Replace deprecated git lfs clone with git clone + git lfs fetch --all.
|
||||||
|
[Rodos]
|
||||||
|
|
||||||
|
git lfs clone is deprecated - modern git clone handles LFS automatically.
|
||||||
|
Using git lfs fetch --all ensures all LFS objects across all refs are
|
||||||
|
backed up, matching the existing bare clone behavior and providing
|
||||||
|
complete LFS backups.
|
||||||
|
|
||||||
|
Closes #379
|
||||||
|
- Add Windows support with entry_points and os.replace. [Rodos]
|
||||||
|
|
||||||
|
- Replace os.rename() with os.replace() for atomic file operations
|
||||||
|
on Windows (os.rename fails if destination exists on Windows)
|
||||||
|
- Add entry_points console_scripts for proper .exe generation on Windows
|
||||||
|
- Create github_backup/cli.py with main() entry point
|
||||||
|
- Add github_backup/__main__.py for python -m github_backup support
|
||||||
|
- Keep bin/github-backup as thin wrapper for backwards compatibility
|
||||||
|
|
||||||
|
Closes #112
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Docs: add "Restoring from Backup" section to README. [Rodos]
|
||||||
|
|
||||||
|
Clarifies that this tool is backup-only with no inbuilt restore.
|
||||||
|
Documents that git repos can be pushed back, but issues/PRs have
|
||||||
|
GitHub API limitations affecting all backup tools.
|
||||||
|
|
||||||
|
Closes #246
|
||||||
|
- Chore(deps): bump urllib3 in the python-packages group.
|
||||||
|
[dependabot[bot]]
|
||||||
|
|
||||||
|
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).
|
||||||
|
|
||||||
|
|
||||||
|
Updates `urllib3` from 2.6.0 to 2.6.1
|
||||||
|
- [Release notes](https://github.com/urllib3/urllib3/releases)
|
||||||
|
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
|
||||||
|
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.1)
|
||||||
|
|
||||||
|
---
|
||||||
|
updated-dependencies:
|
||||||
|
- dependency-name: urllib3
|
||||||
|
dependency-version: 2.6.1
|
||||||
|
dependency-type: direct:production
|
||||||
|
update-type: version-update:semver-patch
|
||||||
|
dependency-group: python-packages
|
||||||
|
...
|
||||||
|
- Chore(deps): bump the python-packages group with 3 updates.
|
||||||
|
[dependabot[bot]]
|
||||||
|
|
||||||
|
Bumps the python-packages group with 3 updates: [black](https://github.com/psf/black), [pytest](https://github.com/pytest-dev/pytest) and [platformdirs](https://github.com/tox-dev/platformdirs).
|
||||||
|
|
||||||
|
|
||||||
|
Updates `black` from 25.11.0 to 25.12.0
|
||||||
|
- [Release notes](https://github.com/psf/black/releases)
|
||||||
|
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
|
||||||
|
- [Commits](https://github.com/psf/black/compare/25.11.0...25.12.0)
|
||||||
|
|
||||||
|
Updates `pytest` from 9.0.1 to 9.0.2
|
||||||
|
- [Release notes](https://github.com/pytest-dev/pytest/releases)
|
||||||
|
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
|
||||||
|
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.1...9.0.2)
|
||||||
|
|
||||||
|
Updates `platformdirs` from 4.5.0 to 4.5.1
|
||||||
|
- [Release notes](https://github.com/tox-dev/platformdirs/releases)
|
||||||
|
- [Changelog](https://github.com/tox-dev/platformdirs/blob/main/CHANGES.rst)
|
||||||
|
- [Commits](https://github.com/tox-dev/platformdirs/compare/4.5.0...4.5.1)
|
||||||
|
|
||||||
|
---
|
||||||
|
updated-dependencies:
|
||||||
|
- dependency-name: black
|
||||||
|
dependency-version: 25.12.0
|
||||||
|
dependency-type: direct:production
|
||||||
|
update-type: version-update:semver-minor
|
||||||
|
dependency-group: python-packages
|
||||||
|
- dependency-name: pytest
|
||||||
|
dependency-version: 9.0.2
|
||||||
|
dependency-type: direct:production
|
||||||
|
update-type: version-update:semver-patch
|
||||||
|
dependency-group: python-packages
|
||||||
|
- dependency-name: platformdirs
|
||||||
|
dependency-version: 4.5.1
|
||||||
|
dependency-type: direct:production
|
||||||
|
update-type: version-update:semver-patch
|
||||||
|
dependency-group: python-packages
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
|
0.55.0 (2025-12-07)
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Improve error messages for inaccessible repos and empty wikis. [Rodos]
|
||||||
|
- --all-starred now clones repos without --repositories. [Rodos]
|
||||||
|
- Warn when --private used without authentication. [Rodos]
|
||||||
|
- Warn and skip when --starred-gists used for different user. [Rodos]
|
||||||
|
|
||||||
|
GitHub's API only allows retrieving starred gists for the authenticated
|
||||||
|
user. Previously, using --starred-gists when backing up a different user
|
||||||
|
would silently return no relevant data.
|
||||||
|
|
||||||
|
Now warns and skips the retrieval entirely when the target user differs
|
||||||
|
from the authenticated user. Uses case-insensitive comparison to match
|
||||||
|
GitHub's username handling.
|
||||||
|
|
||||||
|
Fixes #93
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Test: add missing test coverage for case sensitivity fix. [Rodos]
|
||||||
|
- Docs: fix RST formatting in Known blocking errors section. [Rodos]
|
||||||
|
- Chore(deps): bump urllib3 from 2.5.0 to 2.6.0. [dependabot[bot]]
|
||||||
|
|
||||||
|
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
|
||||||
|
- [Release notes](https://github.com/urllib3/urllib3/releases)
|
||||||
|
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
|
||||||
|
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)
|
||||||
|
|
||||||
|
---
|
||||||
|
updated-dependencies:
|
||||||
|
- dependency-name: urllib3
|
||||||
|
dependency-version: 2.6.0
|
||||||
|
dependency-type: direct:production
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
|
0.54.0 (2025-12-03)
|
||||||
|
-------------------
|
||||||
|
|
||||||
Fix
|
Fix
|
||||||
~~~
|
~~~
|
||||||
- Send INFO/DEBUG to stdout, WARNING/ERROR to stderr. [Rodos]
|
- Send INFO/DEBUG to stdout, WARNING/ERROR to stderr. [Rodos]
|
||||||
|
|||||||
25
README.rst
25
README.rst
@@ -215,6 +215,8 @@ When you use the ``--lfs`` option, you will need to make sure you have Git LFS i
|
|||||||
|
|
||||||
Instructions on how to do this can be found on https://git-lfs.github.com.
|
Instructions on how to do this can be found on https://git-lfs.github.com.
|
||||||
|
|
||||||
|
LFS objects are fetched for all refs, not just the current checkout, ensuring a complete backup of all LFS content across all branches and history.
|
||||||
|
|
||||||
|
|
||||||
About Attachments
|
About Attachments
|
||||||
-----------------
|
-----------------
|
||||||
@@ -281,7 +283,7 @@ If the incremental argument is used, this will result in the next backup only re
|
|||||||
|
|
||||||
It's therefore recommended to only use the incremental argument if the output/result is being actively monitored, or complimented with periodic full non-incremental runs, to avoid unexpected missing data in a regular backup runs.
|
It's therefore recommended to only use the incremental argument if the output/result is being actively monitored, or complimented with periodic full non-incremental runs, to avoid unexpected missing data in a regular backup runs.
|
||||||
|
|
||||||
1. **Starred public repo hooks blocking**
|
**Starred public repo hooks blocking**
|
||||||
|
|
||||||
Since the ``--all`` argument includes ``--hooks``, if you use ``--all`` and ``--all-starred`` together to clone a users starred public repositories, the backup will likely error and block the backup continuing.
|
Since the ``--all`` argument includes ``--hooks``, if you use ``--all`` and ``--all-starred`` together to clone a users starred public repositories, the backup will likely error and block the backup continuing.
|
||||||
|
|
||||||
@@ -301,6 +303,8 @@ Starred gists vs starred repo behaviour
|
|||||||
|
|
||||||
The starred normal repo cloning (``--all-starred``) argument stores starred repos separately to the users own repositories. However, using ``--starred-gists`` will store starred gists within the same directory as the users own gists ``--gists``. Also, all gist repo directory names are IDs not the gist's name.
|
The starred normal repo cloning (``--all-starred``) argument stores starred repos separately to the users own repositories. However, using ``--starred-gists`` will store starred gists within the same directory as the users own gists ``--gists``. Also, all gist repo directory names are IDs not the gist's name.
|
||||||
|
|
||||||
|
Note: ``--starred-gists`` only retrieves starred gists for the authenticated user, not the target user, due to a GitHub API limitation.
|
||||||
|
|
||||||
|
|
||||||
Skip existing on incomplete backups
|
Skip existing on incomplete backups
|
||||||
-----------------------------------
|
-----------------------------------
|
||||||
@@ -358,6 +362,25 @@ Debug an error/block or incomplete backup into a temporary directory. Omit "incr
|
|||||||
github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
|
github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
|
||||||
|
|
||||||
|
|
||||||
|
Restoring from Backup
|
||||||
|
=====================
|
||||||
|
|
||||||
|
This tool creates backups only, there is no inbuilt restore command.
|
||||||
|
|
||||||
|
**Git repositories, wikis, and gists** can be restored by pushing them back to GitHub as you would any git repository. For example, to restore a bare repository backup::
|
||||||
|
|
||||||
|
cd /tmp/white-house/repositories/petitions/repository
|
||||||
|
git push --mirror git@github.com:WhiteHouse/petitions.git
|
||||||
|
|
||||||
|
**Issues, pull requests, comments, and other metadata** are saved as JSON files for archival purposes. The GitHub API does not support recreating this data faithfully, creating issues via the API has limitations:
|
||||||
|
|
||||||
|
- New issue/PR numbers are assigned (original numbers cannot be set)
|
||||||
|
- Timestamps reflect creation time (original dates cannot be set)
|
||||||
|
- The API caller becomes the author (original authors cannot be set)
|
||||||
|
- Cross-references between issues and PRs will break
|
||||||
|
|
||||||
|
These are GitHub API limitations that affect all backup and migration tools, not just this one. Recreating issues with these limitations via the GitHub API is an exercise for the reader. The JSON backups remain useful for searching, auditing, or manual reference.
|
||||||
|
|
||||||
|
|
||||||
Development
|
Development
|
||||||
===========
|
===========
|
||||||
|
|||||||
@@ -1,69 +1,18 @@
|
|||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Backwards-compatible wrapper script.
|
||||||
|
|
||||||
|
The recommended way to run github-backup is via the installed command
|
||||||
|
(pip install github-backup) or python -m github_backup.
|
||||||
|
|
||||||
|
This script is kept for backwards compatibility with existing installations
|
||||||
|
that may reference this path directly.
|
||||||
|
"""
|
||||||
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
from github_backup.github_backup import (
|
from github_backup.cli import main
|
||||||
backup_account,
|
from github_backup.github_backup import logger
|
||||||
backup_repositories,
|
|
||||||
check_git_lfs_install,
|
|
||||||
filter_repositories,
|
|
||||||
get_authenticated_user,
|
|
||||||
logger,
|
|
||||||
mkdir_p,
|
|
||||||
parse_args,
|
|
||||||
retrieve_repositories,
|
|
||||||
)
|
|
||||||
|
|
||||||
# INFO and DEBUG go to stdout, WARNING and above go to stderr
|
|
||||||
log_format = logging.Formatter(
|
|
||||||
fmt="%(asctime)s.%(msecs)03d: %(message)s",
|
|
||||||
datefmt="%Y-%m-%dT%H:%M:%S",
|
|
||||||
)
|
|
||||||
|
|
||||||
stdout_handler = logging.StreamHandler(sys.stdout)
|
|
||||||
stdout_handler.setLevel(logging.DEBUG)
|
|
||||||
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
|
|
||||||
stdout_handler.setFormatter(log_format)
|
|
||||||
|
|
||||||
stderr_handler = logging.StreamHandler(sys.stderr)
|
|
||||||
stderr_handler.setLevel(logging.WARNING)
|
|
||||||
stderr_handler.setFormatter(log_format)
|
|
||||||
|
|
||||||
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
args = parse_args()
|
|
||||||
|
|
||||||
if args.quiet:
|
|
||||||
logger.setLevel(logging.WARNING)
|
|
||||||
|
|
||||||
output_directory = os.path.realpath(args.output_directory)
|
|
||||||
if not os.path.isdir(output_directory):
|
|
||||||
logger.info("Create output directory {0}".format(output_directory))
|
|
||||||
mkdir_p(output_directory)
|
|
||||||
|
|
||||||
if args.lfs_clone:
|
|
||||||
check_git_lfs_install()
|
|
||||||
|
|
||||||
if args.log_level:
|
|
||||||
log_level = logging.getLevelName(args.log_level.upper())
|
|
||||||
if isinstance(log_level, int):
|
|
||||||
logger.root.setLevel(log_level)
|
|
||||||
|
|
||||||
if not args.as_app:
|
|
||||||
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
|
|
||||||
authenticated_user = get_authenticated_user(args)
|
|
||||||
else:
|
|
||||||
authenticated_user = {"login": None}
|
|
||||||
|
|
||||||
repositories = retrieve_repositories(args, authenticated_user)
|
|
||||||
repositories = filter_repositories(args, repositories)
|
|
||||||
backup_repositories(args, output_directory, repositories)
|
|
||||||
backup_account(args, output_directory)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
__version__ = "0.54.0"
|
__version__ = "0.56.0"
|
||||||
|
|||||||
13
github_backup/__main__.py
Normal file
13
github_backup/__main__.py
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
"""Allow running as: python -m github_backup"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from github_backup.cli import main
|
||||||
|
from github_backup.github_backup import logger
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(str(e))
|
||||||
|
sys.exit(1)
|
||||||
82
github_backup/cli.py
Normal file
82
github_backup/cli.py
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Command-line interface for github-backup."""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from github_backup.github_backup import (
|
||||||
|
backup_account,
|
||||||
|
backup_repositories,
|
||||||
|
check_git_lfs_install,
|
||||||
|
filter_repositories,
|
||||||
|
get_auth,
|
||||||
|
get_authenticated_user,
|
||||||
|
logger,
|
||||||
|
mkdir_p,
|
||||||
|
parse_args,
|
||||||
|
retrieve_repositories,
|
||||||
|
)
|
||||||
|
|
||||||
|
# INFO and DEBUG go to stdout, WARNING and above go to stderr
|
||||||
|
log_format = logging.Formatter(
|
||||||
|
fmt="%(asctime)s.%(msecs)03d: %(message)s",
|
||||||
|
datefmt="%Y-%m-%dT%H:%M:%S",
|
||||||
|
)
|
||||||
|
|
||||||
|
stdout_handler = logging.StreamHandler(sys.stdout)
|
||||||
|
stdout_handler.setLevel(logging.DEBUG)
|
||||||
|
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
|
||||||
|
stdout_handler.setFormatter(log_format)
|
||||||
|
|
||||||
|
stderr_handler = logging.StreamHandler(sys.stderr)
|
||||||
|
stderr_handler.setLevel(logging.WARNING)
|
||||||
|
stderr_handler.setFormatter(log_format)
|
||||||
|
|
||||||
|
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for github-backup CLI."""
|
||||||
|
args = parse_args()
|
||||||
|
|
||||||
|
if args.private and not get_auth(args):
|
||||||
|
logger.warning(
|
||||||
|
"The --private flag has no effect without authentication. "
|
||||||
|
"Use -t/--token, -f/--token-fine, or -u/--username to authenticate."
|
||||||
|
)
|
||||||
|
|
||||||
|
if args.quiet:
|
||||||
|
logger.setLevel(logging.WARNING)
|
||||||
|
|
||||||
|
output_directory = os.path.realpath(args.output_directory)
|
||||||
|
if not os.path.isdir(output_directory):
|
||||||
|
logger.info("Create output directory {0}".format(output_directory))
|
||||||
|
mkdir_p(output_directory)
|
||||||
|
|
||||||
|
if args.lfs_clone:
|
||||||
|
check_git_lfs_install()
|
||||||
|
|
||||||
|
if args.log_level:
|
||||||
|
log_level = logging.getLevelName(args.log_level.upper())
|
||||||
|
if isinstance(log_level, int):
|
||||||
|
logger.root.setLevel(log_level)
|
||||||
|
|
||||||
|
if not args.as_app:
|
||||||
|
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
|
||||||
|
authenticated_user = get_authenticated_user(args)
|
||||||
|
else:
|
||||||
|
authenticated_user = {"login": None}
|
||||||
|
|
||||||
|
repositories = retrieve_repositories(args, authenticated_user)
|
||||||
|
repositories = filter_repositories(args, repositories)
|
||||||
|
backup_repositories(args, output_directory, repositories)
|
||||||
|
backup_account(args, output_directory)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(str(e))
|
||||||
|
sys.exit(1)
|
||||||
@@ -1038,7 +1038,7 @@ def download_attachment_file(url, path, auth, as_app=False, fine=False):
|
|||||||
bytes_downloaded += len(chunk)
|
bytes_downloaded += len(chunk)
|
||||||
|
|
||||||
# Atomic rename to final location
|
# Atomic rename to final location
|
||||||
os.rename(temp_path, path)
|
os.replace(temp_path, path)
|
||||||
|
|
||||||
metadata["size_bytes"] = bytes_downloaded
|
metadata["size_bytes"] = bytes_downloaded
|
||||||
metadata["success"] = True
|
metadata["success"] = True
|
||||||
@@ -1459,7 +1459,7 @@ def download_attachments(
|
|||||||
|
|
||||||
# Rename to add extension (already atomic from download)
|
# Rename to add extension (already atomic from download)
|
||||||
try:
|
try:
|
||||||
os.rename(filepath, final_filepath)
|
os.replace(filepath, final_filepath)
|
||||||
metadata["saved_as"] = os.path.basename(final_filepath)
|
metadata["saved_as"] = os.path.basename(final_filepath)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
@@ -1490,7 +1490,7 @@ def download_attachments(
|
|||||||
manifest_path = os.path.join(attachments_dir, "manifest.json")
|
manifest_path = os.path.join(attachments_dir, "manifest.json")
|
||||||
with open(manifest_path + ".temp", "w") as f:
|
with open(manifest_path + ".temp", "w") as f:
|
||||||
json.dump(manifest, f, indent=2)
|
json.dump(manifest, f, indent=2)
|
||||||
os.rename(manifest_path + ".temp", manifest_path) # Atomic write
|
os.replace(manifest_path + ".temp", manifest_path) # Atomic write
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"Wrote manifest for {0} #{1}: {2} attachments".format(
|
"Wrote manifest for {0} #{1}: {2} attachments".format(
|
||||||
item_type_display, number, len(attachment_metadata_list)
|
item_type_display, number, len(attachment_metadata_list)
|
||||||
@@ -1565,6 +1565,12 @@ def retrieve_repositories(args, authenticated_user):
|
|||||||
repos.extend(gists)
|
repos.extend(gists)
|
||||||
|
|
||||||
if args.include_starred_gists:
|
if args.include_starred_gists:
|
||||||
|
if not authenticated_user.get("login") or args.user.lower() != authenticated_user["login"].lower():
|
||||||
|
logger.warning(
|
||||||
|
"Cannot retrieve starred gists for '%s'. GitHub only allows access to the authenticated user's starred gists.",
|
||||||
|
args.user,
|
||||||
|
)
|
||||||
|
else:
|
||||||
starred_gists_template = "https://{0}/gists/starred".format(
|
starred_gists_template = "https://{0}/gists/starred".format(
|
||||||
get_github_api_host(args)
|
get_github_api_host(args)
|
||||||
)
|
)
|
||||||
@@ -1666,9 +1672,10 @@ def backup_repositories(args, output_directory, repositories):
|
|||||||
repo_url = get_github_repo_url(args, repository)
|
repo_url = get_github_repo_url(args, repository)
|
||||||
|
|
||||||
include_gists = args.include_gists or args.include_starred_gists
|
include_gists = args.include_gists or args.include_starred_gists
|
||||||
|
include_starred = args.all_starred and repository.get("is_starred")
|
||||||
if (args.include_repository or args.include_everything) or (
|
if (args.include_repository or args.include_everything) or (
|
||||||
include_gists and repository.get("is_gist")
|
include_gists and repository.get("is_gist")
|
||||||
):
|
) or include_starred:
|
||||||
repo_name = (
|
repo_name = (
|
||||||
repository.get("name")
|
repository.get("name")
|
||||||
if not repository.get("is_gist")
|
if not repository.get("is_gist")
|
||||||
@@ -1804,7 +1811,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
|
|||||||
|
|
||||||
with codecs.open(issue_file + ".temp", "w", encoding="utf-8") as f:
|
with codecs.open(issue_file + ".temp", "w", encoding="utf-8") as f:
|
||||||
json_dump(issue, f)
|
json_dump(issue, f)
|
||||||
os.rename(issue_file + ".temp", issue_file) # Unlike json_dump, this is atomic
|
os.replace(issue_file + ".temp", issue_file) # Atomic write
|
||||||
|
|
||||||
|
|
||||||
def backup_pulls(args, repo_cwd, repository, repos_template):
|
def backup_pulls(args, repo_cwd, repository, repos_template):
|
||||||
@@ -1879,7 +1886,7 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
|
|||||||
|
|
||||||
with codecs.open(pull_file + ".temp", "w", encoding="utf-8") as f:
|
with codecs.open(pull_file + ".temp", "w", encoding="utf-8") as f:
|
||||||
json_dump(pull, f)
|
json_dump(pull, f)
|
||||||
os.rename(pull_file + ".temp", pull_file) # Unlike json_dump, this is atomic
|
os.replace(pull_file + ".temp", pull_file) # Atomic write
|
||||||
|
|
||||||
|
|
||||||
def backup_milestones(args, repo_cwd, repository, repos_template):
|
def backup_milestones(args, repo_cwd, repository, repos_template):
|
||||||
@@ -2017,12 +2024,9 @@ def fetch_repository(
|
|||||||
):
|
):
|
||||||
if bare_clone:
|
if bare_clone:
|
||||||
if os.path.exists(local_dir):
|
if os.path.exists(local_dir):
|
||||||
clone_exists = (
|
clone_exists = subprocess.check_output(
|
||||||
subprocess.check_output(
|
|
||||||
["git", "rev-parse", "--is-bare-repository"], cwd=local_dir
|
["git", "rev-parse", "--is-bare-repository"], cwd=local_dir
|
||||||
)
|
) == b"true\n"
|
||||||
== b"true\n"
|
|
||||||
)
|
|
||||||
else:
|
else:
|
||||||
clone_exists = False
|
clone_exists = False
|
||||||
else:
|
else:
|
||||||
@@ -2037,10 +2041,13 @@ def fetch_repository(
|
|||||||
"git ls-remote " + remote_url, stdout=FNULL, stderr=FNULL, shell=True
|
"git ls-remote " + remote_url, stdout=FNULL, stderr=FNULL, shell=True
|
||||||
)
|
)
|
||||||
if initialized == 128:
|
if initialized == 128:
|
||||||
|
if ".wiki.git" in remote_url:
|
||||||
logger.info(
|
logger.info(
|
||||||
"Skipping {0} ({1}) since it's not initialized".format(
|
"Skipping {0} wiki (wiki is enabled but has no content)".format(name)
|
||||||
name, masked_remote_url
|
|
||||||
)
|
)
|
||||||
|
else:
|
||||||
|
logger.info(
|
||||||
|
"Skipping {0} (repository not accessible - may be empty, private, or credentials invalid)".format(name)
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -2082,12 +2089,14 @@ def fetch_repository(
|
|||||||
if no_prune:
|
if no_prune:
|
||||||
git_command.pop()
|
git_command.pop()
|
||||||
logging_subprocess(git_command, cwd=local_dir)
|
logging_subprocess(git_command, cwd=local_dir)
|
||||||
else:
|
|
||||||
if lfs_clone:
|
|
||||||
git_command = ["git", "lfs", "clone", remote_url, local_dir]
|
|
||||||
else:
|
else:
|
||||||
git_command = ["git", "clone", remote_url, local_dir]
|
git_command = ["git", "clone", remote_url, local_dir]
|
||||||
logging_subprocess(git_command)
|
logging_subprocess(git_command)
|
||||||
|
if lfs_clone:
|
||||||
|
git_command = ["git", "lfs", "fetch", "--all", "--prune"]
|
||||||
|
if no_prune:
|
||||||
|
git_command.pop()
|
||||||
|
logging_subprocess(git_command, cwd=local_dir)
|
||||||
|
|
||||||
|
|
||||||
def backup_account(args, output_directory):
|
def backup_account(args, output_directory):
|
||||||
@@ -2196,5 +2205,5 @@ def json_dump_if_changed(data, output_file_path):
|
|||||||
temp_file = output_file_path + ".temp"
|
temp_file = output_file_path + ".temp"
|
||||||
with codecs.open(temp_file, "w", encoding="utf-8") as f:
|
with codecs.open(temp_file, "w", encoding="utf-8") as f:
|
||||||
f.write(new_content)
|
f.write(new_content)
|
||||||
os.rename(temp_file, output_file_path) # Atomic on POSIX systems
|
os.replace(temp_file, output_file_path) # Atomic write
|
||||||
return True
|
return True
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
autopep8==2.3.2
|
autopep8==2.3.2
|
||||||
black==25.11.0
|
black==25.12.0
|
||||||
bleach==6.3.0
|
bleach==6.3.0
|
||||||
certifi==2025.11.12
|
certifi==2025.11.12
|
||||||
charset-normalizer==3.4.4
|
charset-normalizer==3.4.4
|
||||||
@@ -8,7 +8,7 @@ colorama==0.4.6
|
|||||||
docutils==0.22.3
|
docutils==0.22.3
|
||||||
flake8==7.3.0
|
flake8==7.3.0
|
||||||
gitchangelog==3.0.4
|
gitchangelog==3.0.4
|
||||||
pytest==9.0.1
|
pytest==9.0.2
|
||||||
idna==3.11
|
idna==3.11
|
||||||
importlib-metadata==8.7.0
|
importlib-metadata==8.7.0
|
||||||
jaraco.classes==3.4.0
|
jaraco.classes==3.4.0
|
||||||
@@ -21,7 +21,7 @@ mypy-extensions==1.1.0
|
|||||||
packaging==25.0
|
packaging==25.0
|
||||||
pathspec==0.12.1
|
pathspec==0.12.1
|
||||||
pkginfo==1.12.1.2
|
pkginfo==1.12.1.2
|
||||||
platformdirs==4.5.0
|
platformdirs==4.5.1
|
||||||
pycodestyle==2.14.0
|
pycodestyle==2.14.0
|
||||||
pyflakes==3.4.0
|
pyflakes==3.4.0
|
||||||
Pygments==2.19.2
|
Pygments==2.19.2
|
||||||
@@ -35,6 +35,6 @@ setuptools==80.9.0
|
|||||||
six==1.17.0
|
six==1.17.0
|
||||||
tqdm==4.67.1
|
tqdm==4.67.1
|
||||||
twine==6.2.0
|
twine==6.2.0
|
||||||
urllib3==2.5.0
|
urllib3==2.6.1
|
||||||
webencodings==0.5.1
|
webencodings==0.5.1
|
||||||
zipp==3.23.0
|
zipp==3.23.0
|
||||||
|
|||||||
6
setup.py
6
setup.py
@@ -33,7 +33,11 @@ setup(
|
|||||||
author="Jose Diaz-Gonzalez",
|
author="Jose Diaz-Gonzalez",
|
||||||
author_email="github-backup@josediazgonzalez.com",
|
author_email="github-backup@josediazgonzalez.com",
|
||||||
packages=["github_backup"],
|
packages=["github_backup"],
|
||||||
scripts=["bin/github-backup"],
|
entry_points={
|
||||||
|
"console_scripts": [
|
||||||
|
"github-backup=github_backup.cli:main",
|
||||||
|
],
|
||||||
|
},
|
||||||
url="http://github.com/josegonzalez/python-github-backup",
|
url="http://github.com/josegonzalez/python-github-backup",
|
||||||
license="MIT",
|
license="MIT",
|
||||||
classifiers=[
|
classifiers=[
|
||||||
|
|||||||
161
tests/test_all_starred.py
Normal file
161
tests/test_all_starred.py
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
"""Tests for --all-starred flag behavior (issue #225)."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from github_backup import github_backup
|
||||||
|
|
||||||
|
|
||||||
|
class TestAllStarredCloning:
|
||||||
|
"""Test suite for --all-starred repository cloning behavior.
|
||||||
|
|
||||||
|
Issue #225: --all-starred should clone starred repos without requiring --repositories.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _create_mock_args(self, **overrides):
|
||||||
|
"""Create a mock args object with sensible defaults."""
|
||||||
|
args = Mock()
|
||||||
|
args.user = "testuser"
|
||||||
|
args.output_directory = "/tmp/backup"
|
||||||
|
args.include_repository = False
|
||||||
|
args.include_everything = False
|
||||||
|
args.include_gists = False
|
||||||
|
args.include_starred_gists = False
|
||||||
|
args.all_starred = False
|
||||||
|
args.skip_existing = False
|
||||||
|
args.bare_clone = False
|
||||||
|
args.lfs_clone = False
|
||||||
|
args.no_prune = False
|
||||||
|
args.include_wiki = False
|
||||||
|
args.include_issues = False
|
||||||
|
args.include_issue_comments = False
|
||||||
|
args.include_issue_events = False
|
||||||
|
args.include_pulls = False
|
||||||
|
args.include_pull_comments = False
|
||||||
|
args.include_pull_commits = False
|
||||||
|
args.include_pull_details = False
|
||||||
|
args.include_labels = False
|
||||||
|
args.include_hooks = False
|
||||||
|
args.include_milestones = False
|
||||||
|
args.include_releases = False
|
||||||
|
args.include_assets = False
|
||||||
|
args.include_attachments = False
|
||||||
|
args.incremental = False
|
||||||
|
args.incremental_by_files = False
|
||||||
|
args.github_host = None
|
||||||
|
args.prefer_ssh = False
|
||||||
|
args.token_classic = None
|
||||||
|
args.token_fine = None
|
||||||
|
args.username = None
|
||||||
|
args.password = None
|
||||||
|
args.as_app = False
|
||||||
|
args.osx_keychain_item_name = None
|
||||||
|
args.osx_keychain_item_account = None
|
||||||
|
|
||||||
|
for key, value in overrides.items():
|
||||||
|
setattr(args, key, value)
|
||||||
|
|
||||||
|
return args
|
||||||
|
|
||||||
|
@patch('github_backup.github_backup.fetch_repository')
|
||||||
|
@patch('github_backup.github_backup.get_github_repo_url')
|
||||||
|
def test_all_starred_clones_without_repositories_flag(self, mock_get_url, mock_fetch):
|
||||||
|
"""--all-starred should clone starred repos without --repositories flag.
|
||||||
|
|
||||||
|
This is the core fix for issue #225.
|
||||||
|
"""
|
||||||
|
args = self._create_mock_args(all_starred=True)
|
||||||
|
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
|
||||||
|
|
||||||
|
# A starred repository (is_starred flag set by retrieve_repositories)
|
||||||
|
starred_repo = {
|
||||||
|
"name": "awesome-project",
|
||||||
|
"full_name": "otheruser/awesome-project",
|
||||||
|
"owner": {"login": "otheruser"},
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
"has_wiki": False,
|
||||||
|
"is_starred": True, # This flag is set for starred repos
|
||||||
|
}
|
||||||
|
|
||||||
|
with patch('github_backup.github_backup.mkdir_p'):
|
||||||
|
github_backup.backup_repositories(args, "/tmp/backup", [starred_repo])
|
||||||
|
|
||||||
|
# fetch_repository should be called for the starred repo
|
||||||
|
assert mock_fetch.called, "--all-starred should trigger repository cloning"
|
||||||
|
mock_fetch.assert_called_once()
|
||||||
|
call_args = mock_fetch.call_args
|
||||||
|
assert call_args[0][0] == "awesome-project" # repo name
|
||||||
|
|
||||||
|
@patch('github_backup.github_backup.fetch_repository')
|
||||||
|
@patch('github_backup.github_backup.get_github_repo_url')
|
||||||
|
def test_starred_repo_not_cloned_without_all_starred_flag(self, mock_get_url, mock_fetch):
|
||||||
|
"""Starred repos should NOT be cloned if --all-starred is not set."""
|
||||||
|
args = self._create_mock_args(all_starred=False)
|
||||||
|
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
|
||||||
|
|
||||||
|
starred_repo = {
|
||||||
|
"name": "awesome-project",
|
||||||
|
"full_name": "otheruser/awesome-project",
|
||||||
|
"owner": {"login": "otheruser"},
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
"has_wiki": False,
|
||||||
|
"is_starred": True,
|
||||||
|
}
|
||||||
|
|
||||||
|
with patch('github_backup.github_backup.mkdir_p'):
|
||||||
|
github_backup.backup_repositories(args, "/tmp/backup", [starred_repo])
|
||||||
|
|
||||||
|
# fetch_repository should NOT be called
|
||||||
|
assert not mock_fetch.called, "Starred repos should not be cloned without --all-starred"
|
||||||
|
|
||||||
|
@patch('github_backup.github_backup.fetch_repository')
|
||||||
|
@patch('github_backup.github_backup.get_github_repo_url')
|
||||||
|
def test_non_starred_repo_not_cloned_with_only_all_starred(self, mock_get_url, mock_fetch):
|
||||||
|
"""Non-starred repos should NOT be cloned when only --all-starred is set."""
|
||||||
|
args = self._create_mock_args(all_starred=True)
|
||||||
|
mock_get_url.return_value = "https://github.com/testuser/my-project.git"
|
||||||
|
|
||||||
|
# A regular (non-starred) repository
|
||||||
|
regular_repo = {
|
||||||
|
"name": "my-project",
|
||||||
|
"full_name": "testuser/my-project",
|
||||||
|
"owner": {"login": "testuser"},
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
"has_wiki": False,
|
||||||
|
# No is_starred flag
|
||||||
|
}
|
||||||
|
|
||||||
|
with patch('github_backup.github_backup.mkdir_p'):
|
||||||
|
github_backup.backup_repositories(args, "/tmp/backup", [regular_repo])
|
||||||
|
|
||||||
|
# fetch_repository should NOT be called for non-starred repos
|
||||||
|
assert not mock_fetch.called, "Non-starred repos should not be cloned with only --all-starred"
|
||||||
|
|
||||||
|
@patch('github_backup.github_backup.fetch_repository')
|
||||||
|
@patch('github_backup.github_backup.get_github_repo_url')
|
||||||
|
def test_repositories_flag_still_works(self, mock_get_url, mock_fetch):
|
||||||
|
"""--repositories flag should still clone repos as before."""
|
||||||
|
args = self._create_mock_args(include_repository=True)
|
||||||
|
mock_get_url.return_value = "https://github.com/testuser/my-project.git"
|
||||||
|
|
||||||
|
regular_repo = {
|
||||||
|
"name": "my-project",
|
||||||
|
"full_name": "testuser/my-project",
|
||||||
|
"owner": {"login": "testuser"},
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
"has_wiki": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
with patch('github_backup.github_backup.mkdir_p'):
|
||||||
|
github_backup.backup_repositories(args, "/tmp/backup", [regular_repo])
|
||||||
|
|
||||||
|
# fetch_repository should be called
|
||||||
|
assert mock_fetch.called, "--repositories should trigger repository cloning"
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
pytest.main([__file__, "-v"])
|
||||||
112
tests/test_case_sensitivity.py
Normal file
112
tests/test_case_sensitivity.py
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
"""Tests for case-insensitive username/organization filtering."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
|
from github_backup import github_backup
|
||||||
|
|
||||||
|
|
||||||
|
class TestCaseSensitivity:
|
||||||
|
"""Test suite for case-insensitive username matching in filter_repositories."""
|
||||||
|
|
||||||
|
def test_filter_repositories_case_insensitive_user(self):
|
||||||
|
"""Should filter repositories case-insensitively for usernames.
|
||||||
|
|
||||||
|
Reproduces issue #198 where typing 'iamrodos' fails to match
|
||||||
|
repositories with owner.login='Iamrodos' (the canonical case from GitHub API).
|
||||||
|
"""
|
||||||
|
# Simulate user typing lowercase username
|
||||||
|
args = Mock()
|
||||||
|
args.user = "iamrodos" # lowercase (what user typed)
|
||||||
|
args.repository = None
|
||||||
|
args.name_regex = None
|
||||||
|
args.languages = None
|
||||||
|
args.exclude = None
|
||||||
|
args.fork = False
|
||||||
|
args.private = False
|
||||||
|
args.public = False
|
||||||
|
args.all = True
|
||||||
|
|
||||||
|
# Simulate GitHub API returning canonical case
|
||||||
|
repos = [
|
||||||
|
{
|
||||||
|
"name": "repo1",
|
||||||
|
"owner": {"login": "Iamrodos"}, # Capital I (canonical from API)
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "repo2",
|
||||||
|
"owner": {"login": "Iamrodos"},
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
filtered = github_backup.filter_repositories(args, repos)
|
||||||
|
|
||||||
|
# Should match despite case difference
|
||||||
|
assert len(filtered) == 2
|
||||||
|
assert filtered[0]["name"] == "repo1"
|
||||||
|
assert filtered[1]["name"] == "repo2"
|
||||||
|
|
||||||
|
def test_filter_repositories_case_insensitive_org(self):
|
||||||
|
"""Should filter repositories case-insensitively for organizations.
|
||||||
|
|
||||||
|
Tests the example from issue #198 where 'prai-org' doesn't match 'PRAI-Org'.
|
||||||
|
"""
|
||||||
|
args = Mock()
|
||||||
|
args.user = "prai-org" # lowercase (what user typed)
|
||||||
|
args.repository = None
|
||||||
|
args.name_regex = None
|
||||||
|
args.languages = None
|
||||||
|
args.exclude = None
|
||||||
|
args.fork = False
|
||||||
|
args.private = False
|
||||||
|
args.public = False
|
||||||
|
args.all = True
|
||||||
|
|
||||||
|
repos = [
|
||||||
|
{
|
||||||
|
"name": "repo1",
|
||||||
|
"owner": {"login": "PRAI-Org"}, # Different case (canonical from API)
|
||||||
|
"private": False,
|
||||||
|
"fork": False,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
filtered = github_backup.filter_repositories(args, repos)
|
||||||
|
|
||||||
|
# Should match despite case difference
|
||||||
|
assert len(filtered) == 1
|
||||||
|
assert filtered[0]["name"] == "repo1"
|
||||||
|
|
||||||
|
def test_filter_repositories_case_variations(self):
|
||||||
|
"""Should handle various case combinations correctly."""
|
||||||
|
args = Mock()
|
||||||
|
args.user = "TeSt-UsEr" # Mixed case
|
||||||
|
args.repository = None
|
||||||
|
args.name_regex = None
|
||||||
|
args.languages = None
|
||||||
|
args.exclude = None
|
||||||
|
args.fork = False
|
||||||
|
args.private = False
|
||||||
|
args.public = False
|
||||||
|
args.all = True
|
||||||
|
|
||||||
|
repos = [
|
||||||
|
{"name": "repo1", "owner": {"login": "test-user"}, "private": False, "fork": False},
|
||||||
|
{"name": "repo2", "owner": {"login": "TEST-USER"}, "private": False, "fork": False},
|
||||||
|
{"name": "repo3", "owner": {"login": "TeSt-UsEr"}, "private": False, "fork": False},
|
||||||
|
{"name": "repo4", "owner": {"login": "other-user"}, "private": False, "fork": False},
|
||||||
|
]
|
||||||
|
|
||||||
|
filtered = github_backup.filter_repositories(args, repos)
|
||||||
|
|
||||||
|
# Should match first 3 (all case variations of same user)
|
||||||
|
assert len(filtered) == 3
|
||||||
|
assert set(r["name"] for r in filtered) == {"repo1", "repo2", "repo3"}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
pytest.main([__file__, "-v"])
|
||||||
Reference in New Issue
Block a user