mirror of
https://github.com/josegonzalez/python-github-backup.git
synced 2025-12-14 11:51:11 +01:00
Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2bb83d6d8b | ||
|
|
8fcc142621 | ||
|
|
7615ce6102 | ||
|
|
3f1ef821c3 | ||
|
|
3684756eaa | ||
|
|
e745b55755 | ||
|
|
75e6f56773 | ||
|
|
b991c363a0 | ||
|
|
6d74af9126 | ||
|
|
381d67af96 | ||
|
|
2fbe8d272c |
96
CHANGES.rst
96
CHANGES.rst
@@ -1,10 +1,104 @@
|
||||
Changelog
|
||||
=========
|
||||
|
||||
0.55.0 (2025-12-07)
|
||||
0.56.0 (2025-12-11)
|
||||
-------------------
|
||||
------------------------
|
||||
|
||||
Fix
|
||||
~~~
|
||||
- Replace deprecated git lfs clone with git clone + git lfs fetch --all.
|
||||
[Rodos]
|
||||
|
||||
git lfs clone is deprecated - modern git clone handles LFS automatically.
|
||||
Using git lfs fetch --all ensures all LFS objects across all refs are
|
||||
backed up, matching the existing bare clone behavior and providing
|
||||
complete LFS backups.
|
||||
|
||||
Closes #379
|
||||
- Add Windows support with entry_points and os.replace. [Rodos]
|
||||
|
||||
- Replace os.rename() with os.replace() for atomic file operations
|
||||
on Windows (os.rename fails if destination exists on Windows)
|
||||
- Add entry_points console_scripts for proper .exe generation on Windows
|
||||
- Create github_backup/cli.py with main() entry point
|
||||
- Add github_backup/__main__.py for python -m github_backup support
|
||||
- Keep bin/github-backup as thin wrapper for backwards compatibility
|
||||
|
||||
Closes #112
|
||||
|
||||
Other
|
||||
~~~~~
|
||||
- Docs: add "Restoring from Backup" section to README. [Rodos]
|
||||
|
||||
Clarifies that this tool is backup-only with no inbuilt restore.
|
||||
Documents that git repos can be pushed back, but issues/PRs have
|
||||
GitHub API limitations affecting all backup tools.
|
||||
|
||||
Closes #246
|
||||
- Chore(deps): bump urllib3 in the python-packages group.
|
||||
[dependabot[bot]]
|
||||
|
||||
Bumps the python-packages group with 1 update: [urllib3](https://github.com/urllib3/urllib3).
|
||||
|
||||
|
||||
Updates `urllib3` from 2.6.0 to 2.6.1
|
||||
- [Release notes](https://github.com/urllib3/urllib3/releases)
|
||||
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
|
||||
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.1)
|
||||
|
||||
---
|
||||
updated-dependencies:
|
||||
- dependency-name: urllib3
|
||||
dependency-version: 2.6.1
|
||||
dependency-type: direct:production
|
||||
update-type: version-update:semver-patch
|
||||
dependency-group: python-packages
|
||||
...
|
||||
- Chore(deps): bump the python-packages group with 3 updates.
|
||||
[dependabot[bot]]
|
||||
|
||||
Bumps the python-packages group with 3 updates: [black](https://github.com/psf/black), [pytest](https://github.com/pytest-dev/pytest) and [platformdirs](https://github.com/tox-dev/platformdirs).
|
||||
|
||||
|
||||
Updates `black` from 25.11.0 to 25.12.0
|
||||
- [Release notes](https://github.com/psf/black/releases)
|
||||
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
|
||||
- [Commits](https://github.com/psf/black/compare/25.11.0...25.12.0)
|
||||
|
||||
Updates `pytest` from 9.0.1 to 9.0.2
|
||||
- [Release notes](https://github.com/pytest-dev/pytest/releases)
|
||||
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
|
||||
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.1...9.0.2)
|
||||
|
||||
Updates `platformdirs` from 4.5.0 to 4.5.1
|
||||
- [Release notes](https://github.com/tox-dev/platformdirs/releases)
|
||||
- [Changelog](https://github.com/tox-dev/platformdirs/blob/main/CHANGES.rst)
|
||||
- [Commits](https://github.com/tox-dev/platformdirs/compare/4.5.0...4.5.1)
|
||||
|
||||
---
|
||||
updated-dependencies:
|
||||
- dependency-name: black
|
||||
dependency-version: 25.12.0
|
||||
dependency-type: direct:production
|
||||
update-type: version-update:semver-minor
|
||||
dependency-group: python-packages
|
||||
- dependency-name: pytest
|
||||
dependency-version: 9.0.2
|
||||
dependency-type: direct:production
|
||||
update-type: version-update:semver-patch
|
||||
dependency-group: python-packages
|
||||
- dependency-name: platformdirs
|
||||
dependency-version: 4.5.1
|
||||
dependency-type: direct:production
|
||||
update-type: version-update:semver-patch
|
||||
dependency-group: python-packages
|
||||
...
|
||||
|
||||
|
||||
0.55.0 (2025-12-07)
|
||||
-------------------
|
||||
|
||||
Fix
|
||||
~~~
|
||||
- Improve error messages for inaccessible repos and empty wikis. [Rodos]
|
||||
|
||||
21
README.rst
21
README.rst
@@ -215,6 +215,8 @@ When you use the ``--lfs`` option, you will need to make sure you have Git LFS i
|
||||
|
||||
Instructions on how to do this can be found on https://git-lfs.github.com.
|
||||
|
||||
LFS objects are fetched for all refs, not just the current checkout, ensuring a complete backup of all LFS content across all branches and history.
|
||||
|
||||
|
||||
About Attachments
|
||||
-----------------
|
||||
@@ -360,6 +362,25 @@ Debug an error/block or incomplete backup into a temporary directory. Omit "incr
|
||||
github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-commits --labels --milestones --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
|
||||
|
||||
|
||||
Restoring from Backup
|
||||
=====================
|
||||
|
||||
This tool creates backups only, there is no inbuilt restore command.
|
||||
|
||||
**Git repositories, wikis, and gists** can be restored by pushing them back to GitHub as you would any git repository. For example, to restore a bare repository backup::
|
||||
|
||||
cd /tmp/white-house/repositories/petitions/repository
|
||||
git push --mirror git@github.com:WhiteHouse/petitions.git
|
||||
|
||||
**Issues, pull requests, comments, and other metadata** are saved as JSON files for archival purposes. The GitHub API does not support recreating this data faithfully, creating issues via the API has limitations:
|
||||
|
||||
- New issue/PR numbers are assigned (original numbers cannot be set)
|
||||
- Timestamps reflect creation time (original dates cannot be set)
|
||||
- The API caller becomes the author (original authors cannot be set)
|
||||
- Cross-references between issues and PRs will break
|
||||
|
||||
These are GitHub API limitations that affect all backup and migration tools, not just this one. Recreating issues with these limitations via the GitHub API is an exercise for the reader. The JSON backups remain useful for searching, auditing, or manual reference.
|
||||
|
||||
|
||||
Development
|
||||
===========
|
||||
|
||||
@@ -1,76 +1,18 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
Backwards-compatible wrapper script.
|
||||
|
||||
The recommended way to run github-backup is via the installed command
|
||||
(pip install github-backup) or python -m github_backup.
|
||||
|
||||
This script is kept for backwards compatibility with existing installations
|
||||
that may reference this path directly.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
from github_backup.github_backup import (
|
||||
backup_account,
|
||||
backup_repositories,
|
||||
check_git_lfs_install,
|
||||
filter_repositories,
|
||||
get_auth,
|
||||
get_authenticated_user,
|
||||
logger,
|
||||
mkdir_p,
|
||||
parse_args,
|
||||
retrieve_repositories,
|
||||
)
|
||||
|
||||
# INFO and DEBUG go to stdout, WARNING and above go to stderr
|
||||
log_format = logging.Formatter(
|
||||
fmt="%(asctime)s.%(msecs)03d: %(message)s",
|
||||
datefmt="%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
stdout_handler = logging.StreamHandler(sys.stdout)
|
||||
stdout_handler.setLevel(logging.DEBUG)
|
||||
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
|
||||
stdout_handler.setFormatter(log_format)
|
||||
|
||||
stderr_handler = logging.StreamHandler(sys.stderr)
|
||||
stderr_handler.setLevel(logging.WARNING)
|
||||
stderr_handler.setFormatter(log_format)
|
||||
|
||||
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
if args.private and not get_auth(args):
|
||||
logger.warning(
|
||||
"The --private flag has no effect without authentication. "
|
||||
"Use -t/--token, -f/--token-fine, or -u/--username to authenticate."
|
||||
)
|
||||
|
||||
if args.quiet:
|
||||
logger.setLevel(logging.WARNING)
|
||||
|
||||
output_directory = os.path.realpath(args.output_directory)
|
||||
if not os.path.isdir(output_directory):
|
||||
logger.info("Create output directory {0}".format(output_directory))
|
||||
mkdir_p(output_directory)
|
||||
|
||||
if args.lfs_clone:
|
||||
check_git_lfs_install()
|
||||
|
||||
if args.log_level:
|
||||
log_level = logging.getLevelName(args.log_level.upper())
|
||||
if isinstance(log_level, int):
|
||||
logger.root.setLevel(log_level)
|
||||
|
||||
if not args.as_app:
|
||||
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
|
||||
authenticated_user = get_authenticated_user(args)
|
||||
else:
|
||||
authenticated_user = {"login": None}
|
||||
|
||||
repositories = retrieve_repositories(args, authenticated_user)
|
||||
repositories = filter_repositories(args, repositories)
|
||||
backup_repositories(args, output_directory, repositories)
|
||||
backup_account(args, output_directory)
|
||||
|
||||
from github_backup.cli import main
|
||||
from github_backup.github_backup import logger
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
|
||||
@@ -1 +1 @@
|
||||
__version__ = "0.55.0"
|
||||
__version__ = "0.56.0"
|
||||
|
||||
13
github_backup/__main__.py
Normal file
13
github_backup/__main__.py
Normal file
@@ -0,0 +1,13 @@
|
||||
"""Allow running as: python -m github_backup"""
|
||||
|
||||
import sys
|
||||
|
||||
from github_backup.cli import main
|
||||
from github_backup.github_backup import logger
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e:
|
||||
logger.error(str(e))
|
||||
sys.exit(1)
|
||||
82
github_backup/cli.py
Normal file
82
github_backup/cli.py
Normal file
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env python
|
||||
"""Command-line interface for github-backup."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
from github_backup.github_backup import (
|
||||
backup_account,
|
||||
backup_repositories,
|
||||
check_git_lfs_install,
|
||||
filter_repositories,
|
||||
get_auth,
|
||||
get_authenticated_user,
|
||||
logger,
|
||||
mkdir_p,
|
||||
parse_args,
|
||||
retrieve_repositories,
|
||||
)
|
||||
|
||||
# INFO and DEBUG go to stdout, WARNING and above go to stderr
|
||||
log_format = logging.Formatter(
|
||||
fmt="%(asctime)s.%(msecs)03d: %(message)s",
|
||||
datefmt="%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
stdout_handler = logging.StreamHandler(sys.stdout)
|
||||
stdout_handler.setLevel(logging.DEBUG)
|
||||
stdout_handler.addFilter(lambda r: r.levelno < logging.WARNING)
|
||||
stdout_handler.setFormatter(log_format)
|
||||
|
||||
stderr_handler = logging.StreamHandler(sys.stderr)
|
||||
stderr_handler.setLevel(logging.WARNING)
|
||||
stderr_handler.setFormatter(log_format)
|
||||
|
||||
logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler])
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for github-backup CLI."""
|
||||
args = parse_args()
|
||||
|
||||
if args.private and not get_auth(args):
|
||||
logger.warning(
|
||||
"The --private flag has no effect without authentication. "
|
||||
"Use -t/--token, -f/--token-fine, or -u/--username to authenticate."
|
||||
)
|
||||
|
||||
if args.quiet:
|
||||
logger.setLevel(logging.WARNING)
|
||||
|
||||
output_directory = os.path.realpath(args.output_directory)
|
||||
if not os.path.isdir(output_directory):
|
||||
logger.info("Create output directory {0}".format(output_directory))
|
||||
mkdir_p(output_directory)
|
||||
|
||||
if args.lfs_clone:
|
||||
check_git_lfs_install()
|
||||
|
||||
if args.log_level:
|
||||
log_level = logging.getLevelName(args.log_level.upper())
|
||||
if isinstance(log_level, int):
|
||||
logger.root.setLevel(log_level)
|
||||
|
||||
if not args.as_app:
|
||||
logger.info("Backing up user {0} to {1}".format(args.user, output_directory))
|
||||
authenticated_user = get_authenticated_user(args)
|
||||
else:
|
||||
authenticated_user = {"login": None}
|
||||
|
||||
repositories = retrieve_repositories(args, authenticated_user)
|
||||
repositories = filter_repositories(args, repositories)
|
||||
backup_repositories(args, output_directory, repositories)
|
||||
backup_account(args, output_directory)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e:
|
||||
logger.error(str(e))
|
||||
sys.exit(1)
|
||||
@@ -1038,7 +1038,7 @@ def download_attachment_file(url, path, auth, as_app=False, fine=False):
|
||||
bytes_downloaded += len(chunk)
|
||||
|
||||
# Atomic rename to final location
|
||||
os.rename(temp_path, path)
|
||||
os.replace(temp_path, path)
|
||||
|
||||
metadata["size_bytes"] = bytes_downloaded
|
||||
metadata["success"] = True
|
||||
@@ -1459,7 +1459,7 @@ def download_attachments(
|
||||
|
||||
# Rename to add extension (already atomic from download)
|
||||
try:
|
||||
os.rename(filepath, final_filepath)
|
||||
os.replace(filepath, final_filepath)
|
||||
metadata["saved_as"] = os.path.basename(final_filepath)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
@@ -1490,7 +1490,7 @@ def download_attachments(
|
||||
manifest_path = os.path.join(attachments_dir, "manifest.json")
|
||||
with open(manifest_path + ".temp", "w") as f:
|
||||
json.dump(manifest, f, indent=2)
|
||||
os.rename(manifest_path + ".temp", manifest_path) # Atomic write
|
||||
os.replace(manifest_path + ".temp", manifest_path) # Atomic write
|
||||
logger.debug(
|
||||
"Wrote manifest for {0} #{1}: {2} attachments".format(
|
||||
item_type_display, number, len(attachment_metadata_list)
|
||||
@@ -1811,7 +1811,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
|
||||
|
||||
with codecs.open(issue_file + ".temp", "w", encoding="utf-8") as f:
|
||||
json_dump(issue, f)
|
||||
os.rename(issue_file + ".temp", issue_file) # Unlike json_dump, this is atomic
|
||||
os.replace(issue_file + ".temp", issue_file) # Atomic write
|
||||
|
||||
|
||||
def backup_pulls(args, repo_cwd, repository, repos_template):
|
||||
@@ -1886,7 +1886,7 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
|
||||
|
||||
with codecs.open(pull_file + ".temp", "w", encoding="utf-8") as f:
|
||||
json_dump(pull, f)
|
||||
os.rename(pull_file + ".temp", pull_file) # Unlike json_dump, this is atomic
|
||||
os.replace(pull_file + ".temp", pull_file) # Atomic write
|
||||
|
||||
|
||||
def backup_milestones(args, repo_cwd, repository, repos_template):
|
||||
@@ -2089,12 +2089,14 @@ def fetch_repository(
|
||||
if no_prune:
|
||||
git_command.pop()
|
||||
logging_subprocess(git_command, cwd=local_dir)
|
||||
else:
|
||||
if lfs_clone:
|
||||
git_command = ["git", "lfs", "clone", remote_url, local_dir]
|
||||
else:
|
||||
git_command = ["git", "clone", remote_url, local_dir]
|
||||
logging_subprocess(git_command)
|
||||
if lfs_clone:
|
||||
git_command = ["git", "lfs", "fetch", "--all", "--prune"]
|
||||
if no_prune:
|
||||
git_command.pop()
|
||||
logging_subprocess(git_command, cwd=local_dir)
|
||||
|
||||
|
||||
def backup_account(args, output_directory):
|
||||
@@ -2203,5 +2205,5 @@ def json_dump_if_changed(data, output_file_path):
|
||||
temp_file = output_file_path + ".temp"
|
||||
with codecs.open(temp_file, "w", encoding="utf-8") as f:
|
||||
f.write(new_content)
|
||||
os.rename(temp_file, output_file_path) # Atomic on POSIX systems
|
||||
os.replace(temp_file, output_file_path) # Atomic write
|
||||
return True
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
autopep8==2.3.2
|
||||
black==25.11.0
|
||||
black==25.12.0
|
||||
bleach==6.3.0
|
||||
certifi==2025.11.12
|
||||
charset-normalizer==3.4.4
|
||||
@@ -8,7 +8,7 @@ colorama==0.4.6
|
||||
docutils==0.22.3
|
||||
flake8==7.3.0
|
||||
gitchangelog==3.0.4
|
||||
pytest==9.0.1
|
||||
pytest==9.0.2
|
||||
idna==3.11
|
||||
importlib-metadata==8.7.0
|
||||
jaraco.classes==3.4.0
|
||||
@@ -21,7 +21,7 @@ mypy-extensions==1.1.0
|
||||
packaging==25.0
|
||||
pathspec==0.12.1
|
||||
pkginfo==1.12.1.2
|
||||
platformdirs==4.5.0
|
||||
platformdirs==4.5.1
|
||||
pycodestyle==2.14.0
|
||||
pyflakes==3.4.0
|
||||
Pygments==2.19.2
|
||||
@@ -35,6 +35,6 @@ setuptools==80.9.0
|
||||
six==1.17.0
|
||||
tqdm==4.67.1
|
||||
twine==6.2.0
|
||||
urllib3==2.6.0
|
||||
urllib3==2.6.1
|
||||
webencodings==0.5.1
|
||||
zipp==3.23.0
|
||||
|
||||
6
setup.py
6
setup.py
@@ -33,7 +33,11 @@ setup(
|
||||
author="Jose Diaz-Gonzalez",
|
||||
author_email="github-backup@josediazgonzalez.com",
|
||||
packages=["github_backup"],
|
||||
scripts=["bin/github-backup"],
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"github-backup=github_backup.cli:main",
|
||||
],
|
||||
},
|
||||
url="http://github.com/josegonzalez/python-github-backup",
|
||||
license="MIT",
|
||||
classifiers=[
|
||||
|
||||
Reference in New Issue
Block a user