Compare commits

..

11 Commits

Author SHA1 Message Date
GitHub Action
eb5779ac23 Release version 0.55.0 2025-12-07 13:59:35 +00:00
Jose Diaz-Gonzalez
5b52931ebf Merge pull request #461 from Iamrodos/fix-cli-ux-and-cleanup
fix: CLI UX improvements and cleanup
2025-12-07 08:58:59 -05:00
Rodos
1d6d474408 fix: improve error messages for inaccessible repos and empty wikis 2025-12-07 21:50:49 +11:00
Rodos
b80049e96e test: add missing test coverage for case sensitivity fix 2025-12-07 21:21:37 +11:00
Rodos
58ad1c2378 docs: fix RST formatting in Known blocking errors section 2025-12-07 21:21:26 +11:00
Rodos
6e2a7e521c fix: --all-starred now clones repos without --repositories 2025-12-07 21:21:14 +11:00
Rodos
aba048a3e9 fix: warn when --private used without authentication 2025-12-07 21:20:54 +11:00
Jose Diaz-Gonzalez
9f7c08166f Merge pull request #460 from josegonzalez/dependabot/pip/urllib3-2.6.0
chore(deps): bump urllib3 from 2.5.0 to 2.6.0
2025-12-06 22:23:09 -05:00
dependabot[bot]
fdfaaec1ba chore(deps): bump urllib3 from 2.5.0 to 2.6.0
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-06 04:51:42 +00:00
Jose Diaz-Gonzalez
8f9cf7ff89 Merge pull request #459 from Iamrodos/issue-93-starred-gists-warning
fix: warn and skip when --starred-gists used for different user
2025-12-03 23:07:29 -05:00
Rodos
899ab5fdc2 fix: warn and skip when --starred-gists used for different user
GitHub's API only allows retrieving starred gists for the authenticated
user. Previously, using --starred-gists when backing up a different user
would silently return no relevant data.

Now warns and skips the retrieval entirely when the target user differs
from the authenticated user. Uses case-insensitive comparison to match
GitHub's username handling.

Fixes #93
2025-12-04 10:07:43 +11:00
8 changed files with 356 additions and 28 deletions

View File

@@ -1,10 +1,49 @@
Changelog Changelog
========= =========
0.54.0 (2025-12-03) 0.55.0 (2025-12-07)
------------------- -------------------
------------------------ ------------------------
Fix
~~~
- Improve error messages for inaccessible repos and empty wikis. [Rodos]
- --all-starred now clones repos without --repositories. [Rodos]
- Warn when --private used without authentication. [Rodos]
- Warn and skip when --starred-gists used for different user. [Rodos]
GitHub's API only allows retrieving starred gists for the authenticated
user. Previously, using --starred-gists when backing up a different user
would silently return no relevant data.
Now warns and skips the retrieval entirely when the target user differs
from the authenticated user. Uses case-insensitive comparison to match
GitHub's username handling.
Fixes #93
Other
~~~~~
- Test: add missing test coverage for case sensitivity fix. [Rodos]
- Docs: fix RST formatting in Known blocking errors section. [Rodos]
- Chore(deps): bump urllib3 from 2.5.0 to 2.6.0. [dependabot[bot]]
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)
---
updated-dependencies:
- dependency-name: urllib3
dependency-version: 2.6.0
dependency-type: direct:production
...
0.54.0 (2025-12-03)
-------------------
Fix Fix
~~~ ~~~
- Send INFO/DEBUG to stdout, WARNING/ERROR to stderr. [Rodos] - Send INFO/DEBUG to stdout, WARNING/ERROR to stderr. [Rodos]

View File

@@ -281,11 +281,11 @@ If the incremental argument is used, this will result in the next backup only re
It's therefore recommended to only use the incremental argument if the output/result is being actively monitored, or complimented with periodic full non-incremental runs, to avoid unexpected missing data in a regular backup runs. It's therefore recommended to only use the incremental argument if the output/result is being actively monitored, or complimented with periodic full non-incremental runs, to avoid unexpected missing data in a regular backup runs.
1. **Starred public repo hooks blocking** **Starred public repo hooks blocking**
Since the ``--all`` argument includes ``--hooks``, if you use ``--all`` and ``--all-starred`` together to clone a users starred public repositories, the backup will likely error and block the backup continuing. Since the ``--all`` argument includes ``--hooks``, if you use ``--all`` and ``--all-starred`` together to clone a users starred public repositories, the backup will likely error and block the backup continuing.
This is due to needing the correct permission for ``--hooks`` on public repos. This is due to needing the correct permission for ``--hooks`` on public repos.
"bare" is actually "mirror" "bare" is actually "mirror"
@@ -301,6 +301,8 @@ Starred gists vs starred repo behaviour
The starred normal repo cloning (``--all-starred``) argument stores starred repos separately to the users own repositories. However, using ``--starred-gists`` will store starred gists within the same directory as the users own gists ``--gists``. Also, all gist repo directory names are IDs not the gist's name. The starred normal repo cloning (``--all-starred``) argument stores starred repos separately to the users own repositories. However, using ``--starred-gists`` will store starred gists within the same directory as the users own gists ``--gists``. Also, all gist repo directory names are IDs not the gist's name.
Note: ``--starred-gists`` only retrieves starred gists for the authenticated user, not the target user, due to a GitHub API limitation.
Skip existing on incomplete backups Skip existing on incomplete backups
----------------------------------- -----------------------------------

View File

@@ -9,6 +9,7 @@ from github_backup.github_backup import (
backup_repositories, backup_repositories,
check_git_lfs_install, check_git_lfs_install,
filter_repositories, filter_repositories,
get_auth,
get_authenticated_user, get_authenticated_user,
logger, logger,
mkdir_p, mkdir_p,
@@ -37,6 +38,12 @@ logging.basicConfig(level=logging.INFO, handlers=[stdout_handler, stderr_handler
def main(): def main():
args = parse_args() args = parse_args()
if args.private and not get_auth(args):
logger.warning(
"The --private flag has no effect without authentication. "
"Use -t/--token, -f/--token-fine, or -u/--username to authenticate."
)
if args.quiet: if args.quiet:
logger.setLevel(logging.WARNING) logger.setLevel(logging.WARNING)

View File

@@ -1 +1 @@
__version__ = "0.54.0" __version__ = "0.55.0"

View File

@@ -561,7 +561,7 @@ def get_github_host(args):
def read_file_contents(file_uri): def read_file_contents(file_uri):
return open(file_uri[len(FILE_URI_PREFIX) :], "rt").readline().strip() return open(file_uri[len(FILE_URI_PREFIX):], "rt").readline().strip()
def get_github_repo_url(args, repository): def get_github_repo_url(args, repository):
@@ -1565,16 +1565,22 @@ def retrieve_repositories(args, authenticated_user):
repos.extend(gists) repos.extend(gists)
if args.include_starred_gists: if args.include_starred_gists:
starred_gists_template = "https://{0}/gists/starred".format( if not authenticated_user.get("login") or args.user.lower() != authenticated_user["login"].lower():
get_github_api_host(args) logger.warning(
) "Cannot retrieve starred gists for '%s'. GitHub only allows access to the authenticated user's starred gists.",
starred_gists = retrieve_data( args.user,
args, starred_gists_template, single_request=False )
) else:
# flag each repo as a starred gist for downstream processing starred_gists_template = "https://{0}/gists/starred".format(
for item in starred_gists: get_github_api_host(args)
item.update({"is_gist": True, "is_starred": True}) )
repos.extend(starred_gists) starred_gists = retrieve_data(
args, starred_gists_template, single_request=False
)
# flag each repo as a starred gist for downstream processing
for item in starred_gists:
item.update({"is_gist": True, "is_starred": True})
repos.extend(starred_gists)
return repos return repos
@@ -1666,9 +1672,10 @@ def backup_repositories(args, output_directory, repositories):
repo_url = get_github_repo_url(args, repository) repo_url = get_github_repo_url(args, repository)
include_gists = args.include_gists or args.include_starred_gists include_gists = args.include_gists or args.include_starred_gists
include_starred = args.all_starred and repository.get("is_starred")
if (args.include_repository or args.include_everything) or ( if (args.include_repository or args.include_everything) or (
include_gists and repository.get("is_gist") include_gists and repository.get("is_gist")
): ) or include_starred:
repo_name = ( repo_name = (
repository.get("name") repository.get("name")
if not repository.get("is_gist") if not repository.get("is_gist")
@@ -2017,12 +2024,9 @@ def fetch_repository(
): ):
if bare_clone: if bare_clone:
if os.path.exists(local_dir): if os.path.exists(local_dir):
clone_exists = ( clone_exists = subprocess.check_output(
subprocess.check_output( ["git", "rev-parse", "--is-bare-repository"], cwd=local_dir
["git", "rev-parse", "--is-bare-repository"], cwd=local_dir ) == b"true\n"
)
== b"true\n"
)
else: else:
clone_exists = False clone_exists = False
else: else:
@@ -2037,11 +2041,14 @@ def fetch_repository(
"git ls-remote " + remote_url, stdout=FNULL, stderr=FNULL, shell=True "git ls-remote " + remote_url, stdout=FNULL, stderr=FNULL, shell=True
) )
if initialized == 128: if initialized == 128:
logger.info( if ".wiki.git" in remote_url:
"Skipping {0} ({1}) since it's not initialized".format( logger.info(
name, masked_remote_url "Skipping {0} wiki (wiki is enabled but has no content)".format(name)
)
else:
logger.info(
"Skipping {0} (repository not accessible - may be empty, private, or credentials invalid)".format(name)
) )
)
return return
if clone_exists: if clone_exists:

View File

@@ -35,6 +35,6 @@ setuptools==80.9.0
six==1.17.0 six==1.17.0
tqdm==4.67.1 tqdm==4.67.1
twine==6.2.0 twine==6.2.0
urllib3==2.5.0 urllib3==2.6.0
webencodings==0.5.1 webencodings==0.5.1
zipp==3.23.0 zipp==3.23.0

161
tests/test_all_starred.py Normal file
View File

@@ -0,0 +1,161 @@
"""Tests for --all-starred flag behavior (issue #225)."""
import pytest
from unittest.mock import Mock, patch
from github_backup import github_backup
class TestAllStarredCloning:
"""Test suite for --all-starred repository cloning behavior.
Issue #225: --all-starred should clone starred repos without requiring --repositories.
"""
def _create_mock_args(self, **overrides):
"""Create a mock args object with sensible defaults."""
args = Mock()
args.user = "testuser"
args.output_directory = "/tmp/backup"
args.include_repository = False
args.include_everything = False
args.include_gists = False
args.include_starred_gists = False
args.all_starred = False
args.skip_existing = False
args.bare_clone = False
args.lfs_clone = False
args.no_prune = False
args.include_wiki = False
args.include_issues = False
args.include_issue_comments = False
args.include_issue_events = False
args.include_pulls = False
args.include_pull_comments = False
args.include_pull_commits = False
args.include_pull_details = False
args.include_labels = False
args.include_hooks = False
args.include_milestones = False
args.include_releases = False
args.include_assets = False
args.include_attachments = False
args.incremental = False
args.incremental_by_files = False
args.github_host = None
args.prefer_ssh = False
args.token_classic = None
args.token_fine = None
args.username = None
args.password = None
args.as_app = False
args.osx_keychain_item_name = None
args.osx_keychain_item_account = None
for key, value in overrides.items():
setattr(args, key, value)
return args
@patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url')
def test_all_starred_clones_without_repositories_flag(self, mock_get_url, mock_fetch):
"""--all-starred should clone starred repos without --repositories flag.
This is the core fix for issue #225.
"""
args = self._create_mock_args(all_starred=True)
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
# A starred repository (is_starred flag set by retrieve_repositories)
starred_repo = {
"name": "awesome-project",
"full_name": "otheruser/awesome-project",
"owner": {"login": "otheruser"},
"private": False,
"fork": False,
"has_wiki": False,
"is_starred": True, # This flag is set for starred repos
}
with patch('github_backup.github_backup.mkdir_p'):
github_backup.backup_repositories(args, "/tmp/backup", [starred_repo])
# fetch_repository should be called for the starred repo
assert mock_fetch.called, "--all-starred should trigger repository cloning"
mock_fetch.assert_called_once()
call_args = mock_fetch.call_args
assert call_args[0][0] == "awesome-project" # repo name
@patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url')
def test_starred_repo_not_cloned_without_all_starred_flag(self, mock_get_url, mock_fetch):
"""Starred repos should NOT be cloned if --all-starred is not set."""
args = self._create_mock_args(all_starred=False)
mock_get_url.return_value = "https://github.com/otheruser/awesome-project.git"
starred_repo = {
"name": "awesome-project",
"full_name": "otheruser/awesome-project",
"owner": {"login": "otheruser"},
"private": False,
"fork": False,
"has_wiki": False,
"is_starred": True,
}
with patch('github_backup.github_backup.mkdir_p'):
github_backup.backup_repositories(args, "/tmp/backup", [starred_repo])
# fetch_repository should NOT be called
assert not mock_fetch.called, "Starred repos should not be cloned without --all-starred"
@patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url')
def test_non_starred_repo_not_cloned_with_only_all_starred(self, mock_get_url, mock_fetch):
"""Non-starred repos should NOT be cloned when only --all-starred is set."""
args = self._create_mock_args(all_starred=True)
mock_get_url.return_value = "https://github.com/testuser/my-project.git"
# A regular (non-starred) repository
regular_repo = {
"name": "my-project",
"full_name": "testuser/my-project",
"owner": {"login": "testuser"},
"private": False,
"fork": False,
"has_wiki": False,
# No is_starred flag
}
with patch('github_backup.github_backup.mkdir_p'):
github_backup.backup_repositories(args, "/tmp/backup", [regular_repo])
# fetch_repository should NOT be called for non-starred repos
assert not mock_fetch.called, "Non-starred repos should not be cloned with only --all-starred"
@patch('github_backup.github_backup.fetch_repository')
@patch('github_backup.github_backup.get_github_repo_url')
def test_repositories_flag_still_works(self, mock_get_url, mock_fetch):
"""--repositories flag should still clone repos as before."""
args = self._create_mock_args(include_repository=True)
mock_get_url.return_value = "https://github.com/testuser/my-project.git"
regular_repo = {
"name": "my-project",
"full_name": "testuser/my-project",
"owner": {"login": "testuser"},
"private": False,
"fork": False,
"has_wiki": False,
}
with patch('github_backup.github_backup.mkdir_p'):
github_backup.backup_repositories(args, "/tmp/backup", [regular_repo])
# fetch_repository should be called
assert mock_fetch.called, "--repositories should trigger repository cloning"
if __name__ == "__main__":
pytest.main([__file__, "-v"])

View File

@@ -0,0 +1,112 @@
"""Tests for case-insensitive username/organization filtering."""
import pytest
from unittest.mock import Mock
from github_backup import github_backup
class TestCaseSensitivity:
"""Test suite for case-insensitive username matching in filter_repositories."""
def test_filter_repositories_case_insensitive_user(self):
"""Should filter repositories case-insensitively for usernames.
Reproduces issue #198 where typing 'iamrodos' fails to match
repositories with owner.login='Iamrodos' (the canonical case from GitHub API).
"""
# Simulate user typing lowercase username
args = Mock()
args.user = "iamrodos" # lowercase (what user typed)
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
# Simulate GitHub API returning canonical case
repos = [
{
"name": "repo1",
"owner": {"login": "Iamrodos"}, # Capital I (canonical from API)
"private": False,
"fork": False,
},
{
"name": "repo2",
"owner": {"login": "Iamrodos"},
"private": False,
"fork": False,
},
]
filtered = github_backup.filter_repositories(args, repos)
# Should match despite case difference
assert len(filtered) == 2
assert filtered[0]["name"] == "repo1"
assert filtered[1]["name"] == "repo2"
def test_filter_repositories_case_insensitive_org(self):
"""Should filter repositories case-insensitively for organizations.
Tests the example from issue #198 where 'prai-org' doesn't match 'PRAI-Org'.
"""
args = Mock()
args.user = "prai-org" # lowercase (what user typed)
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
repos = [
{
"name": "repo1",
"owner": {"login": "PRAI-Org"}, # Different case (canonical from API)
"private": False,
"fork": False,
},
]
filtered = github_backup.filter_repositories(args, repos)
# Should match despite case difference
assert len(filtered) == 1
assert filtered[0]["name"] == "repo1"
def test_filter_repositories_case_variations(self):
"""Should handle various case combinations correctly."""
args = Mock()
args.user = "TeSt-UsEr" # Mixed case
args.repository = None
args.name_regex = None
args.languages = None
args.exclude = None
args.fork = False
args.private = False
args.public = False
args.all = True
repos = [
{"name": "repo1", "owner": {"login": "test-user"}, "private": False, "fork": False},
{"name": "repo2", "owner": {"login": "TEST-USER"}, "private": False, "fork": False},
{"name": "repo3", "owner": {"login": "TeSt-UsEr"}, "private": False, "fork": False},
{"name": "repo4", "owner": {"login": "other-user"}, "private": False, "fork": False},
]
filtered = github_backup.filter_repositories(args, repos)
# Should match first 3 (all case variations of same user)
assert len(filtered) == 3
assert set(r["name"] for r in filtered) == {"repo1", "repo2", "repo3"}
if __name__ == "__main__":
pytest.main([__file__, "-v"])