Release version 0.50.0

chore: fix inline comments

CHANGES.rst

@@ -1,9 +1,223 @@
 Changelog
 =========
 
-0.47.0 (2024-12-09)
+0.50.0 (2025-02-22)
 -------------------
 ------------------------
+- Chore: fix inline comments. [Jose Diaz-Gonzalez]
+- Chore(deps): bump the python-packages group across 1 directory with 2
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 2 updates in the / directory: [flake8](https://github.com/pycqa/flake8) and [pkginfo](https://code.launchpad.net/~tseaver/pkginfo/trunk).
+
+
+  Updates `flake8` from 7.1.1 to 7.1.2
+  - [Commits](https://github.com/pycqa/flake8/compare/7.1.1...7.1.2)
+
+  Updates `pkginfo` from 1.12.0 to 1.12.1.2
+
+  ---
+  updated-dependencies:
+  - dependency-name: flake8
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  - dependency-name: pkginfo
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  ...
+
+
+0.49.0 (2025-02-01)
+-------------------
+- Convert timestamp to string, although maybe the other way around would
+  be better ... [Honza Maly]
+- Implementing incremental by files, safer version of incremental
+  backup. [Honza Maly]
+- Chore(deps): bump the python-packages group across 1 directory with 7
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 7 updates in the / directory:
+
+  | Package | From | To |
+  | --- | --- | --- |
+  | [autopep8](https://github.com/hhatto/autopep8) | `2.3.1` | `2.3.2` |
+  | [black](https://github.com/psf/black) | `24.10.0` | `25.1.0` |
+  | [certifi](https://github.com/certifi/python-certifi) | `2024.12.14` | `2025.1.31` |
+  | [importlib-metadata](https://github.com/python/importlib_metadata) | `8.5.0` | `8.6.1` |
+  | [more-itertools](https://github.com/more-itertools/more-itertools) | `10.5.0` | `10.6.0` |
+  | [setuptools](https://github.com/pypa/setuptools) | `75.7.0` | `75.8.0` |
+  | [twine](https://github.com/pypa/twine) | `6.0.1` | `6.1.0` |
+
+
+
+  Updates `autopep8` from 2.3.1 to 2.3.2
+  - [Release notes](https://github.com/hhatto/autopep8/releases)
+  - [Commits](https://github.com/hhatto/autopep8/compare/v2.3.1...v2.3.2)
+
+  Updates `black` from 24.10.0 to 25.1.0
+  - [Release notes](https://github.com/psf/black/releases)
+  - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
+  - [Commits](https://github.com/psf/black/compare/24.10.0...25.1.0)
+
+  Updates `certifi` from 2024.12.14 to 2025.1.31
+  - [Commits](https://github.com/certifi/python-certifi/compare/2024.12.14...2025.01.31)
+
+  Updates `importlib-metadata` from 8.5.0 to 8.6.1
+  - [Release notes](https://github.com/python/importlib_metadata/releases)
+  - [Changelog](https://github.com/python/importlib_metadata/blob/main/NEWS.rst)
+  - [Commits](https://github.com/python/importlib_metadata/compare/v8.5.0...v8.6.1)
+
+  Updates `more-itertools` from 10.5.0 to 10.6.0
+  - [Release notes](https://github.com/more-itertools/more-itertools/releases)
+  - [Commits](https://github.com/more-itertools/more-itertools/compare/v10.5.0...v10.6.0)
+
+  Updates `setuptools` from 75.7.0 to 75.8.0
+  - [Release notes](https://github.com/pypa/setuptools/releases)
+  - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
+  - [Commits](https://github.com/pypa/setuptools/compare/v75.7.0...v75.8.0)
+
+  Updates `twine` from 6.0.1 to 6.1.0
+  - [Release notes](https://github.com/pypa/twine/releases)
+  - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
+  - [Commits](https://github.com/pypa/twine/compare/6.0.1...6.1.0)
+
+  ---
+  updated-dependencies:
+  - dependency-name: autopep8
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  - dependency-name: black
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+    dependency-group: python-packages
+  - dependency-name: certifi
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+    dependency-group: python-packages
+  - dependency-name: importlib-metadata
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: more-itertools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: setuptools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: twine
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+- Chore(deps): bump the python-packages group across 1 directory with 2
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 2 updates in the / directory: [pygments](https://github.com/pygments/pygments) and [setuptools](https://github.com/pypa/setuptools).
+
+
+  Updates `pygments` from 2.18.0 to 2.19.1
+  - [Release notes](https://github.com/pygments/pygments/releases)
+  - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
+  - [Commits](https://github.com/pygments/pygments/compare/2.18.0...2.19.1)
+
+  Updates `setuptools` from 75.6.0 to 75.7.0
+  - [Release notes](https://github.com/pypa/setuptools/releases)
+  - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
+  - [Commits](https://github.com/pypa/setuptools/compare/v75.6.0...v75.7.0)
+
+  ---
+  updated-dependencies:
+  - dependency-name: pygments
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: setuptools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+
+
+0.48.0 (2025-01-04)
+-------------------
+- Chore: reformat file to fix lint issues. [Jose Diaz-Gonzalez]
+- Chore(deps): bump the python-packages group across 1 directory with 4
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 4 updates in the / directory: [charset-normalizer](https://github.com/jawah/charset_normalizer), [click](https://github.com/pallets/click), [keyring](https://github.com/jaraco/keyring) and [urllib3](https://github.com/urllib3/urllib3).
+
+
+  Updates `charset-normalizer` from 3.4.0 to 3.4.1
+  - [Release notes](https://github.com/jawah/charset_normalizer/releases)
+  - [Changelog](https://github.com/jawah/charset_normalizer/blob/master/CHANGELOG.md)
+  - [Commits](https://github.com/jawah/charset_normalizer/compare/3.4.0...3.4.1)
+
+  Updates `click` from 8.1.7 to 8.1.8
+  - [Release notes](https://github.com/pallets/click/releases)
+  - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst)
+  - [Commits](https://github.com/pallets/click/compare/8.1.7...8.1.8)
+
+  Updates `keyring` from 25.5.0 to 25.6.0
+  - [Release notes](https://github.com/jaraco/keyring/releases)
+  - [Changelog](https://github.com/jaraco/keyring/blob/main/NEWS.rst)
+  - [Commits](https://github.com/jaraco/keyring/compare/v25.5.0...v25.6.0)
+
+  Updates `urllib3` from 2.2.3 to 2.3.0
+  - [Release notes](https://github.com/urllib3/urllib3/releases)
+  - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
+  - [Commits](https://github.com/urllib3/urllib3/compare/2.2.3...2.3.0)
+
+  ---
+  updated-dependencies:
+  - dependency-name: charset-normalizer
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  - dependency-name: click
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  - dependency-name: keyring
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: urllib3
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+- Fix typo README.rst: --starred-gists that should be --gists. [Michael
+  D. Adams]
+- Remove fixed release issue from known blocking errors. [Ethan White]
+
+  The issue with --release producing errors documented in #209 (the linked issue) and #234 appears to have been fixed in #257.
+
+  This change removes the associated warning from the README.
+- Chore(deps): bump certifi in the python-packages group.
+  [dependabot[bot]]
+
+  Bumps the python-packages group with 1 update: [certifi](https://github.com/certifi/python-certifi).
+
+
+  Updates `certifi` from 2024.8.30 to 2024.12.14
+  - [Commits](https://github.com/certifi/python-certifi/compare/2024.08.30...2024.12.14)
+
+  ---
+  updated-dependencies:
+  - dependency-name: certifi
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+
+
+0.47.0 (2024-12-09)
+-------------------
 - Detect empty HTTPS contexts. [John Doe]
 
   Some users are relying solely on the certifi package to provide their CA certs, as requests does this by default.

README.rst

@@ -80,6 +80,7 @@ CLI Help output::
                             log level to use (default: info, possible levels:
                             debug, info, warning, error, critical)
       -i, --incremental     incremental backup
+      --incremental-by-files incremental backup using modified time of files
       --starred             include JSON output of starred repositories in backup
       --all-starred         include starred repositories in backup [*]
       --watched             include JSON output of watched repositories in backup
@@ -167,7 +168,7 @@ Customise the permissions for your use case, but for a personal account full bac
 
 **User permissions**: Read access to followers, starring, and watching.
 
-**Repository permissions**: Read access to code, commit statuses, issues, metadata, pages, pull requests, and repository hooks.
+**Repository permissions**: Read access to contents, issues, metadata, pull requests, and webhooks.
 
 
 Prefer SSH
@@ -225,7 +226,7 @@ Gotchas / Known-issues
 All is not everything
 ---------------------
 
-The ``--all`` argument does not include: cloning private repos (``-P, --private``), cloning forks (``-F, --fork``), cloning starred repositories (``--all-starred``), ``--pull-details``, cloning LFS repositories (``--lfs``), cloning gists (``--starred-gists``) or cloning starred gist repos (``--starred-gists``). See examples for more.
+The ``--all`` argument does not include: cloning private repos (``-P, --private``), cloning forks (``-F, --fork``), cloning starred repositories (``--all-starred``), ``--pull-details``, cloning LFS repositories (``--lfs``), cloning gists (``--gists``) or cloning starred gist repos (``--starred-gists``). See examples for more.
 
 Cloning all starred size
 ------------------------
@@ -239,6 +240,12 @@ Using (``-i, --incremental``) will only request new data from the API **since th
 
 This means any blocking errors on previous runs can cause a large amount of missing data in backups.
 
+Using (``--incremental-by-files``) will request new data from the API **based on when the file was modified on filesystem**. e.g. if you modify the file yourself you may miss something.
+
+Still saver than the previous version.
+
+Specifically, issues and pull requests are handled like this.
+
 Known blocking errors
 ---------------------
 
@@ -254,12 +261,6 @@ It's therefore recommended to only use the incremental argument if the output/re
 
    This is due to needing the correct permission for ``--hooks`` on public repos.
 
-2. **Releases blocking**
-
-   A known ``--releases`` (required for ``--assets``) error will sometimes block the backup. 
-
-   If you're backing up a lot of repositories with releases e.g. an organisation or ``--all-starred``. You may need to remove ``--releases`` (and therefore ``--assets``) to complete a backup. Documented in `issue 209 <https://github.com/josegonzalez/python-github-backup/issues/209>`_.
-
 
 "bare" is actually "mirror"
 ---------------------------

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = "0.47.0"
+__version__ = "0.50.0"

github_backup/github_backup.py

@@ -19,12 +19,12 @@ import ssl
 import subprocess
 import sys
 import time
+from datetime import datetime
 from http.client import IncompleteRead
 from urllib.error import HTTPError, URLError
 from urllib.parse import quote as urlquote
 from urllib.parse import urlencode, urlparse
 from urllib.request import HTTPRedirectHandler, Request, build_opener, urlopen
-from datetime import datetime
 
 try:
     from . import __version__
@@ -40,16 +40,21 @@ logger = logging.getLogger(__name__)
 https_ctx = ssl.create_default_context()
 if not https_ctx.get_ca_certs():
     import warnings
-    warnings.warn('\n\nYOUR DEFAULT CA CERTS ARE EMPTY.\n' +
-                  'PLEASE POPULATE ANY OF:' +
-                  ''.join([
-                    '\n - ' + x
-                    for x in ssl.get_default_verify_paths()
-                    if type(x) is str
-                  ]) + '\n', stacklevel=2)
+
+    warnings.warn(
+        "\n\nYOUR DEFAULT CA CERTS ARE EMPTY.\n"
+        + "PLEASE POPULATE ANY OF:"
+        + "".join(
+            ["\n - " + x for x in ssl.get_default_verify_paths() if type(x) is str]
+        )
+        + "\n",
+        stacklevel=2,
+    )
     import certifi
+
     https_ctx = ssl.create_default_context(cafile=certifi.where())
 
+
 def logging_subprocess(
     popenargs, stdout_log_level=logging.DEBUG, stderr_log_level=logging.ERROR, **kwargs
 ):
@@ -176,6 +181,12 @@ def parse_args(args=None):
         dest="incremental",
         help="incremental backup",
     )
+    parser.add_argument(
+        "--incremental-by-files",
+        action="store_true",
+        dest="incremental_by_files",
+        help="incremental backup based on modification date of files",
+    )
     parser.add_argument(
         "--starred",
         action="store_true",
@@ -795,13 +806,15 @@ def download_file(url, path, auth, as_app=False, fine=False):
     if os.path.exists(path):
         return
 
-    request = _construct_request(per_page=100,
+    request = _construct_request(
+        per_page=100,
         page=1,
         query_args={},
         template=url,
         auth=auth,
         as_app=as_app,
-                                 fine=fine)
+        fine=fine,
+    )
     request.add_header("Accept", "application/octet-stream")
     opener = build_opener(S3HTTPRedirectHandler)
 
@@ -944,11 +957,15 @@ def filter_repositories(args, unfiltered_repositories):
             if r.get("language") and r.get("language").lower() in languages
         ]  # noqa
     if name_regex:
-        repositories = [r for r in repositories if "name" not in r or name_regex.match(r["name"])]
+        repositories = [
+            r for r in repositories if "name" not in r or name_regex.match(r["name"])
+        ]
     if args.skip_archived:
         repositories = [r for r in repositories if not r.get("archived")]
     if args.exclude:
-        repositories = [r for r in repositories if "name" not in r or r["name"] not in args.exclude]
+        repositories = [
+            r for r in repositories if "name" not in r or r["name"] not in args.exclude
+        ]
 
     return repositories
 
@@ -1103,6 +1120,14 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     comments_template = _issue_template + "/{0}/comments"
     events_template = _issue_template + "/{0}/events"
     for number, issue in list(issues.items()):
+        issue_file = "{0}/{1}.json".format(issue_cwd, number)
+        if args.incremental_by_files and os.path.isfile(issue_file):
+            modified = os.path.getmtime(issue_file)
+            modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
+            if modified > issue["updated_at"]:
+                logger.info("Skipping issue {0} because it wasn't modified since last backup".format(number))
+                continue
+
         if args.include_issue_comments or args.include_everything:
             template = comments_template.format(number)
             issues[number]["comment_data"] = retrieve_data(args, template)
@@ -1110,9 +1135,9 @@ def backup_issues(args, repo_cwd, repository, repos_template):
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
 
-        issue_file = "{0}/{1}.json".format(issue_cwd, number)
-        with codecs.open(issue_file, "w", encoding="utf-8") as f:
+        with codecs.open(issue_file + ".temp", "w", encoding="utf-8") as f:
             json_dump(issue, f)
+            os.rename(issue_file + ".temp", issue_file)  # Unlike json_dump, this is atomic
 
 
 def backup_pulls(args, repo_cwd, repository, repos_template):
@@ -1165,6 +1190,13 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
     comments_template = _pulls_template + "/{0}/comments"
     commits_template = _pulls_template + "/{0}/commits"
     for number, pull in list(pulls.items()):
+        pull_file = "{0}/{1}.json".format(pulls_cwd, number)
+        if args.incremental_by_files and os.path.isfile(pull_file):
+            modified = os.path.getmtime(pull_file)
+            modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
+            if modified > pull["updated_at"]:
+                logger.info("Skipping pull request {0} because it wasn't modified since last backup".format(number))
+                continue
         if args.include_pull_comments or args.include_everything:
             template = comments_regular_template.format(number)
             pulls[number]["comment_regular_data"] = retrieve_data(args, template)
@@ -1174,9 +1206,9 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
             template = commits_template.format(number)
             pulls[number]["commit_data"] = retrieve_data(args, template)
 
-        pull_file = "{0}/{1}.json".format(pulls_cwd, number)
-        with codecs.open(pull_file, "w", encoding="utf-8") as f:
+        with codecs.open(pull_file + ".temp", "w", encoding="utf-8") as f:
             json_dump(pull, f)
+            os.rename(pull_file + ".temp", pull_file)  # Unlike json_dump, this is atomic
 
 
 def backup_milestones(args, repo_cwd, repository, repos_template):
@@ -1244,9 +1276,15 @@ def backup_releases(args, repo_cwd, repository, repos_template, include_assets=F
     if args.skip_prerelease:
         releases = [r for r in releases if not r["prerelease"] and not r["draft"]]
 
-    if args.number_of_latest_releases and args.number_of_latest_releases < len(releases):
-        releases.sort(key=lambda item: datetime.strptime(item["created_at"], "%Y-%m-%dT%H:%M:%SZ"),
-                      reverse=True)
+    if args.number_of_latest_releases and args.number_of_latest_releases < len(
+        releases
+    ):
+        releases.sort(
+            key=lambda item: datetime.strptime(
+                item["created_at"], "%Y-%m-%dT%H:%M:%SZ"
+            ),
+            reverse=True,
+        )
         releases = releases[: args.number_of_latest_releases]
         logger.info("Saving the latest {0} releases to disk".format(len(releases)))
     else:
@@ -1274,7 +1312,7 @@ def backup_releases(args, repo_cwd, repository, repos_template, include_assets=F
                         os.path.join(release_assets_cwd, asset["name"]),
                         get_auth(args, encode=not args.as_app),
                         as_app=args.as_app,
-                        fine=True if args.token_fine is not None else False
+                        fine=True if args.token_fine is not None else False,
                     )
 
 

release-requirements.txt

@@ -1,39 +1,39 @@
-autopep8==2.3.1
-black==24.10.0
+autopep8==2.3.2
+black==25.1.0
 bleach==6.2.0
-certifi==2024.8.30
-charset-normalizer==3.4.0
-click==8.1.7
+certifi==2025.1.31
+charset-normalizer==3.4.1
+click==8.1.8
 colorama==0.4.6
 docutils==0.21.2
-flake8==7.1.1
+flake8==7.1.2
 gitchangelog==3.0.4
 idna==3.10
-importlib-metadata==8.5.0
+importlib-metadata==8.6.1
 jaraco.classes==3.4.0
-keyring==25.5.0
+keyring==25.6.0
 markdown-it-py==3.0.0
 mccabe==0.7.0
 mdurl==0.1.2
-more-itertools==10.5.0
+more-itertools==10.6.0
 mypy-extensions==1.0.0
 packaging==24.2
 pathspec==0.12.1
-pkginfo==1.12.0
+pkginfo==1.12.1.2
 platformdirs==4.3.6
 pycodestyle==2.12.1
 pyflakes==3.2.0
-Pygments==2.18.0
+Pygments==2.19.1
 readme-renderer==44.0
 requests==2.32.3
 requests-toolbelt==1.0.0
 restructuredtext-lint==1.4.0
 rfc3986==2.0.0
 rich==13.9.4
-setuptools==75.6.0
+setuptools==75.8.0
 six==1.17.0
 tqdm==4.67.1
-twine==6.0.1
-urllib3==2.2.3
+twine==6.1.0
+urllib3==2.3.0
 webencodings==0.5.1
 zipp==3.21.0