Release version 0.49.0

Implementing incremental by files, safer version of incremental backup.

CHANGES.rst

@@ -1,9 +1,123 @@
 Changelog
 =========
 
-0.48.0 (2025-01-04)
+0.49.0 (2025-02-01)
 -------------------
 ------------------------
+- Convert timestamp to string, although maybe the other way around would
+  be better ... [Honza Maly]
+- Implementing incremental by files, safer version of incremental
+  backup. [Honza Maly]
+- Chore(deps): bump the python-packages group across 1 directory with 7
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 7 updates in the / directory:
+
+  | Package | From | To |
+  | --- | --- | --- |
+  | [autopep8](https://github.com/hhatto/autopep8) | `2.3.1` | `2.3.2` |
+  | [black](https://github.com/psf/black) | `24.10.0` | `25.1.0` |
+  | [certifi](https://github.com/certifi/python-certifi) | `2024.12.14` | `2025.1.31` |
+  | [importlib-metadata](https://github.com/python/importlib_metadata) | `8.5.0` | `8.6.1` |
+  | [more-itertools](https://github.com/more-itertools/more-itertools) | `10.5.0` | `10.6.0` |
+  | [setuptools](https://github.com/pypa/setuptools) | `75.7.0` | `75.8.0` |
+  | [twine](https://github.com/pypa/twine) | `6.0.1` | `6.1.0` |
+
+
+
+  Updates `autopep8` from 2.3.1 to 2.3.2
+  - [Release notes](https://github.com/hhatto/autopep8/releases)
+  - [Commits](https://github.com/hhatto/autopep8/compare/v2.3.1...v2.3.2)
+
+  Updates `black` from 24.10.0 to 25.1.0
+  - [Release notes](https://github.com/psf/black/releases)
+  - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
+  - [Commits](https://github.com/psf/black/compare/24.10.0...25.1.0)
+
+  Updates `certifi` from 2024.12.14 to 2025.1.31
+  - [Commits](https://github.com/certifi/python-certifi/compare/2024.12.14...2025.01.31)
+
+  Updates `importlib-metadata` from 8.5.0 to 8.6.1
+  - [Release notes](https://github.com/python/importlib_metadata/releases)
+  - [Changelog](https://github.com/python/importlib_metadata/blob/main/NEWS.rst)
+  - [Commits](https://github.com/python/importlib_metadata/compare/v8.5.0...v8.6.1)
+
+  Updates `more-itertools` from 10.5.0 to 10.6.0
+  - [Release notes](https://github.com/more-itertools/more-itertools/releases)
+  - [Commits](https://github.com/more-itertools/more-itertools/compare/v10.5.0...v10.6.0)
+
+  Updates `setuptools` from 75.7.0 to 75.8.0
+  - [Release notes](https://github.com/pypa/setuptools/releases)
+  - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
+  - [Commits](https://github.com/pypa/setuptools/compare/v75.7.0...v75.8.0)
+
+  Updates `twine` from 6.0.1 to 6.1.0
+  - [Release notes](https://github.com/pypa/twine/releases)
+  - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
+  - [Commits](https://github.com/pypa/twine/compare/6.0.1...6.1.0)
+
+  ---
+  updated-dependencies:
+  - dependency-name: autopep8
+    dependency-type: direct:production
+    update-type: version-update:semver-patch
+    dependency-group: python-packages
+  - dependency-name: black
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+    dependency-group: python-packages
+  - dependency-name: certifi
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+    dependency-group: python-packages
+  - dependency-name: importlib-metadata
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: more-itertools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: setuptools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: twine
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+- Chore(deps): bump the python-packages group across 1 directory with 2
+  updates. [dependabot[bot]]
+
+  Bumps the python-packages group with 2 updates in the / directory: [pygments](https://github.com/pygments/pygments) and [setuptools](https://github.com/pypa/setuptools).
+
+
+  Updates `pygments` from 2.18.0 to 2.19.1
+  - [Release notes](https://github.com/pygments/pygments/releases)
+  - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
+  - [Commits](https://github.com/pygments/pygments/compare/2.18.0...2.19.1)
+
+  Updates `setuptools` from 75.6.0 to 75.7.0
+  - [Release notes](https://github.com/pypa/setuptools/releases)
+  - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
+  - [Commits](https://github.com/pypa/setuptools/compare/v75.6.0...v75.7.0)
+
+  ---
+  updated-dependencies:
+  - dependency-name: pygments
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  - dependency-name: setuptools
+    dependency-type: direct:production
+    update-type: version-update:semver-minor
+    dependency-group: python-packages
+  ...
+
+
+0.48.0 (2025-01-04)
+-------------------
 - Chore: reformat file to fix lint issues. [Jose Diaz-Gonzalez]
 - Chore(deps): bump the python-packages group across 1 directory with 4
   updates. [dependabot[bot]]

README.rst

@@ -80,6 +80,7 @@ CLI Help output::
                             log level to use (default: info, possible levels:
                             debug, info, warning, error, critical)
       -i, --incremental     incremental backup
+      --incremental-by-files incremental backup using modified time of files
       --starred             include JSON output of starred repositories in backup
       --all-starred         include starred repositories in backup [*]
       --watched             include JSON output of watched repositories in backup
@@ -239,6 +240,12 @@ Using (``-i, --incremental``) will only request new data from the API **since th
 
 This means any blocking errors on previous runs can cause a large amount of missing data in backups.
 
+Using (``--incremental-by-files``) will request new data from the API **based on when the file was modified on filesystem**. e.g. if you modify the file yourself you may miss something.
+
+Still saver than the previous version.
+
+Specifically, issues and pull requests are handled like this.
+
 Known blocking errors
 ---------------------
 

github_backup/__init__.py

@@ -1 +1 @@
-__version__ = "0.48.0"
+__version__ = "0.49.0"

github_backup/github_backup.py

@@ -181,6 +181,12 @@ def parse_args(args=None):
         dest="incremental",
         help="incremental backup",
     )
+    parser.add_argument(
+        "--incremental-by-files",
+        action="store_true",
+        dest="incremental_by_files",
+        help="incremental backup based on modification date of files",
+    )
     parser.add_argument(
         "--starred",
         action="store_true",
@@ -1114,6 +1120,14 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     comments_template = _issue_template + "/{0}/comments"
     events_template = _issue_template + "/{0}/events"
     for number, issue in list(issues.items()):
+        issue_file = "{0}/{1}.json".format(issue_cwd, number)
+        if args.incremental_by_files and os.path.isfile(issue_file):
+            modified = os.path.getmtime(issue_file)
+            modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
+            if modified > issue["updated_at"]:
+                logger.info("Skipping issue {0} because it wasn't modified since last backup".format(number))
+                continue
+
         if args.include_issue_comments or args.include_everything:
             template = comments_template.format(number)
             issues[number]["comment_data"] = retrieve_data(args, template)
@@ -1121,9 +1135,9 @@ def backup_issues(args, repo_cwd, repository, repos_template):
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
 
-        issue_file = "{0}/{1}.json".format(issue_cwd, number)
+        with codecs.open(issue_file + ".temp", "w", encoding="utf-8") as f:
-        with codecs.open(issue_file, "w", encoding="utf-8") as f:
             json_dump(issue, f)
+            os.rename(issue_file + ".temp", issue_file) # Unlike json_dump, this is atomic
 
 
 def backup_pulls(args, repo_cwd, repository, repos_template):
@@ -1176,6 +1190,13 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
     comments_template = _pulls_template + "/{0}/comments"
     commits_template = _pulls_template + "/{0}/commits"
     for number, pull in list(pulls.items()):
+        pull_file = "{0}/{1}.json".format(pulls_cwd, number)
+        if args.incremental_by_files and os.path.isfile(pull_file):
+            modified = os.path.getmtime(pull_file)
+            modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
+            if modified > pull["updated_at"]:
+                logger.info("Skipping pull request {0} because it wasn't modified since last backup".format(number))
+                continue
         if args.include_pull_comments or args.include_everything:
             template = comments_regular_template.format(number)
             pulls[number]["comment_regular_data"] = retrieve_data(args, template)
@@ -1185,9 +1206,9 @@ def backup_pulls(args, repo_cwd, repository, repos_template):
             template = commits_template.format(number)
             pulls[number]["commit_data"] = retrieve_data(args, template)
 
-        pull_file = "{0}/{1}.json".format(pulls_cwd, number)
+        with codecs.open(pull_file + ".temp", "w", encoding="utf-8") as f:
-        with codecs.open(pull_file, "w", encoding="utf-8") as f:
             json_dump(pull, f)
+            os.rename(pull_file + ".temp", pull_file) # Unlike json_dump, this is atomic
 
 
 def backup_milestones(args, repo_cwd, repository, repos_template):

release-requirements.txt

@@ -1,7 +1,7 @@
-autopep8==2.3.1
+autopep8==2.3.2
-black==24.10.0
+black==25.1.0
 bleach==6.2.0
-certifi==2024.12.14
+certifi==2025.1.31
 charset-normalizer==3.4.1
 click==8.1.8
 colorama==0.4.6
@@ -9,13 +9,13 @@ docutils==0.21.2
 flake8==7.1.1
 gitchangelog==3.0.4
 idna==3.10
-importlib-metadata==8.5.0
+importlib-metadata==8.6.1
 jaraco.classes==3.4.0
 keyring==25.6.0
 markdown-it-py==3.0.0
 mccabe==0.7.0
 mdurl==0.1.2
-more-itertools==10.5.0
+more-itertools==10.6.0
 mypy-extensions==1.0.0
 packaging==24.2
 pathspec==0.12.1
@@ -23,17 +23,17 @@ pkginfo==1.12.0
 platformdirs==4.3.6
 pycodestyle==2.12.1
 pyflakes==3.2.0
-Pygments==2.18.0
+Pygments==2.19.1
 readme-renderer==44.0
 requests==2.32.3
 requests-toolbelt==1.0.0
 restructuredtext-lint==1.4.0
 rfc3986==2.0.0
 rich==13.9.4
-setuptools==75.6.0
+setuptools==75.8.0
 six==1.17.0
 tqdm==4.67.1
-twine==6.0.1
+twine==6.1.0
 urllib3==2.3.0
 webencodings==0.5.1
 zipp==3.21.0