Merge gitdb and smmap into the GitPython repository by Byron · Pull Request #2179 · gitpython-developers/GitPython

Byron · 2026-07-21T05:20:13Z

Tasks

This section is for Byron only. Models continuing this PR must not add, remove, check, uncheck, rename, or reorder checkboxes here.

refackiew

Everything below this line was generated by Codex GPT-5.

Created by Codex on behalf of Byron. Byron will review before this is ready to merge.

Reported issue

Use this information to plan for a subtree merge, combining gitdb and ssmap repositories into GitPython.

The requested ssmap repository is interpreted as smmap, matching the named GitPython dependency and the source repository discussed in the linked issues.

Summary

Merge the complete smmap history at 8ce61ad5cc4016bffaf25080bc0d69b3acbe8555 into smmap/.
Merge the complete gitdb history at 2da3232f9d58e7761e384ac6d32f7b1ed77a74a2 into gitdb/.
Replace the former gitlinks with regular monorepo directories while retaining separate git, gitdb, and smmap distributions and import APIs.
Teach local tests, packaging, documentation, and CI to consume the in-tree dependency projects.
Leave repository archival, issue migration, and release-process consolidation for separate follow-up work.

Context

This implements the subtree direction discussed in GitPython issues #511, #933, and #1716, gitdb issue #116, and smmap pull request #53.

Validation

GitPython focused fixture, installation, historical-submodule, repository, and documentation tests
smmap: 9 passed
gitdb: 24 passed, 1 skipped
GitPython broader run: 689 passed, 73 skipped, 1 xfailed; remaining failures were local Python 3.14 or Git configuration/environment failures unrelated to the subtree migration
GitPython, gitdb, and smmap wheel and sdist builds
mypy for GitPython
dependency syntax checks and workflow YAML parsing
no tracked gitlinks and clean git diff --check

Change force_bytes to use UTF-8 encoding by default

gitpython-developers/smmap#44

Tell PyPI long_description is Markdown

Mitigates pypa/setuptools#1440

…pped-python-2-support Remove setup.cfg to have a pure python wheel

[skip CI]

Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>

…-smmap Publish updated package metadata by bumping smmap to 5.0.3

Bumps [gitdb/ext/smmap](https://github.com/gitpython-developers/smmap) from `5ec977a` to `e4ad410`. - [Release notes](https://github.com/gitpython-developers/smmap/releases) - [Commits](gitpython-developers/smmap@5ec977a...e4ad410) --- updated-dependencies: - dependency-name: gitdb/ext/smmap dependency-version: e4ad410 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

…es/gitdb/ext/smmap-e4ad410 Bump gitdb/ext/smmap from `5ec977a` to `e4ad410`

Python 3.14 emits a DeprecationWarning for codecs.open(), which gitdb hits inside ReferenceDB._update_dbs_from_ref_file: DeprecationWarning: codecs.open() is deprecated. Use open() instead. The built-in open() has supported the encoding kwarg since Python 3.0 and the call site already passes encoding="utf-8", so the replacement is byte-for-byte equivalent on every supported Python version. Dropped the now-unused codecs import. Verified the change with the existing test_ref.py suite. Closes#128

fix: replace deprecated codecs.open with built-in open (#128)

Closes#120 DecompressMemMapReader.read(N) could return b'' mid-stream when zlib consumed input without producing output on a single decompress call (small N, header / dictionary frames in flight). The original `if dcompdat and ...` guard at the recursion site skipped the "refill to size" recursion in that case, so a caller using the standard idiom while chunk := stream.read(4096): yield chunk terminated at the first empty chunk -- before _br == _s. The guard exists for compressed_bytes_read(), which manipulates _br=0 and then drains the inner zip past its EOF. Recursing there would loop forever because the inner zip is already done. The fix uses zlib's own `eof` attribute (available on standard zlib.Decompress objects since Python 3.6) to distinguish: - dcompdat empty AND zip not at EOF -> still digesting, recurse - dcompdat empty AND zip at EOF -> compressed_bytes_read scrub or genuine EOF; do not recurse. `getattr(_zip, 'eof', False)` keeps the conservative behavior when running against a custom zlib object that does not expose the attribute. Adds a regression test that reads with chunk_size in {1, 4, 16, 64} from a 13 KB highly-compressible stream. With the old guard, the chunk_size <= 16 cases stopped at byte 0; the new test asserts they read all 13000 bytes. The full existing test suite (24 tests) still passes, including test_decompress_reader_special_case and test_pack which exercise the compressed_bytes_read scrub path that the original guard existed to protect.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Fix DecompressMemMapReader.read returning b'' before EOF (#120)

Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

…tions/actions/checkout-7 Bump actions/checkout from 6 to 7

Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

…ctions/actions/checkout-7 Bump actions/checkout from 6 to 7

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

…ctions/actions/setup-python-7 Bump actions/setup-python from 6 to 7

git-subtree-dir: smmap git-subtree-mainline: faf3c09 git-subtree-split: 8ce61ad

git-subtree-dir: gitdb git-subtree-mainline: b3d9e00 git-subtree-split: 2da3232

Byron · 2026-07-25T08:22:54Z

@EliahKagan While I reviewed it hunk by hunk, the change also touches dependabot, and it's unclear if it will still pick up its work for the now 2 additional top-level packages. Let's forward-fix this if needed.

Also, I have never tried to release and would hope GitPython is unaffected, but I will find out. Also GitDB and SMMap releases are uncertain now, but if there is an issue I think it's easy to fix as well, maybe even with unified script machinery.

GitPython previously depended on gitdb and smmap through nested git submodules. That made a complete checkout depend on extra repository state and forced related changes to be coordinated across three repositories. Keep both projects as independently buildable distributions under their own top-level directories, while updating GitPython packaging, CI, documentation, and test fixtures to consume the in-tree sources. Remove the obsolete gitlinks and imported repository automation so a normal clone contains all code needed to build and test the three packages. Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>

Copilot

Pull request overview

This pull request vendors the gitdb and smmap dependency projects into the GitPython repository (monorepo layout), and updates GitPython’s tests, packaging, and CI to use the in-tree projects instead of submodules.

Changes:

Removes the gitdb submodule and adds in-tree gitdb/ and smmap/ projects (code, tests, docs, packaging).
Updates GitPython tests/fixtures to reconstruct dependency repos from the merged history for deterministic local cloning/submodule tests.
Adjusts packaging/typing/tooling and CI workflows to install and validate smmap and gitdb from local paths.

Reviewed changes

Copilot reviewed 107 out of 118 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/test_submodule.py	Updates submodule tests to use reconstructed local dependency repo URLs.
test/test_repo.py	Reworks submodule-related repo tests to use a RW fixture repo.
test/test_installation.py	Installs smmap/gitdb and then GitPython in venv for installation validation.
test/test_fun.py	Adds coverage for `tree_entries_from_data` accepting `bytearray`.
test/test_fixture_health.py	Replaces submodule fixture checks with “no gitlinks” and trust checks.
test/test_docs.py	Makes tutorial submodule snippet run against a RW fixture repo and local URLs.
test/test_clone.py	Uses `no_hardlinks=True` to avoid Windows cleanup issues when cloning locally.
test/lib/helper.py	Adds dependency-repo reconstruction helpers for gitdb/smmap from merged history.
smmap/tox.ini	Adds smmap tox configuration for lint/test.
smmap/smmap/util.py	Adds smmap utility mapping/window primitives.
smmap/smmap/test/test_util.py	Adds smmap util tests.
smmap/smmap/test/test_tutorial.py	Adds a tutorial-style test for smmap usage.
smmap/smmap/test/test_mman.py	Adds smmap memory manager tests.
smmap/smmap/test/test_buf.py	Adds smmap buffer tests (including perf-oriented test).
smmap/smmap/test/lib.py	Adds smmap test support utilities.
smmap/smmap/test/init.py	Initializes smmap test package.
smmap/smmap/buf.py	Adds smmap buffer implementation.
smmap/smmap/init.py	Adds smmap package init and re-exports.
smmap/setup.py	Adds smmap packaging entrypoint.
smmap/setup.cfg	Adds smmap flake8 configuration.
smmap/SECURITY.md	Points smmap security reporting to GitPython’s policy.
smmap/README.md	Adds smmap README in-repo.
smmap/MANIFEST.in	Adds smmap manifest.
smmap/Makefile	Adds smmap convenience Makefile targets.
smmap/LICENSE	Adds smmap license text.
smmap/doc/source/tutorial.rst	Vendors smmap tutorial documentation.
smmap/doc/source/intro.rst	Vendors smmap intro documentation.
smmap/doc/source/index.rst	Vendors smmap documentation index.
smmap/doc/source/conf.py	Vendors smmap Sphinx config.
smmap/doc/source/changes.rst	Vendors smmap changelog.
smmap/doc/source/api.rst	Vendors smmap API docs.
smmap/doc/Makefile	Vendors smmap doc Makefile.
smmap/doc/make.bat	Vendors smmap doc Windows build script.
smmap/doc/.gitignore	Adds smmap doc build ignores.
smmap/.gitignore	Adds smmap repo ignores.
smmap/.coveragerc	Adds smmap coverage configuration.
setup.py	Restricts root package discovery to `git` packages for GitPython distribution.
README.md	Updates dev instructions for editable installs in the monorepo.
pyproject.toml	Updates mypy/pyright/ruff excludes and paths for in-tree gitdb/smmap.
init-tests-after-clone.sh	Removes submodule initialization step (no longer applicable).
gitdb/setup.py	Adds gitdb packaging entrypoint.
gitdb/SECURITY.md	Points gitdb security reporting to GitPython’s policy.
gitdb/requirements.txt	Adds gitdb runtime dependency constraints for smmap.
gitdb/README.rst	Adds gitdb README in-repo.
gitdb/MANIFEST.in	Adds gitdb manifest.
gitdb/Makefile	Adds gitdb convenience Makefile targets.
gitdb/LICENSE	Adds gitdb license text.
gitdb/gitdb/utils/encoding.py	Adds gitdb encoding helpers.
gitdb/gitdb/utils/init.py	Initializes gitdb utils package.
gitdb/gitdb/typ.py	Adds gitdb type constants.
gitdb/gitdb/test/test_util.py	Adds gitdb utility tests.
gitdb/gitdb/test/test_stream.py	Adds gitdb stream tests (incl. regression coverage).
gitdb/gitdb/test/test_pack.py	Adds gitdb pack tests.
gitdb/gitdb/test/test_example.py	Adds gitdb example/tutorial tests.
gitdb/gitdb/test/test_base.py	Adds gitdb base type tests.
gitdb/gitdb/test/performance/test_stream.py	Vendors gitdb performance stream tests.
gitdb/gitdb/test/performance/test_pack.py	Vendors gitdb performance pack tests.
gitdb/gitdb/test/performance/test_pack_streaming.py	Vendors gitdb pack streaming performance tests.
gitdb/gitdb/test/performance/lib.py	Adds gitdb performance test base.
gitdb/gitdb/test/performance/init.py	Initializes gitdb performance test package.
gitdb/gitdb/test/lib.py	Adds gitdb test utilities and fixtures.
gitdb/gitdb/test/db/test_ref.py	Adds gitdb ReferenceDB tests.
gitdb/gitdb/test/db/test_pack.py	Adds gitdb PackedDB tests.
gitdb/gitdb/test/db/test_mem.py	Adds gitdb MemoryDB tests.
gitdb/gitdb/test/db/test_loose.py	Adds gitdb LooseObjectDB tests.
gitdb/gitdb/test/db/test_git.py	Adds gitdb GitDB tests.
gitdb/gitdb/test/db/lib.py	Adds shared gitdb DB test helpers.
gitdb/gitdb/test/db/init.py	Initializes gitdb DB test package.
gitdb/gitdb/test/init.py	Initializes gitdb test package.
gitdb/gitdb/exc.py	Adds gitdb exception types.
gitdb/gitdb/db/ref.py	Adds gitdb ReferenceDB implementation.
gitdb/gitdb/db/pack.py	Adds gitdb PackedDB implementation.
gitdb/gitdb/db/mem.py	Adds gitdb MemoryDB implementation.
gitdb/gitdb/db/loose.py	Adds gitdb LooseObjectDB implementation.
gitdb/gitdb/db/git.py	Adds gitdb GitDB compound implementation.
gitdb/gitdb/db/base.py	Adds gitdb DB base interfaces/compound DB logic.
gitdb/gitdb/db/init.py	Re-exports gitdb db modules.
gitdb/gitdb/const.py	Adds gitdb constants.
gitdb/gitdb/base.py	Adds gitdb core stream/info types.
gitdb/gitdb/init.py	Adds gitdb package init and re-exports.
gitdb/gitdb.pro	Vendors gitdb project metadata file.
gitdb/doc/source/tutorial.rst	Vendors gitdb tutorial docs.
gitdb/doc/source/intro.rst	Vendors gitdb intro docs.
gitdb/doc/source/index.rst	Vendors gitdb docs index.
gitdb/doc/source/conf.py	Vendors gitdb Sphinx config.
gitdb/doc/source/changes.rst	Vendors gitdb changelog.
gitdb/doc/source/api.rst	Vendors gitdb API docs.
gitdb/doc/source/algorithm.rst	Vendors gitdb algorithm discussion docs.
gitdb/doc/Makefile	Vendors gitdb doc Makefile.
gitdb/doc/.gitignore	Adds gitdb doc build ignores.
gitdb/AUTHORS	Adds gitdb authors list.
gitdb/.gitignore	Adds gitdb repo ignores.
gitdb/.coveragerc	Adds gitdb coverage configuration.
git/objects/fun.py	Ensures tree parsing works for `bytearray` by forcing `bytes(...)`.
doc/source/intro.rst	Removes submodule init instructions from GitPython docs.
.pre-commit-config.yaml	Excludes gitdb/smmap paths from pre-commit hooks.
.gitmodules	Removes gitdb submodule configuration.
.github/workflows/pythonpackage.yml	Installs local smmap/gitdb in CI before running GitPython tests.
.github/workflows/dependencies.yml	Adds CI workflow to lint/test smmap and gitdb as standalone packages.
.github/workflows/cygwin-test.yml	Updates Cygwin CI to install local smmap/gitdb and removes submodule trust paths.
.github/workflows/codeql.yml	Ignores gitdb/smmap test paths for CodeQL analysis.
.github/workflows/alpine-test.yml	Updates Alpine CI to install local smmap/gitdb and removes submodule trust paths.
.github/dependabot.yml	Removes submodule update configuration from dependabot.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+ def __contains__(self, sha):
+ return self.has_obj
+


+ print(sys.stderr, "PDB Streaming: Wrote pack of size %i kb in %f s (%f kb/s)" %
+ (total_kb, elapsed, total_kb / (elapsed or 1)), sys.stderr)


@@ -0,0 +1,222 @@
+"""Module containing a memory memory manager which provides a sliding window on a number of memory mapped files"""


-The submodules must have been cloned for that to work, but that will already be the case if you have run `./init-tests-after-clone.sh`. You can use `pip list` to check which packages are installed editably and which are installed normally.
+You can use `pip list` to check which packages are installed editably and which are installed normally.

 To reiterate, this approach should only rarely be used. For most development it is preferable to allow the gitdb and smmap dependencices to be retrieved automatically from PyPI in their latest stable packaged versions.


+ def set_ostream(self, stream):
+ """:raise TypeError: if the stream does not support the Sha1Writer interface"""
+ if stream is not None and not isinstance(stream, Sha1Writer):
+ raise TypeError("Output stream musst support the %s interface" % Sha1Writer.__name__)
+ return super().set_ostream(stream)


Byron · 2026-07-25T12:38:15Z

Ignoring all review comments as this is old code, which won't work worse than before without addressing them.

Byronand others added 30 commits February 17, 2020 11:15

Merge branch 'master' ofhttps://github.com/gitpython-developers/gitdb
b27e72f

Now with PR: 3.0.2
f356e12

Merge branch 'master' into PR-UnicodeEncodeError
d829152

Merge pull request#49from xarx00/PR-UnicodeEncodeError
5699c13
Change force_bytes to use UTF-8 encoding by default

Restrict smmap2 version to <3
02de02c

Update requirements.txt
139811a

Improve changelog for v3.0.2
09aa35b

Merge tag 'v2.0.5'
8e69f21

Change package name to 'smmap'; bump version to 3.0.0
541472d
gitpython-developers/smmap#44

Update makefile to sign with new signature
1b92f4f

v3.0.3.post1
74aef73

Merge pull request#40from adamchainz/markdown
9bca8b2
Tell PyPI long_description is Markdown

Remove carriage returns in setup.py long_description
72698cc
Mitigates pypa/setuptools#1440

v3.0.1
d076f66

v4.0.1
253dfe7

Remove setup.cfg because the gitdb dropped Python 2 support
02d83a8

Merge pull request#62from Nicusor97/remove-setup-cfg-because-it-dro…
0016dae
…pped-python-2-support Remove setup.cfg to have a pure python wheel

v4.0.2
9b03095

Create pythonpackage.yml
77b39e9

Replace travis with github actions
cc21afa

don't try to use __file__ when using pyoxidizer
e6cc702

Add github actions
aa12752

Fix github actions - no dependencies need to be installed
b22f159

Remove basestring reference, no py2 support
2ceedc9

Update badges to represent reality
ebec896

bump patch level
f4d7a58

bump patch level; mark travis-ci as unused
57d3f75

update changelog; remove sublime text (editor) configuration
0002408

Bump patch again; …upload error due to state on disk, yikes
b8ab128

Remove windows build badge
4f08b0a
[skip CI]

chatgpt-codex-connectorBotand others added 17 commits March 9, 2026 11:33

Bump smmap version to 5.0.3 for Python 3.13 metadata release
e4ad410
Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>

Merge pull request#66from gitpython-developers/copilot/fix-issue-65…
10aedd8
…-smmap Publish updated package metadata by bumping smmap to 5.0.3

Merge pull request#133from gitpython-developers/dependabot/submodul…
5c1b303
…es/gitdb/ext/smmap-e4ad410 Bump gitdb/ext/smmap from `5ec977a` to `e4ad410`

Merge pull request#134from mvanhorn/fix/replace-codecs-open-issue-128
53c94d6
fix: replace deprecated codecs.open with built-in open (#128)

Apply suggestions from code review
ed41d2c
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge pull request#135from mvanhorn/fix/120-empty-read-before-eof
0a019a2
Fix DecompressMemMapReader.read returning b'' before EOF (#120)

Merge pull request#67from gitpython-developers/dependabot/github_ac…
8ce61ad
…tions/actions/checkout-7 Bump actions/checkout from 6 to 7

Merge pull request#136from gitpython-developers/dependabot/github_a…
4950ea9
…ctions/actions/checkout-7 Bump actions/checkout from 6 to 7

Merge pull request#137from gitpython-developers/dependabot/github_a…
2da3232
…ctions/actions/setup-python-7 Bump actions/setup-python from 6 to 7

Merge smmap into the GitPython repository
b3d9e00
git-subtree-dir: smmap git-subtree-mainline: faf3c09 git-subtree-split: 8ce61ad

Merge gitdb into the GitPython repository
a0cbcc2
git-subtree-dir: gitdb git-subtree-mainline: b3d9e00 git-subtree-split: 2da3232

github-advanced-securityAI found potential problems Jul 21, 2026
View reviewed changes

Comment threadgitdb/gitdb/test/lib.py Fixed
Comment threadgitdb/gitdb/test/test_pack.py Fixed
Comment threadgitdb/gitdb/test/test_pack.py Fixed
Comment threadgitdb/gitdb/test/test_pack.py Fixed
Comment threadgitdb/gitdb/test/test_util.py Fixed
Comment threadgitdb/gitdb/test/test_util.py Fixed

Byron force-pushed the subtree-merge-deps branch 2 times, most recently from f377fc7 to 7e817cdCompare July 25, 2026 07:56

Byron force-pushed the subtree-merge-deps branch from 7e817cd to 8a08afaCompare July 25, 2026 08:23

Byron marked this pull request as ready for review July 25, 2026 08:47

CopilotAI review requested due to automatic review settings July 25, 2026 08:47

Byron merged commit 8ecda9e into mainJul 25, 2026
50 of 51 checks passed

Copilotstarted reviewing on behalf of Byron July 25, 2026 08:47 View session

Byron deleted the subtree-merge-deps branch July 25, 2026 08:47

CopilotAI reviewed Jul 25, 2026
View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

Merge gitdb and smmap into the GitPython repository - #2179

Merge gitdb and smmap into the GitPython repository#2179
Byron merged 699 commits into
mainfrom
subtree-merge-deps

Byron commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Byron commented Jul 25, 2026•
edited
Loading

Uh oh!

Uh oh!

CopilotAI left a comment

Uh oh!

Byron commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

20 participants

		print(sys.stderr, "PDB Streaming: Wrote pack of size %i kb in %f s (%f kb/s)" %
		(total_kb, elapsed, total_kb / (elapsed or 1)), sys.stderr)

		@@ -0,0 +1,222 @@
		"""Module containing a memory memory manager which provides a sliding window on a number of memory mapped files"""

Uh oh!

Uh oh!

Conversation

Byron commented Jul 21, 2026

Tasks

Reported issue

Summary

Context

Validation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Byron commented Jul 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Byron commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

20 participants

Byron commented Jul 25, 2026•
edited
Loading