pack-objects: add --path-walk option for better deltas

[derrickstolee]

This is a reviewable series introducing the path-walk API and its application within the 'git pack-objects --path-walk' machinery. This API was previously discussed in the path-walk RFC [1] and the patch series around the --full-name-hash option (branch ds/pack-name-hash-tweak) [2]. This series conflicts with ds/pack-name-hash-tweak, but that was on hold because it did not seem as critical based on community interest.

[1] https://lore.kernel.org/git/[email protected]

[2] https://lore.kernel.org/git/[email protected]

The primary motivation for this feature is its use to shrink the packfile created by 'git push' when there are many name-hash collisions. This need was discovered in several Javascript repositories that use the beachball tool [3] to generate CHANGELOG.md and CHANGELOG.json files. When a batch of these files are created at the same time and pushed to a release branch, the 'git pack-objects' process has many collisions among these files and delta bases are selected poorly.

[3] https://github.com/microsoft/beachball

In some cases, 'git push' is pushing 60-100 MB when the new path-walk algorithm will identify better delta bases and pack the same objects into a thin pack less than 1 MB. This was the most extreme example we could find and is present in a private monorepo. However, the microsoft/fluentui repo [4] is a good example for demonstrating similar improvements. The patch descriptions frequently refer to this repo and which commit should be checked out to reproduce this behavior.

[4] https://github.com/microsoft/fluentui

The path-walk API is a new way to walk objects. Traditionally, the revision API walks objects by visiting a commit, then visiting its root tree and recursing through trees to visit reachable objects that were not previously visited. The path-walk API visits objects in batches based on the path walked from a root tree to that object. (Only the first discovered path is chosen; this avoids certain kinds of Git bombs.)

This has an immediate application to 'git pack-objects'.

When using the traditional revision API to walk objects, each object is emitted with an associated path. Since this path may appear for many objects spread across the full list, a heuristic is used: the "name-hash" is stored for that object instead of the full path name. This name-hash will group objects at the same path together, but also has a form of "locality" to group likely-similar objects together. When there are few collisions in the name-hash function, this helps compress objects that appear at the same path as well as help compress objects across different paths that have similar suffixes. When there are many versions of the same path, then finding delta bases across that family of objects is very important. When there are few versions of the same path, then finding cross-name delta bases is also important. The former is important for clones and repacks while the latter is important for shallow clones. They can both be important for pushes. In all cases, this approach is fraught when there are many name-hash collisions as the window size becomes a limiting factor for finding quality delta bases.

When using the path-walk API to walk objects, we group objects by the same path from the start. We don't need to store the path name, since we have the objects already in a group. We can compute deltas within that group and then use the name-hash approach to resort the object list and look for opportunistic cross-path deltas. Thus, the path-walk approach allows finding delta bases at least as good as the traditional revision API approach. (Caveat: if we assume delta reuse and the existing deltas were computed with the revision API approach, then the path-walk API approach may result in slightly worse delta compression. The performance tests in this series use --no-reuse-delta for this reason.)

Once 'git pack-objects --path-walk' exists, we have a few ways to take advantage of it in other places:

• The new 'pack.usePathWalk' config option will assume the --path-walk option. This allows 'git push' to assume this value and get the effect we want. This is similar to the 'pack.useSparse' config option that uses a similar path-based walk to limit the set of boundary objects.

• 'git repack' learns a '--path-walk' option to pass to its child 'git pack-objects' process. This is also implied by 'pack.usePathWalk' but allows for testing without modifying repository config.

I'll repeat the following table of repacking results when using 'git repack -adf [--path-walk]' on a set of repositories with many name-hash collisions. Only the microsoft/fluentui repository is publicly available for testing, so the others are left as Repo B/C/D.

| Repo     | Standard Repack | With --path-walk |
|----------|-----------------|------------------|
| fluentui |         438 MB  |          148 MB  |
| Repo B   |       6,255 MB  |          778 MB  |
| Repo C   |      37,737 MB  |        6,158 MB  |
| Repo D   |     130,049 MB  |        4,432 MB  |

While this series is replacing ds/pack-name-hash-tweak and its introduction of the --full-name-hash option, it is worth comparing that option to the --path-walk option.

• The --full-name-hash option is a much smaller code change, as it drops into the existing uses of the name-hash function.

• The --full-name-hash option is more likely to integrate with server-side features such as delta islands and reachability bitmaps due to not changing the object walk. It was already noted that the .bitmap files store name-hash values, so there is some compatibility required to integrate with bitmaps. The --path-walk option will be more difficult to integrate with those options (at least during a repack), but maybe is not impossible; the --path-walk option will not work when reading reachability bitmaps, since we are avoiding walking trees entirely.

• The --full-name-hash option is good when there are many name-hash collisions and many versions of the paths with those collisions. When creating a shallow clone or certain kinds of pushes, the --full-name-hash option is much worse at finding cross-path delta bases since it loses the locality of the standard name-hash function. The --path-walk option includes a second pass of delta computation using the standard name-hash function and thus finds good cross-path delta bases when they improve upon the same-path delta bases.

There are a few differences from the RFC version of this series:

1. The last two patches refactor the approach to perform delta calculations by path after the object walk and then allows those delta calculations to happen in a threaded manner.

2. Both 'git pack-objects' and 'git repack' are removed from the t0450 exclusion list.

3. The path-walk API has improved technical documentation that is extended as its functionality is expanded.

4. Various bugs have been patched with matching tests. The new 'test-tool path-walk' helper allows for careful testing of the API separately from its use within other commands.

Updates in v2

I'm sending this v2 to request some review feedback on the series. I'm sorry it's so long.

There are two updates in this version:

• Fixed a performance issue in the presence of many annotated tags. This is caught by p5313 when run on a repo with 10,000+ annotated tags.
• The Scalar config was previously wrong and should be pack.usePathWalk, not push.usePathWalk.

Thanks, - Stolee

cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: Jonathan Tan [email protected]

[derrickstolee]

/submit

[gitgitgadget]

Submitted as [email protected]

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-1813/derrickstolee/path-walk-upstream-v1

To fetch this version to local tag pr-1813/derrickstolee/path-walk-upstream-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-1813/derrickstolee/path-walk-upstream-v1

[gitgitgadget]

This patch series was integrated into seen via git@b0017d4.

[gitgitgadget]

This patch series was integrated into seen via git@e1dc636.

[gitgitgadget]

This branch is now known as ds/path-walk.

[gitgitgadget]

This patch series was integrated into seen via git@be8d416.

[gitgitgadget]

This patch series was integrated into seen via git@5ddc828.

[gitgitgadget]

There was a status update in the "Cooking" section about the branch ds/path-walk on the Git mailing list:

A new algorithm for object graph traversal to favor visiting the
objects at the same tree path in succession (as opposed to visiting
objects that are different between trees as we walk commit
histories) is introduced to optimize object packing.

Needs review.
source: <[email protected]>

[gitgitgadget]

This patch series was integrated into seen via git@7d1078a.

[gitgitgadget]

This patch series was integrated into seen via git@24e32b0.

[gitgitgadget]

There was a status update in the "Cooking" section about the branch ds/path-walk on the Git mailing list:

A new algorithm for object graph traversal to favor visiting the
objects at the same tree path in succession (as opposed to visiting
objects that are different between trees as we walk commit
histories) is introduced to optimize object packing.

Needs review.
source: <[email protected]>

[gitgitgadget]

This patch series was integrated into seen via git@542db49.

[gitgitgadget]

This patch series was integrated into seen via git@33f243d.

[gitgitgadget]

This patch series was integrated into seen via git@1a84029.

[gitgitgadget]

This patch series was integrated into seen via git@f497f37.

[gitgitgadget]

This patch series was integrated into seen via git@5f6571b.

[derrickstolee]

/submit

[gitgitgadget]

Submitted as [email protected]

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-1813/derrickstolee/path-walk-upstream-v2

To fetch this version to local tag pr-1813/derrickstolee/path-walk-upstream-v2:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-1813/derrickstolee/path-walk-upstream-v2

[gitgitgadget]

On the Git mailing list, Taylor Blau wrote (reply to this):

On Sun, Oct 20, 2024 at 01:43:13PM +0000, Derrick Stolee via GitGitGadget wrote:
> Updates in v2
> =============
>
> I'm sending this v2 to request some review feedback on the series. I'm sorry
> it's so long.
>
> There are two updates in this version:
>
>  * Fixed a performance issue in the presence of many annotated tags. This is
>    caught by p5313 when run on a repo with 10,000+ annotated tags.
>  * The Scalar config was previously wrong and should be pack.usePathWalk,
>    not push.usePathWalk.

Thanks. I queued the new round. As an aside, I would like to find the
time to review this series in depth, but haven't been able to do so
while also trying to keep up with the volume of the rest of the list.

I know that this topic was split out of a larger one. It may be worth
seeing if there is a way to split this topic out into multiple series
that are more easily review-able, but still sensible on their own.

I haven't looked in enough depth to know myself whether such a cut
exists, but it is worth thinking about if you haven't done so already.

Thanks,
Taylor

[gitgitgadget]

This patch series was integrated into seen via git@a3cf0fb.

[gitgitgadget]

This patch series was integrated into seen via git@900e2f6.

[gitgitgadget]

This patch series was integrated into seen via git@05f5018.

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 10/21/24 5:43 PM, Taylor Blau wrote:
> On Sun, Oct 20, 2024 at 01:43:13PM +0000, Derrick Stolee via GitGitGadget wrote:
>> Updates in v2
>> =============
>>
>> I'm sending this v2 to request some review feedback on the series. I'm sorry
>> it's so long.
>>
>> There are two updates in this version:
>>
>>   * Fixed a performance issue in the presence of many annotated tags. This is
>>     caught by p5313 when run on a repo with 10,000+ annotated tags.
>>   * The Scalar config was previously wrong and should be pack.usePathWalk,
>>     not push.usePathWalk.
> > Thanks. I queued the new round. As an aside, I would like to find the
> time to review this series in depth, but haven't been able to do so
> while also trying to keep up with the volume of the rest of the list.
> > I know that this topic was split out of a larger one. It may be worth
> seeing if there is a way to split this topic out into multiple series
> that are more easily review-able, but still sensible on their own.

I'll see what I can do. I needed to re-roll after discovering an issue
when trying to integrate the algorithm with shallow clones. The solution
ends up simplifying the code, which is nice.

> I haven't looked in enough depth to know myself whether such a cut
> exists, but it is worth thinking about if you haven't done so already.

In the current series, there's a natural cut between patches 1-4
and the rest, if we want to put the API in without a non-test consumer.

I could also split out the 'git repack' changes into a third series.

Finally, the threading implementation could be done separately, but I
think it's not complicated enough to leave out from the first version
of the --path-walk option in 'git pack-objects'.

Thanks,
-Stolee

[gitgitgadget]

On the Git mailing list, Taylor Blau wrote (reply to this):

On Thu, Oct 24, 2024 at 09:29:02AM -0400, Derrick Stolee wrote:
> > I know that this topic was split out of a larger one. It may be worth
> > seeing if there is a way to split this topic out into multiple series
> > that are more easily review-able, but still sensible on their own.
>
> I'll see what I can do. I needed to re-roll after discovering an issue
> when trying to integrate the algorithm with shallow clones. The solution
> ends up simplifying the code, which is nice.

It's always nice when that happens :-).

Should we avoid reviewing the current round in anticipation of a
somewhat restructured series, or would you like us to review the current
round as well?

> > I haven't looked in enough depth to know myself whether such a cut
> > exists, but it is worth thinking about if you haven't done so already.
>
> In the current series, there's a natural cut between patches 1-4
> and the rest, if we want to put the API in without a non-test consumer.
>
> I could also split out the 'git repack' changes into a third series.
>
> Finally, the threading implementation could be done separately, but I
> think it's not complicated enough to leave out from the first version
> of the --path-walk option in 'git pack-objects'.

I'd suggest erring on the side of more smaller series rather than a
single large one. If you feel like there are cut points where we can
review them in isolation and still see some benefit, or at least
clearly how they each fit into the larger puzzle, I think that is worth
doing.

But I trust your judgement here, so if you think that the series is best
reviewed as a whole, then that's fine too. Just my $.02 :-).

Thanks,
Taylor

[gitgitgadget]

This patch series was integrated into seen via git@d8f1bc3.

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 10/29/24 6:16 PM, Taylor Blau wrote:
> On Tue, Oct 29, 2024 at 02:36:57PM -0700, Jonathan Tan wrote:
>> By incompatibility, do you mean the incompatibility between bitmaps
>> and the overall --path-walk feature as implemented collectively by the
>> patches in Stolee's patch set? If so, I suspect that we will need a
>> parallel code path that takes in the "want" and "uninteresting" commits
>> and emits the list of objects (possibly before sorting the objects by
>> path hash), much like in builtin/pack-objects.c, so I think there will
>> be some effort involved in making the two work together.
> > I am not sure yet, in all honesty, because I haven't had enough time to
> spend yet reviewing these patches to have anything intelligent to say.

I think that the --path-walk option is fundamentally incompatible with
the --use-bitmap-index option (using reachability bitmaps to reduce how
many objects we parse to discover reachability) but is not necessarily
incompatible with writing bitmaps. But it would require testing to be
sure that there are no surprises due to something like an object order
changing or something like that.

The feature is also not currently integrated with delta islands, so
that would need integration and testing to make sure things are
grouped together and delta chains go in the right direction.

These things may be something possible to overcome in the future, but
the lack of current integration points and testing makes me want to
leave this version with guard rails that prevent users from getting
into a bind.

Thanks,
-Stolee

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 10/28/24 3:54 PM, Jonathan Tan wrote:
> "Derrick Stolee via GitGitGadget" <[email protected]> writes:
>> This new walk is incompatible with some features and is ignored by
>> others:
>>
>>   * Object filters are not currently integrated with the path-walk API,
>>     such as sparse-checkout or tree depth. A blobless packfile could be
>>     integrated easily, but that is deferred for later.
>>
>>   * Server-focused features such as delta islands, shallow packs, and
>>     using a bitmap index are incompatible with the path-walk API.
>>
>>   * The path walk API is only compatible with the --revs option, not
>>     taking object lists or pack lists over stdin. These alternative ways
>>     to specify the objects currently ignores the --path-walk option
>>     without even a warning.
> > It might be better to declare --path-walk as "internal use only" and
> only supporting what send-pack.c (used by "git push") and "git repack"
> needs. (From this list, it seems that there is a lot of incompatibility,
> some of which can happen without a warning to the user, so it sounds
> better to be up-front and say that we only support what send-pack.c
> needs. This also makes reviewing easier, as we don't have to think about
> the possible interactions with every other rev-list feature - only what
> is used by send-pack.c.)

I do wonder what the value of doing this would be. I consider 'gitpack-objects' to already be a plumbing command, so marking any option
as "internal use only" seems like overkill. It takes effort to combine
the options carefully for the right effect. The tests in p5313 are not
terribly simple, such as needing --no-reuse-delta to guarantee we are
using the desired delta algorithm.

> Also from a reviewer perspective, it might be better to restrict this
> patch set to what send-pack.c needs and leave "git repack" for a future
> patch set. This means that we would not need features such as blob
> and tree exclusions, and possibly not even bitmap use or delta reuse
> (assuming that the user would typically push recently-created objects
> that have not been repacked).

While I can understand that as being a potential place to split the
patch series, the integration to add 'git repack --path-walk' is actually
very simple. Repacking "everything" needs to happen to be able to push a
repo to an empty remote, after all.

There are some subtleties around indexed objects, reflogs, and the like
that add some complexity, but they also are handled in the path-walk API
layer. Some of that complexity was helpful to know about during repack
tests.

Finally, the 'git repack --path-walk' use case is a great one for
demonstrating the benefits to threading the 'git pack-objects
--path-walk' algorithm.

>> +		/* Skip objects that do not exist locally. */
>> +		if (exclude_promisor_objects &&
>> +		    oid_object_info_extended(the_repository, oid, &oi,
>> +					     OBJECT_INFO_FOR_PREFETCH) < 0)
>> +			continue;
> > This functionality is typically triggered by --missing=allow;
> --exclude_promisor_objects means (among other things) that we allow
> a missing link only if that object is known to be a promisor object
> (because another promisor object refers to it) (see Documentation/
> rev-list-options.txt, and also get_reference() and elsewhere in
> revision.c - notice how is_promisor_object() is paired with it)
> > Having said that, we should probably just fail outright on missing
> objects, whether or not we have exclude_promisor_objects. If we have
> computed that we need to push an object, that object needs to exist.
> (Same for repack.)

I think that this is not a reasonable assumption that a hard fail
should be expected. Someone could create a blobless clone with a
sparse-checkout and then add a new file outside their sparse-checkout
without ever having the previous version downloaded.

When pushing, they won't be able to use the previous version as a
delta base, but they would certainly be confused about an object
being downloaded during 'git push'.

While the example I bring up is somewhat contrived, I can easily
imagine cases where missing objects are part of the commit boundary
and could be marked as UNINTERESTING but would still be sent in the
batch to be considered as a delta base.

Thanks,
-Stolee

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this), regarding f8ee11d (outdated):

On 10/29/24 2:07 PM, Taylor Blau wrote:
> On Mon, Oct 28, 2024 at 12:54:04PM -0700, Jonathan Tan wrote:

> Is the thinking there that we care mostly about 'git push' and 'git
> repack' on the client-side?
> > I don't think it's unreasonable necessarily, but I would add that
> client-side users definitely do use bitmaps (though not delta islands),
> either when working in a bare repository (where bitmaps are the default)
> or when using 'git gc' (and/or through 'git maintenance') when
> 'repack.writeBitmaps' is enabled.
I suppose some users do use bitmaps, but in my experience, client-side
pushes are slower with bitmaps because a typical target branch is
faster to compute by doing a commit walk, at least when the bitmaps are
older than the new commits in the topic branch. This may be outdated by
now, as it has been a few years since I did a client-side test of
bitmaps.

Thanks,
-Stolee

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 10/29/24 2:02 PM, Taylor Blau wrote:
> On Mon, Oct 28, 2024 at 03:46:11PM -0400, Derrick Stolee wrote:
>> On 10/28/24 1:25 PM, Taylor Blau wrote:
>>> Unfortunately, there is no easy way to reuse the format of the existing
>>> hashcache extension as-is to indicate to the reader whether they are
>>> recording traditional name-hash values, or the new --path-walk hash
>>> values.
>>
>> The --path-walk option does not mess with the name-hash. You're thinking
>> of the --full-name-hash feature [1] that was pulled out due to a lack of
>> interest (and better results with --path-walk).
>>
>> [1] https://lore.kernel.org/git/[email protected]/
>
> Ah, gotcha. Thanks for clarifying.
>
> What is the incompatibility between the two, then? Is it just that
> bitmaps give us the objects in pack- or pseudo-pack order, and we don't
> have a way to permute that back into the order that --path-walk would
> give us?

The incompatibility of reading bitmaps and using the path-walk API is
that the path-walk API does not check a bitmap to see if an object is
already discovered. Thus, it does not use the reachability information
from the bitmap at all and would parse commits and trees to find the
objects that should be in the pack-file.

It should also be worth noting that using something like 'git repack
--path-walk' does not mean that future 'git pack-objects' executions
from that packfile data need to use the --path-walk option. I expect
that it should be painless to write bitmaps on top of a packfile created
with 'git repack -adf --path-walk', but since most places doing so also
likely want delta islands, I have not explored this option thoroughly.

(Delta islands are their own challenge, since the path-walk API is not
spreading the reachability information across the objects it walks.
However, this could be remedied by doing a separate walk to identify
islands using the normal method. I believe Peff had an idea in that
direction in another thread. This requires some integration and testing
that I don't have the expertise to provide.)

> If so, a couple of thoughts:
> ...

Since the incompatibility is in a different direction, I don't think
these thoughts were relevant to the problem.

> OTOH, the order in which we pack objects is extremely important to
> performance as you no doubt are aware of. So changing that order to more
> closely match the --path-walk option should be done with great care.

This is a place where I'm unsure about how the --path-walk option adjusts
the object order within the pack. The packing list gets resorted to match
the typical method, at least for how the delta compression window works.

This would be another good reason to consider the --path-walk option in
server environments very carefully. My patch series puts up guard rails
specifically because it makes no claim to be effective in all of the
dimensions that matter for those scenarios. Hopefully, others will be
motivated enough to determine if the compression that's possible with
this algorithm could be achieved in a way that is compatible with server
needs.

> Anyway. All of that is to say that I want to better understand what does
> and doesn't work together between bitmaps and path-walk. Given my
> current understanding, it seems there are a couple of approaches to
> unifying these two things together, so it would be nice to be able to
> do so if possible.

I think this is an excellent opportunity for testing and debugging to
build up more intuition with how the path-walk API works. When I submit
the next version later tonight, the path-walk algorithm will be better
documented.

That said, I don't have any personal motivation to integrate the two
together, so I don't expect to be contributing that integration point
myself. I think that the results speak for themselves in the very
common environment of a Git client without bitmaps.

Thanks,
-Stolee

[gitgitgadget]

On the Git mailing list, Taylor Blau wrote (reply to this), regarding f8ee11d (outdated):

On Wed, Oct 30, 2024 at 10:14:08PM -0400, Derrick Stolee wrote:
> On 10/29/24 2:07 PM, Taylor Blau wrote:
> > On Mon, Oct 28, 2024 at 12:54:04PM -0700, Jonathan Tan wrote:
>
> > Is the thinking there that we care mostly about 'git push' and 'git
> > repack' on the client-side?
> >
> > I don't think it's unreasonable necessarily, but I would add that
> > client-side users definitely do use bitmaps (though not delta islands),
> > either when working in a bare repository (where bitmaps are the default)
> > or when using 'git gc' (and/or through 'git maintenance') when
> > 'repack.writeBitmaps' is enabled.
> I suppose some users do use bitmaps, but in my experience, client-side
> pushes are slower with bitmaps because a typical target branch is
> faster to compute by doing a commit walk, at least when the bitmaps are
> older than the new commits in the topic branch. This may be outdated by
> now, as it has been a few years since I did a client-side test of
> bitmaps.

All true, though it's hard to estimate the size of "some". I share your
intuition that bitmaps are often a drag on performance for the
client-side because doing a pure commit walk is often faster, especially
if the client has a reasonably up-to-date commit graph.

Thanks,
Taylor

[gitgitgadget]

On the Git mailing list, Taylor Blau wrote (reply to this):

On Wed, Oct 30, 2024 at 10:28:22PM -0400, Derrick Stolee wrote:
> On 10/29/24 2:02 PM, Taylor Blau wrote:
> > On Mon, Oct 28, 2024 at 03:46:11PM -0400, Derrick Stolee wrote:
> >> On 10/28/24 1:25 PM, Taylor Blau wrote:
> >>> Unfortunately, there is no easy way to reuse the format of the existing
> >>> hashcache extension as-is to indicate to the reader whether they are
> >>> recording traditional name-hash values, or the new --path-walk hash
> >>> values.
> >>
> >> The --path-walk option does not mess with the name-hash. You're thinking
> >> of the --full-name-hash feature [1] that was pulled out due to a lack of
> >> interest (and better results with --path-walk).
> >>
> >> [1] https://lore.kernel.org/git/[email protected]/
> >
> > Ah, gotcha. Thanks for clarifying.
> >
> > What is the incompatibility between the two, then? Is it just that
> > bitmaps give us the objects in pack- or pseudo-pack order, and we don't
> > have a way to permute that back into the order that --path-walk would
> > give us?
>
> The incompatibility of reading bitmaps and using the path-walk API is
> that the path-walk API does not check a bitmap to see if an object is
> already discovered. Thus, it does not use the reachability information
> from the bitmap at all and would parse commits and trees to find the
> objects that should be in the pack-file.

Sure, I think what I'm trying to understand here is whether this
"incapability" is a fundamental one, or just that we haven't implemented
checking bitmaps in the path-walk API yet.

Thanks,
Taylor