Patrick Steinhardt <ps@xxxxxx> writes: > Hi, > > I have recently learned that ref format migrations can take a > significant amount of time in the order of minutes when migrating > millions of refs. This is probably not entirely surprising: the initial > focus for the logic to migrate ref backends was mostly focussed on > getting the basic feature working, and I didn't yet invest any time into > optimizing the code path at all. But I was still mildly surprised that > the migration of a couple million refs was taking minutes to finish. > > This patch series thus optimizes how we migrate ref formats. This is > mostly done by expanding upon the "initial transaction" semantics that > we already use for git-clone(1). These semantics allow us to assume that > the ref backend is completely empty and that there are no concurrent > writers, and thus we are free to perform certain optimizations that > wouldn't have otherwise been possible. On the one hand this allows us to > drop needless collision checks. On the other hand, it also allows us to > write regular refs directly into the "packed-refs" file when migrating > from the "reftable" backend to the "files" backend. > > This leads to some significant speedups. Migrating 1 million refs from > "files" to "reftable": > > Benchmark 1: migrate files:reftable (refcount = 1000000, revision = origin/master) > Time (mean ± σ): 4.580 s ± 0.062 s [User: 1.818 s, System: 2.746 s] > Range (min … max): 4.534 s … 4.743 s 10 runs > > Benchmark 2: migrate files:reftable (refcount = 1000000, revision = pks-refs-optimize-migrations) > Time (mean ± σ): 767.7 ms ± 9.5 ms [User: 629.2 ms, System: 126.1 ms] > Range (min … max): 755.8 ms … 786.9 ms 10 runs > > Summary > migrate files:reftable (refcount = 1000000, revision = pks-refs-optimize-migrations) ran > 5.97 ± 0.11 times faster than migrate files:reftable (refcount = 1000000, revision = origin/master) > > And migrating from "reftable" to "files: > > Benchmark 1: migrate reftable:files (refcount = 1000000, revision = origin/master) > Time (mean ± σ): 35.409 s ± 0.302 s [User: 5.061 s, System: 29.244 s] > Range (min … max): 35.055 s … 35.898 s 10 runs > > Benchmark 2: migrate reftable:files (refcount = 1000000, revision = pks-refs-optimize-migrations) > Time (mean ± σ): 855.9 ms ± 61.5 ms [User: 646.7 ms, System: 187.1 ms] > Range (min … max): 830.0 ms … 1030.3 ms 10 runs > > Summary > migrate reftable:files (refcount = 1000000, revision = pks-refs-optimize-migrations) ran > 41.37 ± 2.99 times faster than migrate reftable:files (refcount = 1000000, revision = origin/master) > > Thanks! > > Patrick > I read through the series, apart from a few small nits, the patches look good and straightforward. Thanks
Attachment:
signature.asc
Description: PGP signature
