Hi,
this patch series implements migration of ref storage formats such that
it is for example possible to migrate a repository with "files" format
to the "reftable" format. The migration logic comes in the form of a new
`git refs migrate` subcommand. As mentioned in [1], I do have plans to
extend the new git-refs(1) over time.
In the current form, the migration logic has three major limitations:
- It does not work with repositories that have worktrees as we have to
migrate multiple ref stores here, one for every worktree. I wanted
to avoid making this series too complex right from the start.
- It does not migrate reflogs, because we have no interfaces to do so.
I want to eventually address this by adding log-only updates to
transactions.
- It is not safe with concurrent writers. This is the limitation that
is most critical in my eyes. The root cause here is that it is
inherently impossible to lock the "files" backend for writes. I have
been thinking about this issue a lot and have not found any solution
that works. There are partial solutions here:
- Create a central lockfile for the "files" backend -- if there,
the backend will refuse to write. If that lock needs to be
acquired during the "commit" phase of transaction though then we
would essentially start to sequentialize all writes in the
"files" backend, which is a non-starter. If not, then processes
which are running already may not have seen it, and thus the
issue still exists.
- Pack all loose refs, remove "refs/" and create a "refs.lock"
file. This isn't safe though because root refs can still be
written.
- Create a log of concurrent writes and apply that to the migrated
refs once done. This is a lot of complexity, and it's unclear
whether it even solves the issue with already-running writers.
- Create a temporary "extensions.refMigration" extension that is
unhandled by Git. New processes will refuse to run in such a
repo and thus cannot write to it. Again, unsafe with running
writers.
Another alternative is that we could just make this a best effort.
The "reftable" backend supports locking, and for the "files" backend
we could just lock "HEAD" and call it a day. I'm not sure whether it
is preferable though to have a "partial" solution compared to having
none at all, as it may cause users to be less mindful. That's why I
decided to just have no solution at all and document the limitation
accordingly.
If anybody has ideas here I'd be very happy to hear them.
Anyway. The current state of this patch series is sufficient to migrate
repos without reflogs and worktrees, and thus mostly applies to bare
repositories, only. This is somewhat intentional -- as a server operator
this is of course our primary usecase at GitLab. We do plan to also
upstream support for writing reflogs though, but in a later step. We do
not plan to implement support for migrating repositories with worktrees,
but I'd be happy to help out with the effort in case somebody else wants
to.
The series is built on top of 4365c6fcf9 (The sixth batch, 2024-05-20).
It pulls in the following two dependencies:
- ps/refs-without-the-repository-updates at 00892786b8 (refs/packed:
remove references to `the_hash_algo`, 2024-05-17). This is mostly to
avoid conflicts.
- ps/pseudo-ref-terminology at 8e4f5c2dc2 (refs: refuse to write
pseudorefs, 2024-05-15). This is a functional prerequisite because
the migration logic relies on the new definition of pseudorefs.
There are two minor textual conflicts when merged to "next" or "seen".
One is a conflicting added header in "refs/reftable-backend.c", and one
is a conflict with added functions in "refs.c". Both of these conflicts
are trivial to solve by accepting both sides.
Thanks!
Patrick
[1]: https://lore.kernel.org/git/ZkNJaaKTTKbns8wo@tanuki/#t
Patrick Steinhardt (9):
setup: unset ref storage when reinitializing repository version
refs: convert ref storage format to an enum
refs: pass storage format to `ref_store_init()` explicitly
refs: allow to skip creation of reflog entries
refs/files: refactor `add_pseudoref_and_head_entries()`
refs/files: extract function to iterate through root refs
refs: implement removal of ref storages
refs: implement logic to migrate between ref storage formats
builtin/refs: new command to migrate ref storage formats
.gitignore | 1 +
Documentation/git-refs.txt | 59 +++++++
Makefile | 1 +
builtin.h | 1 +
builtin/clone.c | 2 +-
builtin/init-db.c | 2 +-
builtin/refs.c | 75 +++++++++
git.c | 1 +
refs.c | 319 +++++++++++++++++++++++++++++++++++--
refs.h | 41 ++++-
refs/files-backend.c | 121 ++++++++++++--
refs/packed-backend.c | 15 ++
refs/refs-internal.h | 7 +
refs/reftable-backend.c | 37 ++++-
repository.c | 3 +-
repository.h | 10 +-
setup.c | 10 +-
setup.h | 9 +-
t/helper/test-ref-store.c | 1 +
t/t1460-refs-migrate.sh | 243 ++++++++++++++++++++++++++++
20 files changed, 913 insertions(+), 45 deletions(-)
create mode 100644 Documentation/git-refs.txt
create mode 100644 builtin/refs.c
create mode 100755 t/t1460-refs-migrate.sh
--
2.45.1.216.g4365c6fcf9.dirty
Attachment:
signature.asc
Description: PGP signature
