Hey Taylor,
On 11 May 2023, at 20:00, Taylor Blau wrote:
> On Thu, May 11, 2023 at 06:10:32PM +0000, John Cai via GitGitGadget wrote:
>> From: John Cai <johncai86@xxxxxxxxx>
>>
>> At GitLab, we have a system that creates ephemeral internal refs that
>> don't live long before getting deleted. Having an option to exclude
>> certain refs from a packed-refs file allows these internal references to
>> be deleted much more efficiently.
>>
>> Add an --exclude option to the pack-refs builtin, and use the ref
>> exclusions API to exclude certain refs from being packed into the final
>> packed-refs file
>>
>> Signed-off-by: John Cai <johncai86@xxxxxxxxx>
>> ---
>> Documentation/git-pack-refs.txt | 12 +++++++++++-
>> builtin/pack-refs.c | 20 ++++++++++++++++----
>> refs.c | 4 ++--
>> refs.h | 7 ++++++-
>> refs/debug.c | 4 ++--
>> refs/files-backend.c | 16 ++++++++++------
>> refs/packed-backend.c | 2 +-
>> refs/refs-internal.h | 3 ++-
>> revision.h | 2 +-
>> t/helper/test-ref-store.c | 3 ++-
>> t/t3210-pack-refs.sh | 16 ++++++++++++++++
>> 11 files changed, 69 insertions(+), 20 deletions(-)
>>
>> diff --git a/Documentation/git-pack-refs.txt b/Documentation/git-pack-refs.txt
>> index e011e5fead3..c0f7426e519 100644
>> --- a/Documentation/git-pack-refs.txt
>> +++ b/Documentation/git-pack-refs.txt
>> @@ -8,7 +8,7 @@ git-pack-refs - Pack heads and tags for efficient repository access
>> SYNOPSIS
>> --------
>> [verse]
>> -'git pack-refs' [--all] [--no-prune]
>> +'git pack-refs' [--all] [--no-prune] [--exclude <pattern>]
>>
>> DESCRIPTION
>> -----------
>> @@ -60,6 +60,16 @@ interests.
>> The command usually removes loose refs under `$GIT_DIR/refs`
>> hierarchy after packing them. This option tells it not to.
>>
>> +--exclude <pattern>::
>> +
>> +Do not pack refs matching the given `glob(7)` pattern. Repetitions of this option
>> +accumulate exclusion patterns. Use `--no-exclude` to clear and reset the list of
>> +patterns. If a ref is already packed, including it with `--exclude` will not
>> +unpack it.
>> +
>> +When used with `--all`, it will use the difference between the set of all refs,
>> +and what is provided to `--exclude`.
>> +
>
> I think this last paragraph could be simplified, though feel free to
> discard my suggestion if you think it makes things less clear.
>
> When used with `--all`, pack only loose refs which do not match any of
> the provided `--exclude` patterns.
I like the wording here, thanks
>
>> int cmd_pack_refs(int argc, const char **argv, const char *prefix)
>> {
>> unsigned int flags = PACK_REFS_PRUNE;
>> + static struct ref_exclusions excludes = REF_EXCLUSIONS_INIT;
>> + struct pack_refs_opts pack_refs_opts = {.exclusions = &excludes, .flags = flags};
>> + static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP;
>> + struct string_list_item *item;
>
> Since this list does not appear to be sensitive to its order, have you
> considered using the strvec API instead of the string_list one?
Thanks for this suggestion--you're right in that the order doesn't matter here.
The only thing is, the only option parsing macro I could find that operates on
strvec is OPT_PASSTHRU_ARGV. I tried it out, and it seem to work just fine.
>
>> diff --git a/refs.c b/refs.c
>> index d2a98e1c21f..881a0da65cf 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2132,9 +2132,9 @@ void base_ref_store_init(struct ref_store *refs, struct repository *repo,
>> }
>>
>> /* backend functions */
>> -int refs_pack_refs(struct ref_store *refs, unsigned int flags)
>> +int refs_pack_refs(struct ref_store *refs, struct pack_refs_opts *opts)
>> {
>> - return refs->be->pack_refs(refs, flags);
>> + return refs->be->pack_refs(refs, opts);
>> }
>>
>> int peel_iterated_oid(const struct object_id *base, struct object_id *peeled)
>> diff --git a/refs.h b/refs.h
>> index 123cfa44244..46020bd335c 100644
>> --- a/refs.h
>> +++ b/refs.h
>> @@ -63,6 +63,11 @@ struct worktree;
>> #define RESOLVE_REF_NO_RECURSE 0x02
>> #define RESOLVE_REF_ALLOW_BAD_NAME 0x04
>>
>> +struct pack_refs_opts {
>> + unsigned int flags;
>> + struct ref_exclusions *exclusions;
>
> I think this would be OK to include directly in the struct instead of
> via a pointer, but either is fine.
>
>> @@ -1175,15 +1176,18 @@ static void prune_refs(struct files_ref_store *refs, struct ref_to_prune **refs_
>> */
>> static int should_pack_ref(const char *refname,
>> const struct object_id *oid, unsigned int ref_flags,
>> - unsigned int pack_flags)
>> + struct pack_refs_opts *opts)
>> {
>> /* Do not pack per-worktree refs: */
>> if (parse_worktree_ref(refname, NULL, NULL, NULL) !=
>> REF_WORKTREE_SHARED)
>> return 0;
>>
>> + if (opts->exclusions && ref_excluded(opts->exclusions, refname))
>> + return 0;
>
> Looks good, here is where we throw out refs that we don't want. I wonder
> if ref_excluded() does the right thing with a zero-initialized argument
> (i.e. that it behaves as if nothing matches).
Yes, I think we can skip checking if opt->exclusions is not null. Junio had
feedback around this as well.
>
> I wonder if it's possible to skip over certain loose references by
> avoiding traversal into the sub-directories for simple prefixes. That
> may be a premature optimization, though, so I don't think you
> necessarily need to worry about it in this round.
>
>> +test_expect_success 'test excluded refs are not packed' '
>> + git branch dont_pack1 &&
>> + git branch dont_pack2 &&
>> + git branch pack_this &&
>> + git pack-refs --all --exclude "refs/heads/dont_pack*" &&
>> + test -f .git/refs/heads/dont_pack1 &&
>> + test -f .git/refs/heads/dont_pack2 &&
>> + ! test -f ./git/refs/heads/pack_this'
>> +
>> +test_expect_success 'test --no-exclude refs clears excluded refs' '
>> + git branch dont_pack3 &&
>> + git branch dont_pack4 &&
>> + git pack-refs --all --exclude "refs/heads/dont_pack*" --no-exclude &&
>> + ! test -f .git/refs/heads/dont_pack3 &&
>> + ! test -f .git/refs/heads/dont_pack4'
>
> Tests look good. The trailing quote is a little odd to be placed on the
> last line of the test instead of off on its own, but I suppose that is
> imitating existing style, which is OK.
thanks for the feedback!
John
>
> Thanks,
> Taylor
