On December 20, 2024 2:38 PM, Patrick Steinhardt wrote:
>On Fri, Dec 20, 2024 at 02:23:49PM -0500, rsbecker@xxxxxxxxxxxxx wrote:
>> On December 20, 2024 2:08 AM, Patrick Steinhardt wrote:
>> >On Thu, Dec 19, 2024 at 03:46:20PM -0500, rsbecker@xxxxxxxxxxxxx wrote:
>> >> On December 18, 2024 11:07 AM, I wrote:
>> >> >All tests, actually.
>> >> >
>> >> >$ GIT_TEST_DEFAULT_REF_FORMAT=reftable GIT_TEST_CLONE_2GB=true sh
>> >> >t0000- basic.sh --verbose -i -x
>> >> >error: reftable: transaction prepare: out of memory
>> >> >error: cannot run git init
>> >>
>> >> Any updates or hypothesis on this? Our test system has loads of
>> >> memory
>> >> - I cannot figure out where the allocation failure takes place.
>> >> There is a limit to how much memory can be allocated, but it is
>> >> very high and our virtual memory is extensive, but this is a 32-bit
build.
>> >
>> >My hypothesis is that this is caused by ps/reftable-alloc-failures,
>> >but I
>> am unable to
>> >tell where exactly the error comes from. So I'm dependent on your input.
>> >
>> >Could you please bisect the error? Finding out where the error is
>> >raised
>> would also
>> >be quite helpful. It has to be one of the reftable functions that
>> >returns REFTABLE_OUT_OF_MEMORY_ERROR, but other than that I do not
>> >have any more gut feeling right now.
>>
>> This is what bisect shows:
>>
>> git bisect start
>> # status: waiting for both good and bad commits # good:
>> [777489f9e09c8d0dd6b12f9d90de6376330577a2] Git 2.47 git bisect good
>> 777489f9e09c8d0dd6b12f9d90de6376330577a2
>> # status: waiting for bad commit, 1 good commit known # bad:
>> [063bcebf0c917140ca0e705cbe0fdea127e90086] Git 2.48-rc0
>>
>> git bisect bad 063bcebf0c917140ca0e705cbe0fdea127e90086
>> # bad: [2037ca85ad93ec905b46543df6df4080f6ca258b] worktree: refactor
>> `repair_worktree_after_gitdir_move()`
>> git bisect bad 2037ca85ad93ec905b46543df6df4080f6ca258b
>> # bad: [6a11438f43469f3815f2f0fc997bd45792ff04c0] The fifth batch git
>> bisect bad 6a11438f43469f3815f2f0fc997bd45792ff04c0
>> # bad: [f004467b042d735a2fe8bd5706b053b04b1aec65] Merge branch
>> 'jh/config-unset-doc-fix'
>> git bisect bad f004467b042d735a2fe8bd5706b053b04b1aec65
>> # bad: [e29296745dc92fb03f8f60111b458adc69ff84c5] Merge branch
>> 'sk/doc-maintenance-schedule'
>> git bisect bad e29296745dc92fb03f8f60111b458adc69ff84c5
>> # bad: [5b67cc6477ce88c499caab5ebcebd492ec78932d] reftable/stack:
>> handle allocation failures in auto compaction git bisect bad
>> 5b67cc6477ce88c499caab5ebcebd492ec78932d
>> # good: [31f5b972e0231d4211987775dd58e67815734989] reftable/record:
>> handle allocation failures when decoding records git bisect good
>> 31f5b972e0231d4211987775dd58e67815734989
>> # bad: [18da60029319733e2d931f2758a8e47b8b25b117] reftable/reader:
>> handle allocation failures for unindexed reader git bisect bad
>> 18da60029319733e2d931f2758a8e47b8b25b117
>> # good: [74d1c18757d1a45b95e46836adf478193a34c42c] reftable/writer:
>> handle allocation failures in `reftable_new_writer()` git bisect good
>> 74d1c18757d1a45b95e46836adf478193a34c42c
>
>This is missing the last step for git-bisect(1). Right now it could be
caused by one of
>these commits:
>
> 18da600293 (reftable/reader: handle allocation failures for unindexed
reader,
>2024-10-02)
> 802c0646ac (reftable/merged: handle allocation failures in
>`merged_table_init_iter()`, 2024-10-02)
They are there, it is just that Outlook wrapped the lines on me. The
802c064 is not in my repo - I bisected from 2.47.0 to 2.48.0-rc0, so may
have skipped a more recent commit than rc0 has.
>The first commit seems quite unlikely to be the root cause. The second
commit is
>rather interesting though. I wonder whether NonStop's malloc returns a NULL
>pointer when given a size of 0?
>
>A quick stab into the dark, but does below patch on top of `master` make
things
>work for you?
>
>Patrick
>
>-- >8 --
>
>diff --git a/reftable/merged.c b/reftable/merged.c index
bb0836e344..7ae6f78d45
>100644
>--- a/reftable/merged.c
>+++ b/reftable/merged.c
>@@ -244,7 +244,7 @@ int merged_table_init_iter(struct reftable_merged_table
>*mt,
> struct merged_iter *mi = NULL;
> int ret;
>
>- REFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);
>+ REFTABLE_CALLOC_ARRAY(subiters, mt->readers_len + 1);
> if (!subiters) {
> ret = REFTABLE_OUT_OF_MEMORY_ERROR;
> goto out;
>diff --git a/reftable/stack.c b/reftable/stack.c index
59fd695a12..1b6b8cc9ea
>100644
>--- a/reftable/stack.c
>+++ b/reftable/stack.c
>@@ -1612,7 +1612,7 @@ static uint64_t
>*stack_table_sizes_for_compaction(struct reftable_stack *st)
> int overhead = header_size(version) - 1;
> uint64_t *sizes;
>
>- REFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);
>+ REFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len + 1);
> if (!sizes)
> return NULL;
>
The fix above does not appear to make any difference.
