Re: [EXT] Re: ls-remote bug

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Am 27.10.23 um 13:16 schrieb Lior Zeltzer:
> The reproduction ,as I wrote in the code, should be done with few threads in parallel
> Each working on a list of ~10 repos with each repo containing a lot of refs/tags (~1000)
> All this should be against gerrit (my gerrit is 3.8.0)
>
> Also read the notes below regarding the code that was moved between 2.31.8 and 2.32.0 in ls-remote.c file
> I can elaborate more, in a zoom meeting.
>
> 10x
> Lior.
>
>
> -----Original Message-----
> From: Bagas Sanjaya <bagasdotme@xxxxxxxxx>
> Sent: Friday, October 27, 2023 4:08 AM
> To: Lior Zeltzer <liorz@xxxxxxxxxxx>; Git Mailing List <git@xxxxxxxxxxxxxxx>
> Cc: Andrzej Hunt <ajrhunt@xxxxxxxxxx>; Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>; Junio C Hamano <gitster@xxxxxxxxx>
> Subject: [EXT] Re: ls-remote bug
>
> External Email
>
> ----------------------------------------------------------------------
> On Tue, Oct 24, 2023 at 10:55:24AM +0000, Lior Zeltzer wrote:
>>
>>> uname -a
>> Linux dc3lp-veld0045 3.10.0-1160.21.1.el7.x86_64 #1 SMP Tue Mar 16
>> 18:28:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Gerrit version :
>> 3.8.0
>>
>> Bug description :
>> When running ls-remote : sometime data gets cut in the middle
>>
>> Reproducing :
>> You need a few files with a few repo names (I used 4 files with 10
>> repos each) Call then l1..l4 And the code below just cd into each of
>> them does ls-remote twice and compares the data Doing it in parallel
>> on all lists.
>> Data received in both ls-remotes should be the same , if not, it
>> prints *** Repos should contain a lot of tags and refs
>
> What repo did you find this regression? Did you mean linux.git (Linux kernel)?
>
>>
>> Note :
>> 1.  without stderr redirection (2>&1) all works well 2. On local repos
>> (not through gerrit) all works well
>>
>> I compared various git vers and found the bug to be between 2.31.8 and
>> 2.32.0 Comparing ls-remote.c file between those vers gave me :
>>
>> Lines :
>> if (transport_disconnect(transport))
>> 		return 1;
>>
>> moved to end of sub
>>
>> copying ls-remote.c from 2.31.8 to 2.32.0 - fixed the bug

This partly undoes 68ffe095a2 (transport: also free remote_refs in
transport_disconnect(), 2021-03-21).  With that patch connections are
kept open during ref sorting and printing.

Perhaps the other side gets tired of waiting and aborts?  Maybe
splitting transport_disconnect() into two functions -- one for
disconnecting and and for cleaning up -- would make sense? And
disconnecting as soon as possible?  Just guessing -- didn't actually
reproduce the bug.  Still, demo patch below.

>>
>>
>>
>> Code reproducing bug :
>>
>> #!/proj/mislcad/areas/DAtools/tools/perl/5.10.1/bin/perl -w use
>> strict; use Cwd qw(cwd);
>>
>> my $count = 4;
>> for my $f (1..$count) {
>>   my $child = fork();
>>   if (!$child) {
>>     my $curr = cwd();
>>    
>>     my @repos = `cat l$f`;
>>     foreach my $repo (@repos) {
>>       chomp $repo;
>>       print "$repo\n";
>>       chdir($repo);
>>       my $remote_tags_str = `git ls-remote  2>&1`;
>>       my $remote_tags_str2 = `git ls-remote  2>&1 `;
>>       chdir($curr);
>>       if ( $remote_tags_str ne $remote_tags_str2) {
>>          print "***\n";
>>       }
>>     }
>>  
>>     exit(0);
>>   }
>> }
>> while (wait != -1) {}
>> 1;
>>
>
> I tried reproducing this regression by:
>
> ```
> $ cd /path/to/git.git
> $ git ls-remote 2>&1 > /tmp/root.list
> $ cd builtin/
> $ git ls-remote 2>&1 > /tmp/builtin.list $ cd ../ $ git diff --no-index /tmp/root.list /tmp/builtin.list ```
>
> And indeed, the diff was empty (which meant that both listings are same).
>
> Confused...
>
> --
> An old man doll... just what I always wanted! - Clara


---
 builtin/ls-remote.c |  8 +++++---
 builtin/remote.c    |  3 ++-
 transport.c         | 15 +++++++++++++--
 transport.h         |  2 ++
 4 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index fc76575430..4c1daa0f92 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -128,6 +128,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 		int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 		repo_set_hash_algo(the_repository, hash_algo);
 	}
+	if (transport_disconnect_raw(transport))
+		status = 1;

 	if (!dest && !quiet)
 		fprintf(stderr, "From %s\n", *remote->url);
@@ -154,12 +156,12 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 		if (show_symref_target && ref->symref)
 			printf("ref: %s\t%s\n", ref->symref, ref->refname);
 		printf("%s\t%s\n", oid_to_hex(&ref->objectname), ref->refname);
-		status = 0; /* we found something */
+		if (status != 1)
+			status = 0; /* we found something */
 	}

 	ref_array_clear(&ref_array);
-	if (transport_disconnect(transport))
-		status = 1;
+	transport_clear(transport);
 	transport_ls_refs_options_release(&transport_options);
 	return status;
 }
diff --git a/builtin/remote.c b/builtin/remote.c
index d91bbe728d..055a221942 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -1000,6 +1000,7 @@ static int get_remote_ref_states(const char *name,
 		transport = transport_get(states->remote, states->remote->url_nr > 0 ?
 			states->remote->url[0] : NULL);
 		remote_refs = transport_get_remote_refs(transport, NULL);
+		transport_disconnect_raw(transport);

 		states->queried = 1;
 		if (query & GET_REF_STATES)
@@ -1008,7 +1009,7 @@ static int get_remote_ref_states(const char *name,
 			get_head_names(remote_refs, states);
 		if (query & GET_PUSH_REF_STATES)
 			get_push_ref_states(remote_refs, states);
-		transport_disconnect(transport);
+		transport_clear(transport);
 	} else {
 		for_each_ref(append_ref_to_tracked_list, states);
 		string_list_sort(&states->tracked);
diff --git a/transport.c b/transport.c
index 219af8fd50..c71dab75e9 100644
--- a/transport.c
+++ b/transport.c
@@ -1584,17 +1584,28 @@ int transport_connect(struct transport *transport, const char *name,
 		die(_("operation not supported by protocol"));
 }

-int transport_disconnect(struct transport *transport)
+int transport_disconnect_raw(struct transport *transport)
 {
 	int ret = 0;
 	if (transport->vtable->disconnect)
 		ret = transport->vtable->disconnect(transport);
+	return ret;
+}
+
+void transport_clear(struct transport *transport)
+{
 	if (transport->got_remote_refs)
 		free_refs((void *)transport->remote_refs);
 	clear_bundle_list(transport->bundles);
 	free(transport->bundles);
 	free(transport);
-	return ret;
+}
+
+int transport_disconnect(struct transport *transport)
+{
+	int ret = transport_disconnect_raw(transport);
+	transport_clear(transport);
+	return 0;
 }

 /*
diff --git a/transport.h b/transport.h
index 6393cd9823..fd75905568 100644
--- a/transport.h
+++ b/transport.h
@@ -320,6 +320,8 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs);
 void transport_unlock_pack(struct transport *transport, unsigned int flags);

 int transport_disconnect(struct transport *transport);
+void transport_clear(struct transport *transport);
+int transport_disconnect_raw(struct transport *transport);
 char *transport_anonymize_url(const char *url);
 void transport_take_over(struct transport *transport,
 			 struct child_process *child);
--
2.42.0






[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux