On 27/09/2023 19:42, Yaroslav Halchenko wrote: > > On Wed, 27 Sep 2023, Bagas Sanjaya wrote: > >> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote: >>> Dear Git Gurus, > >>> In DataLad (https://datalad.org) we are doing lots of automated cloning, >>> fetching etc as part of our CI etc jobs. Once in a while git operations >>> fail [see e.g. 1], and beg us to retry but we need to know when to >>> do so, and not do it upon every failed git invocation since some >>> failures could be legit (repository is gone). While looking how others >>> solve it we found >>> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed >>> which pointed to tools like git-retry and later part of >>> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go >>> which serve as a collection of regexes to be on lookout for to retry. > >>> Would that be the "best" strategy currently? > >> Looking at the actual git_retry.py script [1], it really just wraps >> actual Git commands. IMO, git-retry(1) shell script as you mentioned >> only calls the python version, which adds another level of indirection >> (why not doing it in pure shell?). > > My guess would be that it is just easier to code in Python usually for > such cases with a "registry" of hits etc. But why not just to strip .py > from python script which has shebang already and not require bash > wrapper at all? ;) > >> AFAIK, to solve the retrying problem, we need to have a way to tell >> transport backend (curl/ssh) to resume transfer from the faulty point. > > some times it seems not even getting connected (https) entirely and that > (?) leading to error in the caller above, e.g. from > https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755 > > error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ... > Then the question is how can the server cache generated on-the-fly bundles in hope that future clients can resume the transfer. > >>> As regex matching might eventually break whenever `git` changes >>> anything in the output messages, I wondered if there could be a more >>> robust internal implementation in git itself? Similarly git-annex has >>> annex.retry config setting which sets the count of retries for >>> "retriable" operations. > >> Do you use porcelain interfaces instead of plumbing ones? > > I would say -- a "mix". > Junio, what do you think of above ideas from Yaroslav? -- An old man doll... just what I always wanted! - Clara
