On Wed, 27 Sep 2023, Bagas Sanjaya wrote: > On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote: > > Dear Git Gurus, > > In DataLad (https://datalad.org) we are doing lots of automated cloning, > > fetching etc as part of our CI etc jobs. Once in a while git operations > > fail [see e.g. 1], and beg us to retry but we need to know when to > > do so, and not do it upon every failed git invocation since some > > failures could be legit (repository is gone). While looking how others > > solve it we found > > https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed > > which pointed to tools like git-retry and later part of > > https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go > > which serve as a collection of regexes to be on lookout for to retry. > > Would that be the "best" strategy currently? > Looking at the actual git_retry.py script [1], it really just wraps > actual Git commands. IMO, git-retry(1) shell script as you mentioned > only calls the python version, which adds another level of indirection > (why not doing it in pure shell?). My guess would be that it is just easier to code in Python usually for such cases with a "registry" of hits etc. But why not just to strip .py from python script which has shebang already and not require bash wrapper at all? ;) > AFAIK, to solve the retrying problem, we need to have a way to tell > transport backend (curl/ssh) to resume transfer from the faulty point. some times it seems not even getting connected (https) entirely and that (?) leading to error in the caller above, e.g. from https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755 error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ... > > As regex matching might eventually break whenever `git` changes > > anything in the output messages, I wondered if there could be a more > > robust internal implementation in git itself? Similarly git-annex has > > annex.retry config setting which sets the count of retries for > > "retriable" operations. > Do you use porcelain interfaces instead of plumbing ones? I would say -- a "mix". -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 WWW: http://www.linkedin.com/in/yarik
