Re: [PATCH v2 5/5] fetch: Make --jobs control submodules and remotes

[Palmer Dabbelt <palmer@xxxxxxxxxx>]

On Tue, 13 Aug 2019 13:16:11 PDT (-0700), Johannes.Schindelin@xxxxxx wrote:
> Hi,
>
>
> On Mon, 12 Aug 2019, Palmer Dabbelt wrote:
>
> > The existing --jobs argument was defined to control the number of jobs
> > used for submodule fetching, but it makes more sense to have this
> > argument control the number of jobs to be used when fetching from
> > multiple remotes as well.
> >
> > This patch simply changes the --jobs argument parsing code to set both
> > max_children_for_{submodules,fetch}, as well as noting this new behavior
> > in the documentation.
> >
> > Signed-off-by: Palmer Dabbelt <palmer@xxxxxxxxxx>
> > ---
>
> I very much miss in this description a reflection of my analysis in
> https://public-inbox.org/git/nycvar.QRO.7.76.6.1907191507420.47@xxxxxxxxxxxxxxxxx/
>
> Given that analysis, combined with the fact that the `--jobs` option
> tries to control both the `--multiple` and `--recursive-submodules` code
> paths in the end, anyway, I do doubt that it makes sense to even
> introduce the `--fetch-jobs` and the `--submodule-fetch-jobs` options;
> They are probably only confusing and do not add much benefit to the end
> user.

The cover letter at least attempts to describe this.  I figured I'd have to 
pick one option for a v2, so I went with the more complicated one under the 
assumption it would be easy to re-spin a v3 that drops the extra arguments.  
I'm happy to do so.

> Ciao,
> Johannes
>
> >  Documentation/fetch-options.txt |  4 ++++
> >  builtin/fetch.c                 | 21 ++++++++++++++++++---
> >  2 files changed, 22 insertions(+), 3 deletions(-)
> >
> > diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
> > index 5836024f1934..0915fd4ed6d5 100644
> > --- a/Documentation/fetch-options.txt
> > +++ b/Documentation/fetch-options.txt
> > @@ -160,6 +160,10 @@ ifndef::git-pull[]
> >
> >  -j::
> >  --jobs=<n>::
> > +	Number of parallel children to be used for all forms of fetching.
> > +	This is the same as passing `--submodule-fetch-jobs=<n>` and
> > +	`--fetch-jobs=<n>`.
> > +
> >  --submodule-fetch-jobs=<n>::
> >  	Number of parallel children to be used for fetching submodules.
> >  	Each will fetch from different submodules, such that fetching many
> > diff --git a/builtin/fetch.c b/builtin/fetch.c
> > index 67d001f3f78b..41498e9efb3b 100644
> > --- a/builtin/fetch.c
> > +++ b/builtin/fetch.c
> > @@ -114,6 +114,20 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
> >  	return git_default_config(k, v, cb);
> >  }
> >
> > +static int parse_jobs_arg(const struct option *opt, const char *arg, int unset)
> > +{
> > +	int jobs;
> > +
> > +	jobs = atoi(arg);
> > +	if (jobs < 1)
> > +		die(_("There must be a positive number of jobs"));
> > +
> > +	max_children_for_submodules = jobs;
> > +	max_children_for_fetch = jobs;
> > +
> > +	return 0;
> > +}
> > +
> >  static int parse_refmap_arg(const struct option *opt, const char *arg, int unset)
> >  {
> >  	BUG_ON_OPT_NEG(unset);
> > @@ -142,12 +156,13 @@ static struct option builtin_fetch_options[] = {
> >  		    N_("fetch all tags and associated objects"), TAGS_SET),
> >  	OPT_SET_INT('n', NULL, &tags,
> >  		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
> > -	OPT_INTEGER('j', "jobs", &max_children_for_submodules,
> > +	{ OPTION_CALLBACK, 'j', "jobs", NULL, N_("jobs"),
> > +		    N_("number of parallel tasks to run while fetching"),
> > +		    PARSE_OPT_OPTARG, &parse_jobs_arg },
> > +	OPT_INTEGER(0, "submodule-fetch-jobs", &max_children_for_submodules,
> >  		    N_("number of submodules fetched in parallel")),
> >  	OPT_INTEGER(0, "fetch-jobs", &max_children_for_fetch,
> >  		    N_("number of remotes fetched in parallel")),
> > -	OPT_INTEGER(0, "submodule-fetch-jobs", &max_children_for_submodules,
> > -		    N_("number of submodules fetched in parallel")),
> >  	OPT_BOOL('p', "prune", &prune,
> >  		 N_("prune remote-tracking branches no longer on remote")),
> >  	OPT_BOOL('P', "prune-tags", &prune_tags,
> > --
> > 2.21.0