On Wed, Sep 23, 2015 at 12:34 PM, Junio C Hamano <gitster@xxxxxxxxx> wrote:
> Junio C Hamano <gitster@xxxxxxxxx> writes:
>
>> You are running "git fetch" that are is a lot more heavy-weight.
>> Because once each of them started fully they will be network bound,
>> it is likely that you would want to run more processes than you have
>> core.
>
> I thought the conclusion would be obvious, but just in case the
> readers need the flow of thought completed, from the above it
> follows that on a N-core box (say 8-core) you may want to run the
> fetch with -j16 (or more). If we start everything at once, the time
> before one process starts to produce first meaningful response (I am
> not counting the "starting command for submodule-$i" message as
> "meaningful") would take twice as long under such condition even if
> you have infinite network bandwidth and talking to an infinitely
> fast server.
>
> I agree that a full slow-start ramping-up is not necessary in order
> to hide the start-up lag. All you need to do is to start just one
> and make sure it becomes the foreground, and give it a time alone to
> make enough progress to produce an early output without getting
> slowed down by system activities caused by the other background
> processes in the group that start all at the same time. And once
> the foreground process starts showing its output, you can unleash
> the remainder of the herd to thunder and nobody would notice as they
> run all in the background. Again, I am assuming something like "git
> fetch" that takes reasonably large amount of resource and time to
> start-up and tear-down.
>
I can confirm this now.
git fetch --recurse-submodules=yes -j 400
in an submodule-ified Android tree takes very long to start putting out useful
information, but if I hardcode the SPAWN_CAP to 4 it looks pretty amazing
fast.
I'd guess we could have a reasonable default for SPAWN_CAP by
#define SPAWN_CAP min(pp.max_processes + 1, online_cpus)
>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html