Re: "git grep" parallelism question

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Linus Torvalds wrote:
> Anyway, I think your patch is good if for no other reason that it
> allows this kind of testing, but at least for my machine, clearly the
> current default of eight threads is actually "good enough". Maybe
> somebody with a very different machine might want to run the above
> script and see if how sensitive other machines are to this parameter..

I have four cores with HT, so I expect similar gains from CPU
parallelism (in the hot-cache case).  And the test results meet this
expectation:

1 real 0m1.287s
2 real 0m0.635s
4 real 0m0.460s
8 real 0m0.413s
16 real 0m0.443s
32 real 0m0.715s

I have a rotating hard disk, and expect the IO parallelism (cold-cache
case) benefits to peak at a larger number of threads (than in your SSD
case).  These are the test results:

1 real 0m28.566s
4 real 0m20.361s
8 real 0m16.990s
16 real 0m15.278s
32 real 0m13.710s
64 real 0m12.405s
128 real 0m11.913s
256 real 0m11.759s
512 real 0m12.022s

It looks like 128~256 threads is the sweet spot.

It's impossible for git to determine if the cache is hot or cold,
correct?  Cold cache is a very rare case, and I'm not sure how we can
optimize specifically for that case.  So, I'm not sure how we can
improve grep.

On a related note, one place that IO parallelism can provide massive
benefits is in executing shell scripts.  Accordingly, I always use the
following commands to compile and test git respectively:

    make -j 8 CFLAGS="-g -O0 -Wall"
    make -j 8 DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS="-j 16" test

i.e. always use 8 threads when the task is known to be CPU intensive,
and always use 16 threads when the task is known to be IO intensive.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]