Em Thu, 1 Sep 2016 18:22:09 +0200 Markus Heiser <markus.heiser@xxxxxxxxxxx> escreveu: > Am 01.09.2016 um 16:29 schrieb Jani Nikula <jani.nikula@xxxxxxxxxxxxxxx>: > > > On Thu, 01 Sep 2016, Jonathan Corbet <corbet@xxxxxxx> wrote: > >> On Wed, 24 Aug 2016 15:35:24 +0200 > >> Markus Heiser <markus.heiser@xxxxxxxxxxx> wrote: > >> > >>> With metadata "parallel_read_safe = True" a extension is marked as > >>> save for "parallel reading of source". This is needed if you want > >>> build in parallel with N processes. E.g.: > >>> > >>> make SPHINXOPTS=-j4 htmldocs > >> > >> A definite improvement; applied to the docs tree, thanks. > > > > The Sphinx docs say -jN "should be considered experimental" [1]. Any > > idea *how* experimental that is, really? Could we add some -j by > > default? > > My experience is, that parallel build is only strong on "reading > input" and weak on "writing output". I can't see any rich performance > increase on more than -j2 ... > > Mauro posted [2] his experience with -j8 compared to serial. He > also compares -j8 to -j16: > > > PS: on my server with 16 dual-thread Xeon CPU, the gain with a > > bigger value for -j was not impressive. Got about the same time as > > with -j8 or -j32 there. > > I guess he will get nearly the same results with -j2 ;) > > If we want to add a -j default, I suggest -j2. Actually, here I got better results with -j4, on my NUC with one quad-core, 8 threads Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz and m2SATA SSD disk. This is with -j2: $ make DOCBOOKS="" SPHINXOPTS="-j2" SPHINXDIRS=media SPHINX_CONF="conf_nitpick.py" htmldocs 2>err real 0m46.568s user 1m0.676s sys 0m3.019s And this is with -j4: $ make DOCBOOKS="" SPHINXOPTS="-j4" SPHINXDIRS=media SPHINX_CONF="conf_nitpick.py" htmldocs 2>err real 0m25.356s user 1m1.408s sys 0m2.912s Btw, this is the result on a dual octa-core Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, CPU (total of 32 threads), using PCIe SSD disks: $ for i in $(seq 32 -1 1); do echo "with SPHINXOPTS= -j$i"; make cleandocs;/usr/bin/time --format="real %E nuser %U sys %S" make DOCBOOKS="" SPHINXOPTS="-j$i" SPHINXDIRS=media SPHINX_CONF="conf.py" SPHINXDIRS=media htmldocs >err; done with SPHINXOPTS= -j32 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.05 nuser 98.77 sys 6.45 with SPHINXOPTS= -j31 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:22.81 nuser 97.80 sys 6.12 with SPHINXOPTS= -j30 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.16 nuser 97.68 sys 6.41 with SPHINXOPTS= -j29 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:22.74 nuser 98.02 sys 6.33 with SPHINXOPTS= -j28 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.18 nuser 95.75 sys 6.14 with SPHINXOPTS= -j27 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:22.67 nuser 96.66 sys 6.27 with SPHINXOPTS= -j26 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:22.66 nuser 95.93 sys 6.50 with SPHINXOPTS= -j25 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.71 nuser 95.43 sys 6.43 with SPHINXOPTS= -j24 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.71 nuser 94.27 sys 6.42 with SPHINXOPTS= -j23 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.46 nuser 93.85 sys 6.35 with SPHINXOPTS= -j22 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.28 nuser 91.52 sys 6.29 with SPHINXOPTS= -j21 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.93 nuser 90.37 sys 6.14 with SPHINXOPTS= -j20 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:24.88 nuser 91.40 sys 6.36 with SPHINXOPTS= -j19 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:24.00 nuser 89.68 sys 5.82 with SPHINXOPTS= -j18 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.73 nuser 89.68 sys 5.92 with SPHINXOPTS= -j17 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.75 nuser 87.85 sys 5.58 with SPHINXOPTS= -j16 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:24.54 nuser 87.87 sys 5.94 with SPHINXOPTS= -j15 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:25.45 nuser 88.25 sys 6.28 with SPHINXOPTS= -j14 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.79 nuser 87.23 sys 5.80 with SPHINXOPTS= -j13 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:24.15 nuser 86.06 sys 5.48 with SPHINXOPTS= -j12 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:24.02 nuser 85.94 sys 5.55 with SPHINXOPTS= -j11 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:23.79 nuser 85.12 sys 5.38 with SPHINXOPTS= -j10 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:25.37 nuser 85.32 sys 5.52 with SPHINXOPTS= -j9 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:26.04 nuser 86.96 sys 5.58 with SPHINXOPTS= -j8 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:26.76 nuser 86.38 sys 5.84 with SPHINXOPTS= -j7 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:27.90 nuser 85.33 sys 5.54 with SPHINXOPTS= -j6 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:28.44 nuser 84.59 sys 5.44 with SPHINXOPTS= -j5 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:31.82 nuser 83.98 sys 5.56 with SPHINXOPTS= -j4 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:35.68 nuser 83.98 sys 5.52 with SPHINXOPTS= -j3 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 0:43.00 nuser 85.08 sys 5.78 with SPHINXOPTS= -j2 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 1:10.72 nuser 88.61 sys 6.17 with SPHINXOPTS= -j1 make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. real 1:27.23 nuser 84.49 sys 2.97 There, -j4 is not the best performance. it goes to the ~23-24 seconds only with -j11. So, setting a default is not trivial, and seems to depend a lot on the actual machine used on the build. Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
