On 05/09/2024 16:08, Jani Nikula wrote:
On Thu, 05 Sep 2024, Vegard Nossum <vegard.nossum@xxxxxxxxxx> wrote:
On 05/09/2024 14:20, Jani Nikula wrote:
On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@xxxxxxxxxxxxxxx> wrote:
On Thu, 05 Sep 2024, bernhard+linux-doc@xxxxxxxx wrote:
From: "Bernhard M. Wiedemann" <bwiedemann@xxxxxxx>
Because we want reproducible builds
and https://github.com/sphinx-doc/sphinx/issues/6714
did not receive any love from Sphinx devs in five years,
let's disable parallel doc builds until that Sphinx issue is fixed.
You mention in [1] that this is likely a duplicate of [2] i.e. multiple
Sphinx instances running in parallel and racing in doctree access.
In kernel, does the issue then boil down to:
htmldocs:
@$(srctree)/scripts/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
i.e. multiple Sphinx invocations instead of just one?
Broken record, I still think we should axe the Makefile hacks out of the
Sphinx build altogether, and let Sphinx handle everything, via
extensions if necessary.
Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
processes, each running with -j8, and each racing for the same
doctrees. And the whole thing is apparently parallelized to 64 threads,
which is unlikely to be the most efficient.
There's no reason to blame Sphinx upstream before we get our end
together, and run Sphinx as it was designed.
I don't reproduce this here, either with plain 'make htmldocs' or 'make
-j8 htmldocs' -- in either case the process tree is:
make ... htmldocs
- make ... htmldocs
- make -f ./scripts/Makefile.build obj=Documentation htmldocs
- /bin/sh -c make ...
- python3 ./scripts/jobserver-exec sh
./Documentation/sphinx/parallel-wrapper.sh sphinx-build -b html ...
- /usr/bin/python3 /usr/bin/sphinx-build -j... -b html ...
Neither the $(foreach) in the Makefile nor $(call loop_cmd) should
result in parallel invocations.
Could you double check the process tree once the "Using alabaster theme"
message has appeared?
$ pstree -A 3457144
bash---make---make---make---make---sh---python3---sphinx-build-+-7*[sphinx-build]
`-sphinx-build---perl
Maybe try after 'make cleandocs'.
Right, I get this with make -j8 after 'make cleandocs':
make---make---make---sh---python3---sphinx-build-+-sphinx-build---perl
`-7*[sphinx-build]
Isn't this exactly what we want? There is only a single top-level
sphinx-build process invoked by 'make', and that is running 8 processes
in parallel, not 64 threads.
I think -j8 appear in these child processes simply because it inherited
the command line arguments (so fork() without exec(), essentially).
Vegard
