Hi Rob,
On Wed, Apr 22, 2020 at 6:20 AM Rob Herring <robh@xxxxxxxxxx> wrote:
>
> As the number of schemas has increased, we're starting to hit the error
> "execvp: /bin/sh: Argument list too long". This is due to passing all the
> schema files on the command line to dt-mk-schema. It currently is only
> with out of tree builds and is intermittent depending on the file path
> lengths.
>
> Commit 2ba06cd8565b ("kbuild: Always validate DT binding examples") made
> hitting this proplem more likely since the example validation now always
> gets the full list of schemas.
>
> Fix this by putting the schema file list into a temp file and using xargs.
>
> Reported-by: Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx>
> Cc: Masahiro Yamada <masahiroy@xxxxxxxxxx>
> Signed-off-by: Rob Herring <robh@xxxxxxxxxx>
> ---
> Documentation/devicetree/bindings/.gitignore | 2 +-
> Documentation/devicetree/bindings/Makefile | 3 ++-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/.gitignore b/Documentation/devicetree/bindings/.gitignore
> index 5c6d8ea1a09c..0a6aef915fa4 100644
> --- a/Documentation/devicetree/bindings/.gitignore
> +++ b/Documentation/devicetree/bindings/.gitignore
> @@ -1,3 +1,3 @@
> # SPDX-License-Identifier: GPL-2.0-only
> *.example.dts
> -processed-schema*.yaml
> +processed-schema*.yaml*
> diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
> index 1df680d07461..1c1cad860b7c 100644
> --- a/Documentation/devicetree/bindings/Makefile
> +++ b/Documentation/devicetree/bindings/Makefile
> @@ -14,7 +14,8 @@ $(obj)/%.example.dts: $(src)/%.yaml FORCE
> DT_TMP_SCHEMA := $(obj)/processed-schema-examples.yaml
>
> quiet_cmd_mk_schema = SCHEMA $@
> - cmd_mk_schema = $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@ $(real-prereqs)
> + cmd_mk_schema = $(file >$@.tmp, $(real-prereqs)) \
> + cat $@.tmp | xargs $(DT_MK_SCHEMA) $(DT_MK_SCHEMA_FLAGS) -o $@
>
The built-in function $(file ...) is supported on GNU Make 4.0 or later.
The current minimal version is GNU Make 3.81.
If you want to use this function, you must update
Documentation/process/changes.rst first.
I am pretty sure some conservative distros
still stick to GNU Make 3.8*
but I am open to raising the minimal version
if it is useful.
But, does this code work in the first place?
When a very long command is given, xargs
splits it into smaller chunks, and invokes
the command multiple times, right?
So, it boils down to this question:
Are the following two commands work equivalently?
"dt-mk-schema -o processed-schema-examples.yaml foo.yaml &&
dt-mk-schema -o processed-schema-examples.yaml bar.yaml &&
dt-mk-schema -o processed-schema-examples.yaml baz.yaml"
"dt-mk-schema -o processed-schema-examples.yaml foo.yaml bar.yaml baz.yaml"
I think the answer is no.
I confirmed the produced processed-schema-examples.yaml is broken.
In a normal case, it is more than 50000 lines,
but if xargs split the command, it is much smaller.
masahiro@oscar:~/ref/linux$ make -j24 O=foo/bar dt_binding_check
masahiro@oscar:~/ref/linux$ wc
foo/bar/Documentation/devicetree/bindings/processed-schema-examples.yaml
51461 115967 1516186
foo/bar/Documentation/devicetree/bindings/processed-schema-examples.yaml
masahiro@oscar:~/ref/this/is/a/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/src/path/to/linux$
make -j24 O=foo/bar dt_binding_check
masahiro@oscar:~/ref/this/is/a/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/long/src/path/to/linux$
wc foo/bar/Documentation/devicetree/bindings/processed-schema-examples.yaml
9625 21510 291993
foo/bar/Documentation/devicetree/bindings/processed-schema-examples.yaml
Can dt-mk-schema read the list of schema files
by other means?
> DT_DOCS = $(addprefix $(src)/, \
> $(shell \
> --
> 2.20.1
>
--
Best Regards
Masahiro Yamada
