On Tue, 14 Nov 2017, Masahiro Yamada wrote: > Hi Julia, > > > 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@xxxxxxx>: > > > > > > On Fri, 10 Nov 2017, Julia Lawall wrote: > > > >> > >> > >> On Thu, 9 Nov 2017, Masahiro Yamada wrote: > >> > >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > >> > "coccicheck failed" error messages. > >> > >> The question is where parallelism should be specified. Currently, make > >> coccicheck picks up the number of cores on the machine and passes that to > >> Coccinelle. > >> > >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" > >> > >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, > >> while in practice that degrades the performance as compared to 40 or 80 > >> cores. > >> > >> On the other hand, if we use the make command line argument (-j), then we > >> will only get parallelism up to the number of semantic patches. Since > >> some finish quickly, there will be a lot of wasted cycles. > >> > >> The best would be that the user knows what works well for his machine, and > >> specifies it on the command line, and then that value gets propagated to > >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in > >> parallel but instead would cause Coccinelle to run one semantic patch on 8 > >> files in parallel. But I don't know if that can be done. > > > > Sorry for these fairly nonsensical comments. make -j is going to consider > > every file, then parse and run every semantic patch on that file. If the > > parallelism is pushed down into Coccinelle, each semantic patch will be > > parsed only once, and then Coccinelle will choose the files for which it > > is relevant. If indexing is used (idutils, glimpse), then for semantic > > patches that focus on specific keywords, Coccinelle will efficiently > > ignore files that are not relevant. I don't think there would be many > > cases where make -j would win. Perhaps it would be possible to detect > > its used and abort with an appopriate message? > > > I am afraid you and I are talking different things. > > > For a usual usage of coccicheck, only one thread runs scripts/coccicheck > even if -j is passed from the command line. > > coccicheck provides "J" to specify parallelism. > > if [ -z "$J" ]; then > NPROC=$(getconf _NPROCESSORS_ONLN) > else > NPROC="$J" > fi Even if J is not specified, then it still runs with the maximum number of threads: Coccinelle parallelization --------------------------- By default, coccicheck tries to run as parallel as possible. Indeed, J= does set the number of threads speficied. > My patch addresses a problem where coccicheck is used as CHECK. > The default of CHECK is "sparse", but you can use any checker tool. > > In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel > under control of GNU Make, so scripts/coccicheck is also invoked from > multiple threads. > Passing --jobs to spatch is not sensible because it checks only one file. OK. I tried a simple make coccicheck -j4 and indeed it does not seem to be complaining. The number of spatch processes goes over 160 though. julia -- To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
