On Mon, May 6, 2019 at 12:13 PM Jeff King <peff@xxxxxxxx> wrote: > This reduces the time required to run make coccicheck by a significant > amount of time: > > Prior timing of make coccicheck > real 6m14.090s > user 25m2.606s > sys 1m22.919s > > New timing of make coccicheck > real 1m36.580s > user 7m55.933s > sys 0m18.219s > > This is nearly a 4x decrease in the time required to run make Whoa. > coccicheck. This is due to the overhead of restarting spatch for every > file. By processing all files at once, we can amortize this startup cost > across the total number of files, rather than paying it once per file. > > However, it comes at a cost. The RSS of each spatch process goes from > ~50MB to ~1500MB (and peak memory usage may be even higher if make runs 1.5G should be fine. Trying... Even with no -j, my htop's RES column goes up 6GB and put my laptop in "swap every bit of memory out, including the bits handling the screen" mode :( I don't think it was even the peak. It's probably a bit too much to ask, but is it possible to handle N files at a time (instead of all files), which consumes less memory and runs a bit slower, but still better than the default mode? I can see it already gets tricky doing complicated stuff in Makefile so "no" is perfectly ok. > multiple rule files in parallel due to "-j"). That's enough to make some > systems (like our Travis build!) fail the whole process, or could make > things slower due to swapping. So let's make the new behavior optional, > and people with a lot of memory can choose to use it. > > [peff: modified Jacob's patch to make the behavior optional, since > people reported complications due to the memory use] > > Signed-off-by: Jacob Keller <jacob.keller@xxxxxxxxx> > Signed-off-by: Jeff King <peff@xxxxxxxx> -- Duy
