On 2022/06/13 15:39, Paul Eggert wrote:
In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes soooo long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parallelize 'configure' scripts?
---- I don't know what type of instrumentations you've done over configure, but before investing a much time in optimization, it might be interesting to know where most of the time is being spent.I.e cpu, I/O -- what types of I/O -- actual test-I/O or executable load time.
The reason I say that is that having run configure for the same projects on linux and on cygwin -- I note that the cygwin version is MUCH slowerdoing the same work on the same machine.
A big slowdown in cygwin is loading & starting a new executable/binary. I.e loading 100 programs 10x each will take a disproportionately higher timeon cygwin due to its exec-load penalty (Since windows has no fork, all of the
memory space duplication (and later copies on write) has to be done manually
in cygwin -- very painful. But noting that one of the big boosts in
shell scripts can come from using a parallel option of a util vs. feeding in
file/pathnames one at a time like using 'find -exec "rm {}" \; or similar.
Similarly a big speed up in configure might be to use the bundled version of
coreutils (all binaries in 1 image invoked via different command names), and
put that in the same binary as bash, perhaps via a loadable command, with
any following core-util calls being routed "in-binary" to the already loaded
version. Of course it would likely not be trivial assuring all the
commands can be re-invoked to assure they had their necessary
initializations
redone on each "in-image" launch, but keeping all the coreutil binaries"in-memory", I think would be a big win even if it wasn't multi-threaded. Of course it might be of benefit if the various utils were all thread safe, so
a more powerful dispatcher could use multi-threading w/o worries aboutthread safety, but just eliminating most of the redundant util-loads might be a huge win by itself. That's sorta why I was wondering how much perf-profiling
had been done on config(.sh)... Anyway -- just some random thoughts...
For ideas about this, please see PaSh-JIT:Kallas K, Mustafa T, Bielak J, Karnikis D, Dang THY, Greenberg M, Vasilakis N. Practically correct, just-in-time shell script parallelization. Proc OSDI 22. July 2022. https://nikos.vasilak.is/p/pash:osdi:2022.pdfI've wanted something like this for *years* (I assigned a simpler version to my undergraduates but of course it was too much to expect them to implement it) and I hope some sort of parallelization like this can get into production with Bash at some point (or some other shell if Bash can't use this idea).
