* Ludovic CourtÃs wrote on Fri, Mar 04, 2011 at 06:59:45PM CET: > Ralf Wildenhues writes: > > * Ludovic CourtÃs wrote on Thu, Mar 03, 2011 at 04:42:52PM CET: > >> I ran a series of build time measurements on a 32-core machine, with > >> make -jX, with X in [1..32], and the results are available at: > >> > >> http://hubble.gforge.inria.fr/parallel-builds.html > > > > Thank you! Would you be so kind and also describe what we see in the > > graphs? I'm sorry but I fail to understand what they are showing, what > > the axes really mean, and how to interpret the results. > > Y is the number of packages with a speedup <= X. Does it help? Well, it helps in that it allows me to understand the graphs, but it doesn't allow me to interpret them. How about a histogram with x the speedup (in some number of intervals) and y the number of packages exhibiting that speedup? That would IMVHO be easier to read. > > A few of the packages (using an Autotest test suite: Autoconf, Bison) > > would benefit from you passing TESTSUITEFLAGS=-jN to make. > > Oh, I didnât know that. So âmake -jNâ isnât enough for Autotest? No. Doing that right would require something like http://thread.gmane.org/gmane.comp.sysutils.autoconf.patches/5563/focus=5566 which was never added. (I don't actually remember whether it was rejected or just not bug-free yet.) > > FWIW, parallelizability of Automake's own 'make check' has been improved > > in the git tree (or so at least I hope). > > Yeah, and its âmake checkâ phase already scales relatively well. Oh, I'm willing to bet we scale to 20 now. > > I am fairly surprised GCC build times scaled so little. IIRC I've seen > > way higher numbers. Is you I/O hardware adequate? > > I think so. :-) OK, so I guess I'd like some details as to how you build it. Is bootstrapping enabled (three-stage build), which languages do you build? > > Did you use only -j or also -l for the per-package times (I would > > recommend to not use -l). > > I actually used â-jX -lXâ. What makes you think -l shouldnât be used? Typically, -lX leads to waves in the load, due to the latency between measure and action, and of course the retardation from the measure interval. There are long periods in which processes are already done but the load is still listed as too high. > The main problem Iâm interested in is continuous integration on a > cluster. When building a complete distro on a cluster, thereâs > parallelism to be exploited at the level of package composition (e.g., > build GCC and Glibc at the same time, each with N/2 cores), and > parallelism within a build (âmake -jXâ). > > Suppose youâve scheduled GCC and Glibc on a 4-core machine, you want > each of them to use 2 cores without stepping on each otherâs toes. > I think -l2 may help with this. I understand the problem, but I don't think -lX is a real solution to it. First off, you want -l4 (or maybe even -l6) on a four-way system, not -l2. Then, you still have the waves as above. It might be simpler to just use -j3 for each of them. Do you build GCC and Glibc in separate virtual containers? (If yes, my -l4 suggestion above might be wrong.) If not, we could think of providing a high-level job server that just serves out job cookies to the makes of both projects. It might require a bit of adjustment to GNU make to do this in all sorts of interesting distro build setups. But it would allow for (more) effective load usage. Cheers, Ralf _______________________________________________ Autoconf mailing list Autoconf@xxxxxxx http://lists.gnu.org/mailman/listinfo/autoconf
