Suvayu Ali writes:
Hi Sam, On Sun, May 31, 2015 at 08:05:44PM -0400, Sam Varshavchik wrote: > Suvayu Ali writes: > > > >How does rpmbulid decide how many threads to run? I think it looks at > >the local machine and decides. I also recall some variable called > >RPMBUILD_NCPU or something like that. That would explain why it only > >uses as many threads as the local machine can handle. > > > >Hope this helps, > > Sadly, no it doesn't. As I wrote above: > > >> >> >> ps shows the maximum number of "/usr/bin/distcc /usr/bin/g++ > >[options]" > >> >> >> processes running locally, > > rpmbuild uses the _smp_mflags macro to pass the -j parameter to make, and I > have it correctly configured to kick off ten parallel processes, to match > the maximum number of ten processes distcc is configured (4 local, 6 > remote). Yet, despite the fact that a manual build distributes the compiles > correctly, with rpmbuild distcc throttles the number of concurrent parallel > jobs that it kicks off to four, and runs them locally. I think you misunderstood me, I wasn't clear either. I saw you set _smp_mflags, what I mean is rpmbuild still does some magic underneath. I.e. after you have set that flag, afaik, rpmbuild will fiddle with the value and pass an "appropriate" -j<n> flag to Make in the regular case. It makes this choice based on how many threads it can run locally. At least that's what I think it does. Okay I did some digging. This is what rpmbuild does with _smp_mflags (output from rpmbuild --showrc). -14: _smp_mflags %([ -z "$RPM_BUILD_NCPUS" ] \ && RPM_BUILD_NCPUS="`/usr/bin/getconf _NPROCESSORS_ONLN`"; \ ncpus_max=%{?_smp_ncpus_max}; \if [ -n "$ncpus_max" ] && [ "$ncpus_max" -gt 0 ] && [ "$RPM_BUILD_NCPUS" -gt "$ncpus_max" ]; then RPM_BUILD_NCPUS="$ncpus_max"; fi; \if [ "$RPM_BUILD_NCPUS" -gt 1 ]; then echo "-j$RPM_BUILD_NCPUS"; fi) -14: _smp_ncpus_max 16 I don't have the time to check the above snippet, maybe you can take a look? Hope this time we follow each other. If this doesn't help, I'm out of ideas :-(.
That's nice, except, as I mentioned, I override all of this nonsense in my .rpmmacros, and replace it with a simple
%_smp_mflags -j 10Once again: the correct number of distcc processes get started. But they throttle each other, and only four, at a time, kick off a compile on localhost, the configured distcc max job setting for localhosts, with the others waiting until the earlier ones finish, before starting the real g++. distcc does that on its own. It's configured for a maximum number of concurrent jobs in /etc/distcc/hosts, and if more than that are started, they throttle each other.
I believe I already explained this, but I'm having some apparent difficulty carrying across that, for some reason, distcc ends up ignoring everything but the localhost setting, in /etc/distcc/hosts, when it's started from rpmbuild. That's where the problem is. Obviously rpmbuild has something to do with it, but it's not the direct setting. rpmbuild's %_smp_mflags is configured correctly, and it's doing it's job of kicking off make -j 10, as it should be.
I was hoping that someone came across this glitch before, and knew the root cause. I guess not, and I suppose I'll have to strace the whole thing, and see if that uncovers any clues.
Attachment:
pgpvxxMOAUYx5.pgp
Description: PGP signature
-- users mailing list users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
