Hi, On 14/01/2015 17:56, Ken Dreyer wrote: > On 01/09/2015 08:16 AM, Sage Weil wrote: >> On Fri, 9 Jan 2015, Ken Dreyer wrote: >>> On 01/09/2015 03:33 AM, Loic Dachary wrote: >>>> On 09/01/2015 07:55, David Zafman wrote: >>>>> >>>>> We are seeing gitbuilder failures. This is what I saw on one. >>>>> >>>>> error: Failed build dependencies: >>>>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64 >>>> >>>> This is because https://github.com/ceph/ceph/pull/3036 was recently merged. >>>> However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet >>>> about a month ago. Maybe some gitbuilder machines were not updated >>>> accordingly. If I'm not mistaken this is a manual process but I'm not sure >>>> exactly how to do it myself. >>> >>> I'm not entirely sure of the SOP to apply fabfile changes across our >>> infrastructure, either. Is there a simple "push this fabfile change >>> everywhere" command for gitbuilder? >> >> Not really, but it is high time we established one. >> >> At the top of the fabfile all of the targets are defined, so that if you >> run bin/fab w/o arguments it should run on *all* of them. Unfortunately >> IIRC it's a bit disruptive as it restarts the autobuild service which will >> interrupt and restart the current build. >> >> But, that list isn't well-maintained, so it doesn't map to the current set >> of gitbuilders. That's probably pretty easy to fix. Even if we >> do, though, I think fab isn't super great about skipping targets that are >> down? Not sure. >> >> In any case, the way I've run it in practice is on specific targets taht >> need updating, like so >> >> bin/fab gitbuilder_auto:host=user@foo >> >> (or something like that, I forget the exact syntax). Often on a >> host that isn't in the master list. >> >> More frequently the procedure has been: >> >> - add a dependency to the fabfile for future builders >> - manually install the package on the existing ones to avoid the above >> pain >> >> which mostly works but doesn't regularly verify that the fabfile is >> sufficient and correct. >> >>> In the long term we'll want to get to a point where we're building >>> Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can >>> resolve these sort of things automatically without having to deal with >>> installing new BuildRequires by hand. >> >> That sounds great, although I wonder if it is slow? We used to do >> all of our debian builds with pbuilder which does a similar thing, >> but it is way slow because it reinstalls deps on every run. We moved >> to doing native dpkg builds because time is of the essence for test >> builds and use pbuilder only for releases... > > It sounds like we should research exactly how much time mock and > pbuilder would add. Mock implements caching at a lot of levels, though > it's true that it won't be as fast as just running plain rpmbuild. We > should get some hard numbers so we know how painful the speed hit would be. > > I'm weighing the time that mock would slow down the builds against the > time and energy that it takes to > - Duplicate the BuildRequires in autobuild-ceph.git > - Review the change for correctness and merge the autobuild-ceph.git > pull request > - Deploy the autobuild-ceph.git changes out to the appropriate > gitbuilders It could be changed to use https://github.com/ceph/ceph/blob/master/install-deps.sh instead. > And then there's the "people stuff", like comprehending that all of the > above is even necessary, or communicating about this on mailing lists > like this thread, etc. > > The fact that pbuilders are introduced very late (only during releases) > means that we encounter errors during a release day when Alfredo or I > have tried to get a release out the door. The pbuilder stuff isn't > touched until we try to do a release, and in the mean time, it breaks > because it's not getting continually tested. We're paying the "slow" > penalty in other areas down the road. > > I'm not looking to purposefully slow down other people's work, so I'll > continue to mull over how we can balance speed with correctness. IMHO the overhead of mock is so small that it can be ignored. pbuilder is much more expensive (because of the tarbal / tree removal) and there is no way to reduce that overhead, IIRC. There also is a need to run make check on each platform instead of a small subset (ubuntu 12.04 + 14.04 on amd64 + i386 only). That can only be conveniently done within a container (the pbuilder / mock chroot isolation is not good enough). We could combine the two: move the current packaging process to use containers instead of mock / pbuilder. It has the advantage of being a single method cross distributions and allowing more comprehensive checks (make check) during packaging. My 2cts ;-) -- Loïc Dachary, Artisan Logiciel Libre
Attachment:
signature.asc
Description: OpenPGP digital signature
