On Thu, Jan 27, 2011 at 4:53 PM, Allan McRae <allan@xxxxxxxxxxxxx> wrote: > On 28/01/11 03:36, Thomas S Hatch wrote: > >> I have been passively working on a similar project called quarters, but I >> must admit that my motivation is somewhat low not knowing if the project >> is >> in demand. So here is my question, do we think that something like this >> would be a benefit to Arch? Is this the type of project that should merit >> my >> attention? >> > > An automated build system, in particular for the big rebuilds we do, has > been discussed among the developers many times. So there is definitely > interest in such a system. > > > Also, if we do think that this would be a good thing to have for Arch, I >> would like feedback on what types of features the system would have and >> how >> it would behave. Right now I am following the idea of supporting a >> distributed build system so that we can have any number of build servers >> in >> the fray working away to produce Arch packages for us. I am also >> attempting >> to build it in such a way that a database is not required and that the >> interface would be amazingly simple (this is Arch after all). This would >> mean that by mearly checking into svn a package would be built, and then >> an >> interface would pop up for the right people to sign it off, and once it >> has >> been signed off it would move over. >> > > I have never liked the idea of automatic building after a SVN commit. There > a plenty of times that commits are made to SVN in preparation for an update > at a later time. Instead, I would prefer a command-line queue submission > tool (local or remote...). You could have it flexible enough to take a > "makepkg --source" tarball, or allow configuration for updating then > building from a SVN/git/etc repo. > > I agree with this argument, and I have had many thoughts on how to handle it, the reason is because I really like the idea of managing from SVN commits, I think it would make the whole process much easier. What I don't want is an SVN commit to turn into a package push to a production repo, that would be bad. if there is a commit for future use in SVN, and a build happens, thats fine with me, but it could be easily rebuilt when the "future condition" occurs and the process of moving the package from the "build bin" to a production repo should probably always be a manual one. > Thinking about what I do in packaging (and as someone who does not use the > AUR), I see the following situations that need to be covered: > > > 1) single package build: This would probably be the majority use case > > The big idea here is more that a system can do the chroot build and packages checking for the packager, one of the big motivations :) > > 2) multi-package build: Sometimes packages come in pairs/groups where > updating one requires updating the next. e.g. tcl and tk. That requires > building tcl, adding the new package to the chroot and then building tk. > > Note there could be cases where this is more complicated... e.g. Updating > package A,B,C,D. A is required for B, B is required for C and B is > required for D. Note that installing C in the chroot after building would > be unclean for the build of D. Think updates to desktop environments as an > example of this. > > makechrootpkg has the facility to add packages to a local repo after being > built, so that could be used to implement this. > > I have a solution for this, simply that the builders build every package alone in its own chroot, if it has a dep, then the dep needs to be built and added to a repo that the chroot will be able to get to, so more than one package is never built in a single chroot > > 3) big rebuild: e.g. for library soname bump. There are two types of > builds here. A) the initial builds and B) all the packages requiring > rebuild because of those in A. For B, automatic pkgrel bumping would be > good. Also, some automatic determination of the build order for B would be > very helpful. This would require error handling as if a package low in the > dependency chain fails, the rest of that chain should not be built. Again, > adding the packages to a local repo as the rebuild progresses will allow to > build packages cleanly against those that are already rebuilt. Also not > that there could be no packages in list A (e.g. if we just want to rebuild > all of [core]). > > I don't think that this should be a difficult issue, I am a little concerned with auto bumping the pkgrel numbers and committing back to svn, but I am sure that is something I can figure out how to do in a clean and safe way > > 4) flexibility to add makepkg paramaters: I'm not sure how widespread use > this would get... > e.g. the toolchain is built in the following order: > linux-api-headers->glibc->binutils->gcc->binutils->glibc > The first glibc and binutils builds will be built with the --nocheck option > in the future. The gcc and final binutils and glibc builds are built with > -L to save the testsuite build logs. I guess you would make -L the default > anyway and have them available at the end of the build for inspection. > > This is a tricky problem to solve I think, but doable. > > Anyway, #3 is what I consider most important in an automated build system > designed to make the job of packaging in Arch easier. It is also the level > where I would start using such a system.... Building a single package > locally is a two "makechrootpkg" commands (one per architecture) and I do > not have to transfer the resultant package to test. So I doubt I would use > a build system for that (unless build time was massive) as I can not see the > automated build being more efficient for me. > > Allan > I really do hope it can make Arch packaging easier, and raise the overall quality of Arch packages (not that I think that it is too terribly needed) As for 3 and 4, Koji can't do those, which makes me all the more excited to give them a try! Thanks for the input Allan, it is nice to hear from a big gun. -Thomas S Hatch
