On Sat, Feb 10, 2007 at 12:56:11PM +0100, Michael Schwendt wrote: > The status updates raise a few more questions without that I've tried to > install the project locally: > > * One of the screenshots, > > https://hosted.fedoraproject.org/projects/bodhi/attachment/wiki/Screenshots/bodhi-push.png > > shows a "Push Console" where a single build job's packages are pushed and > createrepo is run afterwards. A huge difference to real-world pushing is > that this single operation does not take just 3 seconds, but several > minutes for each repository. The push process takes care of anything that needs to get done (ie {,un}pushing, {,re}moving), based on developer requests (still due to manual intervention for package signing), not just a single build job. > I fail to see how pushing packages like this can be a real-time operation > managed through a web interface. Unless a queueing server runs in the > background, which handles access to the repositories. Access for a variety > of operations, not limited to metadata creation. Add pruning, repoview, > multi-lib stuff. These operations are mutually exclusive. Right now there is no queueing server, and updates are pushed out in batches by a single person, so being a real-time operation seems, and has already half-assedly shown, to be feasible. Since few people actually do the pushing, I'd like to implement whatever they want to use. Going to a web site, clicking a button and waiting for a bit while being able to see real time status updates hasn't gotten any complaints, but With the current real-time status approach (python generators), it should be trivial to be able to hook up any client to it, like a simple command-line tool to do pushes. If Jesse would rather click a button and then get an email at some point later on about the results, then that's fine too. There is a lock on the repos that can be used to keep mutual exclusion with regard to the pushing and sustainment scripts, but I agree that a queueing service is an optimal approach. > * Currently, we push to a local master repository on the buildsys server > and sync that another machine at RDU. This requires SSH-based > authorization. What are the plans to change that within a web-based > updates system? That shouldn't change; the system is going to need to access the build results of plague anyway, so I'm wondering if it would be best to implement some sort of xmlrpc call to have plague stage the packages for us? Since plague and bodhi (among other new pieces of infrastructure) are in different co-locations, we definitely need some sort of way to communicate. SSHFs was mentioned as a potentially viable solution, along with rsync hackery; any suggestions? > * "Staging area": I've browsed the bodhi wiki pages in search for > information on how to make better use of stages in the life-time of a > build-job's results. How do you suggest we make better use of them? I'm fairly ignorant in the ways of plague, so I don't know how the current staging of a build-job results really works. At the moment there is only a single physical updates-stage on disc. When a package is "pushed", it is copied from the build result over to this updates-stage. Before the push an update is either in 'testing', or is waiting to be signed/pushed/moved. It seems like having some of these staging hooks in plague itself might be a good idea, instead of pulling and moving packages out from under it. > Currently, we push from within a single staging area, plague's > "plague-results" tree. Earlier attempts, like moving packages from one > stage to another, have broken the buildsys (in particular the needsign > repo) too often, because it takes time for published packages to find > their way from fedoraproject.org to download.fedora.redhat.com, and > meanwhile, any pushed package we don't keep in the needsign repo breaks > the build servers' dep-chains. The work-around, that has worked flawlessly > since then, is to mark plague-results directories as "PUSHED" with an > empty file: > > http://buildsys.fedoraproject.org/plague-results/fedora-6-extras/seedit/2.1.0-3.fc6/ > > That way we don't need to mess with the needsign repo rpms and metadata > and can keep the access read-only. With operations like "push" and > "unpush" and a testing repo it likely needs more to keep track of the > package life-cycle. What are the plans here? Well, the current implementation of bodhi starts all updates off as 'testing', and from there it can be moved to final when requested (but eventually after a given number of approvals or pushed through by the security response team). From here an update can raise a few requests: push, unpush, and move; but as far as the filesystem stage is concerned, files only get written when updates are pushed to testing or final. John Polestra made a few diagrams of the update procedure that I'm in the process of touching up. Hopefully with these, and eventually some details on Koji, we can optimize the package lifecycle quite a bit. luke
