On Thu, May 24, 2007 at 01:02:35AM -0400, Bill Nottingham wrote: > Mmm, plumbing. bodhi is heading for production soon. To push updates, what > bodhi currently does is, for any update: > > - sign the package Nope, neither bodhi or the current update system sign any packages. The current system mails releng with the proper command to sign the packages, but it has always been done by hand as far as I know. You and Jesse are the only people I know of that have signed updates. By the looks of the fedora-release-tools module, there are two scripts that have been used to sign packages, ftsign and fedorasign, both of which call /usr/local/bin/rpm-4.1-sign, which is a symlink to /usr/lib/rpm/rpmk. I started implementing an XMLRPC server for bodhi so we can eventually do everything from the command line as well as from the browser. Hopefully we can streamline the signing process as much as possible in a command-line sign/push tool, until it can be fully automated with a signing server (when might this happen, btw?). Koji keeps a sigcache for each package in pkg/ver/rel/data/sigcache/arch/, although I have no idea at what point in the build process this gets created. I'm also under the impression that just having this detached signature isn't enough, and that there still must be some manual intervention? Is there anything else bodhi needs to do other than make sure the corresponding .sig exists for each package? > - copy the package to a 'staging' tree of the entirety of updates > - read a static list of packages that should be multilib, act on that This isn't as bad as the previous biarch-list-of-doom[0] anymore. Bodhi imports it into its Multilib database table[1] during initialization, and doesn't deal with it again. Upon submission of an update, bodhi builds the list of associated packages, taking care of multilib based on what's in the db. The multilib table can then be modified with ease using the TurboGears CatWalk database editor, or a simple command-line tool. > - run createrepo > - check deps on the repo FIXME: I need to track down some false positives in bodhi's closure.py[2] (or rewrite it). Mash would obviously resolve this for us. > - rsync the whole repo out TODO. At the moment bodhi stages to /mnt/koji/updates-stage -- where are we going to sync this to? wallace still? > Older updates are cleaned by a cron script later. TODO. We need something similar to the fedora-updates-clean script that is currently in place (but less hackish), or RepoPrune / repomanage. The TurboGears scheduler[3] is probably a good place for this. I'm going to try and find some time tonight to throw one together. > Advantages of this approach: > - it's simple > - it's easy to clean upthings that Go Wrong (just manually remove them > from the repo and re-sync) This also gives bodhi a LOT more control over the repos, as it maintains the extended updateinfo.xml.gz in the repodata as well. If we use mash we will have to maintain this file outside of the tree and re-insert it post-compose. > Disadvantages: > - multilib. In a world where we continually add new packages, this > *will not scale*. Random idea. Since multilib is handled by mash, which pushed out rawhide nightly, couldn't we just have mash keep the Multilib table up to date? Would this solve the scalability issue wrt new packages? > So, we need at least *some* sort of better workflow. > > One alternative - using mash (what we're using to build rawhide.) It > would go something like this: > > - sign the package > - tag the package (for updates-testing, or updates) > - run mash to create a repo of updates/updates-testing, solve it for > multilib > - rsync it out > > Advantages: > - solves multilib > - doesn't require continually keeping a staging tree around > - depcheck is built in when solving multilib > - builds on koji tags to let anyone easily query what updates are > released > > Disadvantages: > - by rebuilding the repo each time, it's going to be slow once > the repo gets large > - harder to clear out other strangeness > - will only have one version of each updated package > > The last of these isn't as *big* of a concern now, as all builds > will be available through the koji web site, space permitting. > > Other ideas for better workflow? What do the extras push scripts do? > Do we want to add a modified version of mash's multilib solver into > bodhi? I think the mash idea is interesting. Although, due to it's overhead, we would probaby have to resort to pushing out a single batch of updates a day, and maybe some smaller batches of security updates. This might become a pain. I'm going to look around at the multilib solver for mash and extrsa tonight and see if bodhi can steal any of it. Michael Schwendt would probably be the expert in the extras world. luke [0]: https://hosted.fedoraproject.org/projects/bodhi/browser/bodhi/deprecated/biarch.py [1]: https://hosted.fedoraproject.org/projects/bodhi/browser/bodhi/model.py#L63 [2]: https://hosted.fedoraproject.org/projects/bodhi/browser/bodhi/closure.py [3]: http://docs.turbogears.org/1.0/Scheduler
