> What would that change? We've talked about it, criticism has been noted, > and as I've tried to make clear, the checklist should not be > misunderstood. There is no silver bullet. One could create a different > checklist for every different type of package. The biggest hurdle to QA is > lack of common sense. I don't want to spend a lot of time editing > documentation in the Wiki to please a single individual (read "you") who > runs his own independent repository and doesn't really care. I'd rather > like to know how to lower the hurdle for other people who would like to > help, but who still don't know where to start. And that would mean that > they start talking about any problems they see. > As a new-to-fedora.us prospective QA'er, I've been attempting to compose an email on this very subject for the last several days, but haven't managed to complete it. It's going to be long, so please forgive me As a longtime redhat user (4+ yrs) and not-entirely-new rpm packager (2 yrs) and professional system administrator / software engineer, it took me over 8 hours of work before I submitted my first QA review. This is partially because I hadn't been using GPG, partially because I didn't have a bugzilla account, partially due to general incompetence on my part, and partially due to the amount of material in the QA checklist. I'm happy to discuss this stuff in more detail, but I'm just going to throw out some "newbie observations" for you all to think about. I do think that the barrier for entry that I found was entirely too high. I'm just now beginning to understand what is going on after getting some feedback on some of my reviews. 1. Information how "how to do some useful qa" is scattered. Heres only some of the url's I had to look at to figure out whats going. None of them were complete in and of themselves, and my first two attempts at QA were still sub-optimal. http://www.fedora.us/wiki/PackageSubmissionQAPolicy http://www.fedora.us/pipermail/fedora-devel/2003-March/000459.html http://www.fedora.us/wiki/QAChecklist http://www.fedora.us/wiki/HOWTOFindMissingBuildRequires I also got some good mileage out of one of Jef Spaleta's bug triage messages to the list. 2. Too much detail. The QA checklist is great. However, as it turns out, a lot of the stuff being checked there aren't really showstoppers. I ended up turning it into a list 1. Does the package follow the Fedora Package Naming Guidelines? 2. Are the sources from upstream pristine and free from trojans? 3. Are the pre- and post(un)install scripts correct? 4. Epoch 5. Are there no missing BuildRequires? 6. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS. 7. Are all paths within the .spec replaced with macros? 8. If the package has a DesktopEntry, are the VFolder categories ok? 9. Is the package free of any %{buildroot} macros? 10. Is the package free of unowned directories? 11. Is the License tag used rather than Copyright? 12. Is the package free of default passwords? 13. Are all daemon's which can be installed to run as non-root installed as such? 14. Does the package handle a parallel build cleanly? Now. Some of these items apparently are subject to contention. If there's any chance that they aren't showstoppers, lets get them off that list! There should be a template like I have above, that can be answered with simple yes or no answers, that covers all the showstoppers. That way a newbie can fill it in the blank with yes's or no's and no he's done something useful. Since all the checklist items are there, answered with yes or no questions, it's easy to know if they actually made a good faith effort to check the package. The non-showstoppers should be on a second list of "stuff to watch for". This could be far more detailed than the actual QA checklist, and as a newbie gets deeper into packaging lore, they would likely begin filling out more of it. Some critique of the list: 1. Does the package follow the Fedora Package Naming Guidelines? This is pretty darn complicated for a newbie QA'er. They should be allowed to opt out. A "senior developer" could look at the package in 10 seconds and understand whether or not the name is ok. 2. Are the sources from upstream pristine and free from trojans? This is important, but exceedingly difficult to do "properly"... At what point does this become a showstopper? At some point it becomes impossible to verify the source without actually being the author of the code and inspecting it line by line. For a newbie this is very intimidating My suggestion would be to have a set of no-fail, simple steps that result in a concrete answer, like the following: "verify the source url exists, and check to see that it is at the official distribution site for the software. Compare the md5 sums of their posted souce tarball and the one included in the SRPM. If it doesn't, please detail your findings". 3. Are the pre- and post(un)install scripts correct? Honestly. How the heck is a newbie supposed to know this? Again, concrete steps would be better. I'd suggest something like "are all pathnames in the pre/post(un) install scripts complete? Are macros used for all package files? Are all operations backed out? Is ldconfig being run if there are shared libraries installed? Maybe these should be a series of questions Also. Wheres the "does this package appear to work" line item? Do we QA packages and not actually run them? It seems like this is pretty basic criteria. 4. Epoch This appear to be subject to debate, and isn't included in a lot of upstream rpms. That being said it's easy to check so leave it in. Couldn't / doesn't rpmlint check this? 5. Are there no missing BuildRequires? This one took me a LONG time to check. The manual "remove all devel rpms" method is apparently deprecated according to the url above, however fedora.us doesn't even include a copy of mach. No less, mach is pretty complicated and you've got to figure out how to create a fedora.us enabled buildroot. After all that mach works great, except for I found out that it doesn't understand perl(Net::Lib) style depends, so it doesn't handle on buildrequires. There needs to be an official build environment. Mach works. Why not spend the hour it takes to get a copy of mach in the repo, with fc1+extras as the default buildroot? Then you can change this line to "Does the package build under mach". And the "how to build using mach" instructions can be yum install mach mach build mypackage.src.rpm 6. Is the Group tag an official one? See /usr/share/doc/rpm-*/GROUPS. Not sure if this is relevant to anything, but again, it's easy to check. Couldn't rpmlint do this? 7. Are all paths within the .spec replaced with macros? Relatively easy to check, but I'd had to be tasked with this without having built a few rpms in the past. A howto for this process might be nice. 8. If the package has a DesktopEntry, are the VFolder categories ok? I don't have any idea what any of this stuff is. I don't build desktop software. Some documentation about it might be good. 9. Is the package free of any %{buildroot} macros? Apparently this is subject to debate. It's not a showstopper. Make it go away. 10. Is the package free of unowned directories? This is also very hard to check. You mean I have rpm -ivvh this package and then scroll through a whole pile of cruft to see the unowned directories? And then I have to know which ones it's "supposed" to own, and which ones it's now? The packages I checked didn't own /usr, /usr/lib, /usr/bin. I assume they aren't supposed to? This is a good thing to check, but theres gotta be a way to automate this, and if nothing else, explain what paths are ALLOWED to be unowned. 11. Is the License tag used rather than Copyright? Easy. Rpmlint should/can? Check this. 12. Is the package free of default passwords? Not necessarily easy to check, but easy to understand, and important. 13. Are all daemon's which can be installed to run as non-root installed as such? Same as 12. 14. Does the package handle a parallel build cleanly? Does this really matter? Seems like a nice thing to check but not a showstopper. Make it go away. Ok. So. Heres what I suggest to replace it all with Showstoppers: 1. Are the sources from upstream pristine and free from trojans? 2. Does the package build clean under mach? 3. Does the package follow the Fedora Package Naming Guidelines? 4. Does the package appear to work? 5. Is the package free of default passwords? 6. Are all daemon's which can be installed to run as non-root installed as such? Additional Critiques: 7. Are all paths within the .spec replaced with macros? 8. If the package has a DesktopEntry, are the VFolder categories ok? 9. Is the package free of unowned directories? <-- at least explain this 10. Does the package handle a parallel build cleanly? 11. Does it pass rpmlint cleanly? With good solid descriptions for 1-6 especially, I think it will become much easier to get useful information from would-be QA'ers. I'd like to see the how to help page say "Please do QA. Heres how" 1. Create bugzilla login 2. Follow this link to the list of packages needing QA. 3. Follow these instructions and fill out this checklist and post the results in a comment. Some additional notes: 1. Relax on the whole GPG thing. GPG is great. I figured it out. So can anybody else. But don't make it a barrier for entry. For a newbie, it's utterly unnecessary. The first thing you want people to do is to go through the package, create a bugzilla login, and post a completed checklist like I did above. When they first start, their input is going to be suspect anyway, so why slow down the process by an hour by figuring out how to use GPG? Bugzilla passwords aren't entirely insecure... 2. Provide some feedback. I QA'd some packages. I waited. And waited. A week later, AFTER I saw a discussion about bugzilla keywords and added "NEEDSWORK" to the packages I had QA'd, I got my first piece of feedback. This doesn't encourage repeat customers. It's a whole lot more rewarding when you submit something, and an hour later you get a message back from somebody saying "Hey, you messed up, but thanks for trying, can you please check a, b, and c?". I realize everybody's volunteering here, but there has GOT to be a way to detect when somebody new has posted to bugzilla and go check out what they did. If there were a "newbie comments" query available in bugzilla it might help. It's a set the hook sort of thing. 3. Provide a way for people to QA packages which have already been QA'd, so the 2 reviews happens. I think there has been some discussion about this. A link to an appropriate bugzilla query from the "How do I get started" page would be great. I think it's imperative that packages make it through the QA process. It doesn't do any good if packages never make it into the repo. Right now, they appear to just sit there. Then the packager gets fed up, doesn't respond if anybody checks his package, and starts a new repo. This is not building community. This process has GOT to be fast enough that when I submit a package for something that people actually use, it gets QA'd SOON, before I've lost all interest in it. In addition, there needs to be a way for a motivated packager to convince people to QA his work. Is sending email to fedora-devel saying "please QA my new widget package?" acceptable? Maybe a mailing list that automatically gets CC'd all new Fedora Meta New package Requests, NEEDSWORK, and REVIEWED keyword changes would help? Or maybe they should just go to fedora-devel? Anyway, just my $0.02. Hope you're still awake! --erik
