> Sorry I haven't been following your qa-assistant very closely, I don't run > X most of the time. I did download the rpm from sourceforge, and took a look > at it. I'm assuming that version doesn't know how to save the .xml files you > referred to? > Yes. I wanted to get some feedback on the XML save format before releasing something that has a format that's just going to be changed in the next version. I've put up a snapshot of my current tree here (using the current DTD): http://qa-assistant.sf.net/snaps/ It has "save as" functionality but loading is currently broken. > Since you have taken the time to encapsulate the checklist as an XML file, > I think we should all start adopting THAT as the canonical reference, > including going so far as to write an xslt sheet to render it as html and > posting it on the website. > > I've created a first round xslt stylesheet, which can see the results of at > http://www.ilsw.com/~erik/fedoraus.xml > > The xslt stylesheet is at > http://www.ilsw.com/~erik/checklist.xsl > I like that idea alot. Your stylesheet is nice too. The text of the various entries could use some going over by others since they're currently just my interpretation of the wiki and a few other non-controversial suggestions. They could probably do with a bit of reformatting as well as I didn't anticipate them being displayed on a web page. > I also spent some time thinking about how to integrate our code. I'm going to > throw out an idea here, and you can see what you think. If you look at the > changes I made to the first couple of entries in fedoraus.xml, you can see I > added a <script> section, with some simple code in it. My thought is that the > best way to enforce a policy is to put the code to do it in the same place as > the policy definition, aka the checklist definition. > This is a great plan. I've been trying to think of the best way to add automated checking into QA Assistant and this definitely beats the pants off of anything I was thinking of. I'd call it <test> instead of <script> as that's more specific to what we're doing but the idea seems very sound. I'd like to integrate some sort of signature on the checklist though. That way people can tell they're about to execute a script that's part of an official Fedora Checklist, Michael Schwendt's Alternate Fedora Checklist, or Vader's Join The Darkside Luke Checklist. Not sure the best way to achieve this so the programs that utilize the checklist can check it, though. > I also created a simple python script to walk through the checklist, find the > scripts, and run them. With some tweaking, I think it could work admirably > both for an unattended checker, or integrated into a gui like yours. The > script is at http://www.ilsw.com/~erik/runchecklist.py > > Outstanding issues: > We need to agree on a way to define input parameters to pass into scripts, > ie things like a gpg keyring file, or the srpm filename, or ??? You can see > I started doing this, but I haven't finished the job by any means. > Do we want to allow different scripting languages to be used? For most scripting languages we could write the variables directly into the scriptlets. Alternately we could pass them as environment variables and have the script (or shell wrapper) compose them. Using these methods we don't need to worry about the order of the arguments, just what they are named. What things would it be okay for the scriptlet to ask for? Somehow the program either has to know or ask the user to supply the information the script wants in the first place. Should we have <param name="srpm_filename" query="What is the SRPM filename?"> so the program can supply srpm_filename if it already knows it or query the user for it when the script runs? > We need to agree on an output method. I think the easy way to do it is by > mapping return codes to states like I show in the example, or taking the > entire textual output of the program and matching it to a state name. We can > do both, they're both easy. > Return value is good for most things but sometimes the scriptlet could return a better output message than the program would otherwise. I don't want to teach the program what to do for each individual case in the checklist, however. Should we have a tag specify if it returns a message? <message>True</message> Should such scripts output the message as the last line? (We could have all scriptlets be in python and run them in-process so we can retrieve complex return values directly....) > Other comments: > I'm not sure I entirely understand the difference between qasave.dtd and > checklist.dtd. My personal preference is to generate the qasave dtd (xml > schema would be better) and xml directly from the checklist xml. Then there > is 1 and only 1 definition of the policy, description, and storage format. > Could definitely be done. Frugality of disk space led me to define two separate formats but now that I've implemented it once I think it adds unnecessary complexity. My thought on creating one format are that we'd have a base checklist that merged the checklist.dtd and qasave.dtd information together. The resolution would be set to 'Needs-Reviewing' for all entries. When a program ran through a checklist and wanted to save state it would write out a complete checklist file that was based on the original but with changes where the user changed states, edited the output message for a review item, etc. The properties element could use some reworking here as well. The base checklist could specify the properties the checklist supports and the type each is. [Thought: perhaps all input that the scriptlets could use should be a checklist property. Are there times when a scriptlet would need to know something that wasn't a property? gpg keyring is really a program preference for instance... I'd store preferences in GConf and the scriplets could extract it... but you might not want the command line script to need GConf (GConf isn't graphical, but it isn't something you expect a standard command-line app to need....)] > I think the checklist probably needs to carry more of the policy information > we've argued about (and not decided on) with it. IE I'd like for each entry > to have a "class" or something indicating if it was a Blocker, Non-Blocker, > Important, or Cosmetic only. > Each checklist item has a list of allowable resolution states: Needs-Reviewing, Pass, Fail, Non-Blocker, Not-Applicable. Each item must have Pass and Needs-Reviewing. Depending on whether I thought the item was definitively a Blocker or Minor or could fall into either category I added other resolutions. We could change names, add resolution states, and reassign which set of resolutions are available for each item pretty easily. Will that work or do you think we should have actual "classes" that predefine which set of resolutions are available? We'd still need to have the states array as we need to set the output messages on a per state basis.... > I think the checklist xml also needs to carry whatever information might be > needed to create a QA report in the affirmative or negative for that > particular entry. IE you should be able to create some sort of human friendly > listing of the packages status by transforming a "checklist data file" and a > "checklist" together. Maybe you already have this? > If I understand what your asking, this is basically what QA Assistant does. (Or will once loading of save files works correctly.) There might be some value to creating a command line tool to do this as well. Perhaps the commandline script that runs tests embedded in the checklist could use xml as its internal representation and then xslt transform it into a QA Report. Then, run-checklist.py --notest save.xml could do the transform on a saved checklist without running any new tests. -Toshio -- _______S________U________B________L________I________M________E_______ t o s h i o + t i k i - l o u n g e . c o m GA->ME 1999
Attachment:
signature.asc
Description: This is a digitally signed message part
