On Mon, Aug 29, 2005 at 11:15:53AM -0700, Mary Edie Meredith wrote:
> Going forward with the memory regressions, I think people will be in a
> better position to refine the requirements, as they are in the middle of
> development. No matter what, though, we will need to start at a high
> level and work are way down to something that can actually be coded.
> Maybe the high level description should be called something other than
> test case?
'test case description' is fine. Yes, I agree that the process starts
at a high level, then there is a research phase that can be done as you
mentioned, and then the coding. I think the important thing we learned
that when that research step is done, it makes the coding extremely
straightforward. My thought is that identifying it as a different step
from coding might make it palatable for someone else to do; for
instance, given the high level description, you might be able to ask a
developer to read through it and off the top of his head give example
commandlines. The test developer would then be able to flesh out from
there into a full test.
We found that having an even one-off test script from a developer was
extremely useful because even though it might be really particular to
their environment, it captured the names of commands/tools to run,
useful options, and so forth. So, one approach could be just to collect
together any test snippets, examples, and so forth.
> It was not stated in the test case itself, but perhaps I should next
> time? Or was the info in the matrix not sufficient...
I think a bit more background info would be useful. The matrix only had
a brief 1 sentence each. For some cases that was enough, but in hindsight
cases 1, 5, and 6 could have benefitted from a bit more elaboration.
> > 4. Pseudocode. We found that after reviewing the test cases,
> > implementing them in crude pseudocode was worthwhile, because it
> > helped identify the basic logic in a form that could be easily
> > translated into bash. This intermediate format was also simple
> > enough that others could review and comment on it without having to
> > dig through extraneous structural and error checking code.
> >
> > The key objective for writing pseudocode is to work out solutions
> > for the technical details, like how to operate various tools or
> > workloads, or to define algorithms for parsing output from other
> > tools, order of operations, loop structures, and so forth. Things
> > like error handling, output formatting, etc. can be left for the
> > implementor to do.
>
> I'd like to add that into the matrix. Do you have psuedocode posted
> already for the CPU test cases?
Yes, they're in the doc/ directory of the lhcs_regression package. I
also posted them to the mailing list a month or two ago.
http://developer.osdl.org/dev/hotplug/tests/
> Can we work this week to get the database test case running (you
> reported the PPC64 can be enabled now... and I'd like to run it on that
> system rather then reimplementing the DBT2 postresql workload on
> yours...)
Sure, yes I'd love to see how that test case runs.
Bryce
