Bryce, Thanks for taking the time to write this up! (Delayed reaction due to vacation.) Comments below: On Wed, 2005-08-24 at 12:37 -0700, Bryce Harrington wrote: > Lessons Learned from Hotplug CPU Test Case Development > ====================================================== > > Looking back, the Hotplug CPU test case effort worked pretty well. We > were able to involve a lot of different people with different levels of > skill in the process, such that the actual coding of the test cases was > fairly straightforward. > > I think we benefit from following this same model as we start looking at > the memory hotplug testing, and based on the experience with CPU I have > some thoughts on how we could improve the process. The following is > divided into "design-time" and "implementation-time" thoughts. > > > TEST CASE DESIGN > ================ > 1. Make the test descriptions specific - We found that some of the CPU > test cases were easier to code than others, because sometimes items > were described ambiguously. > > For example, "Verify the flux capaciter works" gives you four > questions: At what point do we do the verification? What's a flux > capaciter? How do you know that it works? What do we do if it > doesn't work? > > A better description would say, "After switching it on, check that > LED42 is green on the Flux Capacitor Status Panel, and mark test > FAILED if it isn't." > > Example commandlines are extremely helpful. For instance, instead > of saying, "Use 'foo' to check that bar works," even more useful > would be, "Run something like 'foo -a -b 3 -c 10', and verify that > 'bar' is listed in the output. Just an FYI, it was two relatively ignorant people (markw and myself) who wrote up the test plan. The information is vague exactly because we didn't know how to do what was stated. We knew we would have to research how, unfortunately you got stuck with the effort. Thanks for the hard work to make it happen. Going forward with the memory regressions, I think people will be in a better position to refine the requirements, as they are in the middle of development. No matter what, though, we will need to start at a high level and work are way down to something that can actually be coded. Maybe the high level description should be called something other than test case? > > 2. State even the obvious. With hotplug01, one of the things that > really slowed us down was that we didn't know what an "Affinity > Mask" was, nor how to generate the hex masks, verify it, etc. It > was only after seeing someone else's test case that we finally > understood. In the test definitions, be sure to define new concepts > or give pointers to examples. If unfamiliar terminology gets in the > way, it can really slow down implementation. > > Also, when writing the test case description, write down a few > sentences describing what the test case should do, and what issue it > addresses. In some cases we had to do some research to back out > what the test cases were for, and it'd save time to have this up > front. The person writing the test case probably already has this > in mind at the time they write it, so it should be easy to jot down > early on. Sorry, I thought I did that in the matrix (see "Testing what common regression?" ) http://developer.osdl.org/maryedie/HOTPLUG/planning/hotplug_cpu_test_plan_status.html It was not stated in the test case itself, but perhaps I should next time? Or was the info in the matrix not sufficient... > > 3. Avoid reliance on interactive tools. Some of the biggest challenges > we had with implementing the test cases was with phrases that > specifying monitoring with interactive tools. For example, "Start a > and b, then watch on top to make sure c happens." top is easy for > use in interactive testing, but for test cases it can be complicated > to rely on tools like these. We were able to usually find some > other approach, such as extracting from a file in /proc. Sorry I thought top could record as well as be interactive (remembering a top from another UNIX implementation, no doubt). My mistake. > > 4. Pseudocode. We found that after reviewing the test cases, > implementing them in crude pseudocode was worthwhile, because it > helped identify the basic logic in a form that could be easily > translated into bash. This intermediate format was also simple > enough that others could review and comment on it without having to > dig through extraneous structural and error checking code. > > The key objective for writing pseudocode is to work out solutions > for the technical details, like how to operate various tools or > workloads, or to define algorithms for parsing output from other > tools, order of operations, loop structures, and so forth. Things > like error handling, output formatting, etc. can be left for the > implementor to do. I'd like to add that into the matrix. Do you have psuedocode posted already for the CPU test cases? > > > TEST CASE IMPLEMENTATION > ======================== > 5. Keep It Short and Simple (KISS) - We made each test case focused on > a specific kind of test. We preferred breaking things like file > parsing, workloads, etc. into separate scripts than to try to pile > everything into a single script. Do one thing and do it well. > We also designed the test cases so they can be run one time through > very quickly, or optionally be run in loops; this way a developer > could use the suite as a quick sanity check, while a tester could > configure it to run continously under a workload. > > Think of the individual test cases as building blocks to assemble > more complex tests from later. By keeping the test cases simple, > and doing more sophisticated logic at a higher level, it gives > everyone more flexibility and power for future testing. I'm not sure this works for the database stress test, keeping it "simple", that is. Iterations will be fine, certainly other tests that are stressful should have a separate script... and so forth. Can I assume the database is installed already? The database load (flat) files created already? > > 6. Parameterize with environment variables. In general, each test case > should run with no parameters or with an absolute minimum, so it's > easy to use. However, the test cases are more useful if you can > override certain internal settings such as the number of loops to > run through. > > We found the most convenient way to do this in bash was with > environment variables. Here is the syntax we used: > > HOTPLUG06_LOOPS=${HOTPLUG06_LOOPS:-${LOOPS}} > loop_six=${HOTPLUG06_LOOPS:-1} > > This allows the user to override the looping for this specific test > case by setting the environment variable $HOTPLUG06_LOOPS, or allow > them to set looping for _all_ test cases via the variable $LOOPS. > By default, if neither parameter is specified, it defaults to 1 > loop. > > When describing test cases, also think about other ways to > parameterize it, such as length of time to sleep between various > operations, temporary file names, names and paths for input or > output files, commands that may be platform-specific, etc. > > 7. Cleanup after yourself. The CPU hotplug test cases turned CPUs on > and off. Obviously, it could be annoying if the test suite left > some of your processors off that weren't off before when it > finished! > > We adopted the practice of working to ensure the test cases each > left the system more or less as it found it. Thus, a test case that > attempts to turn on and off all CPUs would keep track of which CPUs > were on or off at the start, then do its testing, and then restore > the CPU's to the original on/off states. > > We also liked Ashok Raj's approach of trapping user interrupt > signals to perform cleanups: > > trap "do_intr" 1 2 15 > > do_intr() > { > echo "HotPlug01 FAIL: User interrupt" > do_clean > exit 1 > } > > This is in test case 1, with plans to add it to all the other test > cases. Can we work this week to get the database test case running (you reported the PPC64 can be enabled now... and I'd like to run it on that system rather then reimplementing the DBT2 postresql workload on yours...) > > > _______________________________________________ > Hotplug_sig mailing list > Hotplug_sig@xxxxxxxxxxxxxx > https://lists.osdl.org/mailman/listinfo/hotplug_sig
