At present we are running automated nightlies to catch
problems that slip past developers or only show up on
long runs, and filing bugs when they fail ... but the
decision of whether or not we are ready to push out a
new release is not yet criterion-based. We have to
start moving towards official release criteria.
I suggest that our initial release criteria should fall
into four categories:
(1) functional validation
100% passage of designated validation suites,
with a formal process for managing the functional
assertions to be tested (or designating specific
assertions to be compliance-optional)
(2) regression tests
100% passage of designated regression suites,
with a formal process for designating which
bugs do and do not require the creation of
new regression test cases.
(3) performance
individual and aggregate throughput measurements,
and key-event timings will be made with controlled
loads on specified hardware configuration, and
compared against performance targets, and a formal
process for defining the target metrics and requirements.
(4) reliability and robustness
a specified number of hours of (client perceived) error
free operation under continuous load (with specified
levels and characteristics), in the face of specified
error injections ... and a formal process for defining
the times, load characteristics, error injections, and
acceptable performance.
Does this seem like the right general form for our release criteria?
What changes would you suggest?
Once we agree on the general form of our release criteria, the next
steps are:
(a) put some stakes in the ground for the initial requirements
(knowing that they will evolve in scope, specificity, and
rigour)
(b) propose some processes for the review, approval, and
evolution of those standards, and the communication of
the (current) requirements and results to the community.
(c) set a date for the first release to be subject to these
criteria
comments?
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
