On 2019/1/23 下午12:18, Dave Chinner wrote: > On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote: >> >> >> On 2019/1/17 上午10:25, Dave Chinner wrote: >>> On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote: >>>> On 2019/1/17 上午8:16, Dave Chinner wrote: >>>>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: >>>>>> E.g. one operation should finish in 30s, but when it takes over 300s, >>>>>> it's definitely a big regression. >>>>>> >>>>>> But considering how many different hardware/VM the test may be run on, >>>>>> I'm not really confident if this is possible. >>>>> >>>>> You can really only determine performance regressions by comparing >>>>> test runtime on kernels with the same features set run on the same >>>>> hardware. Hence you'll need to keep archives from all your test >>>>> machiens and configs and only compare between matching >>>>> configurations. >>>> >>>> Thanks, this matches my current understanding of how the testsuite works. >>>> >>>> It looks like such regression detection can only be implemented outside >>>> of fstests. >>> >>> That's pretty much by design. Analysis of multiple test run results >>> and post-processing them is really not something that the test >>> harness does. The test harness really just runs the tests and >>> records the results.... >> >> What about using some other telemetry other than time to determine >> regreesion? >> >> In my particular case, the correct behavior, some reading like >> generation would only increase by a somewhat predictable number. >> >> While when the regression happens, the generation will go way higher >> than expectation. > > That's something that would be done inside the test, right? i.e. > this has nothing to do with the test harness itself, but is a > failure criteria for the specific test? Yep, a failure criteria for that specific test. > >> Is it acceptable to craft a test case using such measurement? > > If it's reliable and not prone to false positives from future code > changes, yes. Glad to know that. Thanks, Qu > > Cheers, > > Dave. >
Attachment:
signature.asc
Description: OpenPGP digital signature
