On Fri, Jul 21, 2017 at 2:59 PM, Chris Wilson <chris@xxxxxxxxxxxxxxxxxx> wrote: > My dream would be for the system to recognise which tests provide > coverage for a patch and rank them by relevance and then run until death > or timeout. Without such introspection, the best we can do is to have > the developer supply the selection via a common system of tags, one > approximation would be to supply tags based on altered files and > functions. Hmm, that seems doable. In principle I agree, in practice I haven't seen such an oracle that actually works. There's not just the coverage problem, but there's a huge process problem: If our oracle isn't perfect at constructing the test set, or if it's timeout based, there's a good chance that even the same (or very similar patch) will run different sets of testcases. That adds noise to CI results, and noise trains developers to ignore it (and no amount of reminding to look at detailed results will fix that). The other problem is that we need to be able to compare those test runs against a baseline. That means we must run the full regression test set on drm-tip, or pre-merge can't give you results. And drm-tip updates a few times a day already. We could probably batch more, but we must have enough machine time to get through a few full runs, without even taking pre-merge testing into account. That means even the full set can't really take more than perhaps 6-8h, even when you have a perfect oracle that selects the test set for each patch. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list Intel-gfx@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/intel-gfx
