On Thu, Jul 20, 2017 at 6:23 PM, Martin Peres <martin.peres@xxxxxxxxxxxxxxx> wrote: > Hi everyone, > > As some of you may already know, we have made great strides in making our CI > system usable, especially in the last 6 months when everything started > clicking together. > > The CI team is no longer overwhelmed with fires and bug reports, so we > started working on increasing the coverage from just fast-feedback, to a > bigger set of IGT tests. > > As some of you may know, running IGT has been a challenge that few manage to > overcome. Not only is the execution time counted in machine months, but it > can also lead to disk corruption, which does not encourage developers to run > it either. One test takes 21 days, on its own, and it is a subset of another > test which we never ran for obvious reasons. > > I would thus like to get the CI team and developers to work together to > decrease sharply the execution time of IGT, and get these tests run multiple > times per day! > > There are three usages that the CI team envision (up for debate): > - Basic acceptance testing: Meant for developers and CI to check quickly if > a patch series is not completely breaking the world (< 10 minutes, timeout > per test of 30s) > - Full run: Meant to be ran overnight by developers and users (< 6 hours) > - Stress tests: They can be in the test suite as a way to catch rare > issues, but they cannot be part of the default run mode. They likely should > be run on a case-by-case basis, on demand of a developer. Each test could be > allowed to take up to 1h. > > There are multiple ways of getting to this situation (up for debate): > > 1) All the tests exposed by default are fast and meant to be run: > - Fast-feedback is provided by a testlist, for BAT > - Stress tests ran using a special command, kept for on-demand testing > > 2) Tests are all tagged with information about their exec time: > - igt@basic@.*: Meant for BAT > - igt@complete@.*: Meant for FULL > - igt@stress@.*: The stress tests > > 3) Testlists all the way: > - fast-feedback: for BAT > - all: the tests that people are expected to run (CI will run them) > - Stress tests will not be part of any testlist. > > Whatever decision is being accepted, the CI team is mandating global > timeouts for both BAT and FULL testing, in order to guarantee throughput. > This will require the team as a whole to agree on time quotas per > sub-systems, and enforce them. > > Can we try to get some healthy debate and reach a consensus on this? Our CI > efforts are being limited by this issue right now, and we will be doing > whatever we can until the test suite becomes saner and runnable, but this > may be unfair to some developers. > > Looking forward to some constructive feedback and intelligent discussions! > Martin Imo the critical bit for the full run (which should regression test all features while being fast enough that we can use it for pre-merge testing) must be the default set. Default here means what you get without any special cmdline options (to either the test or piglit), and without any special testlist that are separately maintained. Default also means that it will be included by default if you do a new testcase. There's two reasons for that: - Maintaining a separate test list is a pain. Also, it encourages adding tons of tests that no one runs. - If tests aren't run by default we can't test them pre-merging before they land in igt and wreak havoc. Second, we must have a reasonable runtime, and reasonable runtime here means a few hours of machine time for everything, total. There's two reasons for that: - Only pre-merge is early enough to catch regressions. We can lament all day long, but fact is that post-merge regressions don't get fixed or handled in a timely manner, except when they're really serious. This means any testing strategy that depends upon lots of post-merge testing, or expects such post-merge testing to work, is bound to fail. Either we can test everything pre-merge, or there's no regression testing at all. - We can't mix together multiple patch series bisect autobisecting is too unreliable. I've been promised an autobisector for 3 years by about 4 different teams now, making that happen in a reliable way is _really_ hard. Blocking CI on this is not reasonable. Also, the testsuite really should be fast enough that developers can run it locally on their machines in a work day. Current plan is that we can only test on HSW for now, until more budget appears (again, we can lament about this, but it's not going to change), which means developers _must_ be able to run stuff on e.g. SKL in a reasonable amount of time. Right now we have a runtime of the gem|prime tests of around 24 days, and estimated 10 months for the stress tests included. I think the actual machine time we'll have available in the near future, on this HSW farm is going to allow 2-3h for gem tests. That's the time budget for this default set of regression tests. Wrt actually implementing it: I don't care, as long as it fulfills the above. So tagging, per-test comdline options, outright deleting all the tests we can't run anyway, disabling them in the build system or whatever else is all fine with me, as long as the default set doesn't require any special action. For tags this would mean that untagged tests are _all_ included. Cheers, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list Intel-gfx@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/intel-gfx
