> On Nov 20, 2017, at 10:10 AM, Cyril Hrubis <chrubis@xxxxxxx> wrote: > > Hi! >> So why didn???t we report these? As mentioned we???ve been tossing out dodgy >> test cases to get to a clean baseline. We don???t need or want noise. >> >> For LTS, I want the system when it detects a failure to enable a quick >> bisect involving the affected test bucket. Given the nature of kernel >> bugs tho, there is that class of bug which only happens occasionally. > > From my experience debugging kernel bugs requires an actuall human > interaction and there is only certain level of automation that can be > achieved. Don't take me wrong, automatic bisection and other bells and > whistles are a nice to have, but at the end of the day you usually need > someone to reproduce/look at the problem, possibly check the source > code, report a bug, etc. Hence it does not make much sense to have an > automated system without dedicated engineers assigned to review the test > results. You are entirely right automation only gets so far. We have a few lines of defense that probably are worth a mention. 1) infra - sometimes results/runs need to be re-run for whatever reason. 2) triage - Crappy test case or something that is real? 3) kernel - bisecting etc We don’t have huge dedicated teams for each category but likewise each has a team. > -- > Cyril Hrubis > chrubis@xxxxxxx