Replying one more time, because I certainly don't want to create any hard feeling here. On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote: > >>>> By keeping those sets of expectations, we've been able to keep Mesa pretty > >>>> clear of regressions, whilst having a very clear set of things that should > >>>> be fixed to point to. It would be great if those set of things were zero, > >>>> but it just isn't. Having that is far better than the two alternatives: > >>>> either not testing at all (obviously bad), or having the test always be red > >>>> so it's always ignored (might as well just not test). > >>> > >>> Isn't that what happens with flaky tests anyway? > >> > >> For a small minority of tests. Daniel was referring to whole test suites. > >> > >>> Even more so since we have 0 context when updating that list. > >> > >> The commit log can provide whatever context is needed. > > > > Sure, I've yet to see that though. > > > > There's in 6.6-rc1 around 240 reported flaky tests. None of them have > > any context. That new series hads a few dozens too, without any context > > either. And there's no mention about that being a plan, or a patch > > adding a new policy for all tests going forward. > > That does sound bad, would need to be raised in review. > > > Any concern I raised were met with a giant "it worked on Mesa" handwave > > Lessons learned from years of experience with big real-world CI > systems like this are hardly "handwaving". Your (and others) experience certainly isn't. It is valuable, welcome, and very much appreciated. However, my questions and concerns being ignored time and time again about things like what is the process is going to be like, what is going to be tested, who is going to be maintaining that test list, how that interacts with stable, how we can possibly audit the flaky tests list, etc. have felt like they were being handwaived away. I'm not saying that because I disagree, I still do on some, but that's fine to some extent. However, most of these issues are not so much an infrastructure issue, but a community issue. And I don't even expect a perfect solution right now, unlike what you seem to think. But I do expect some kind of plan instead of just ignoring that problem. Like, I had to ask the MT8173 question 3 times in order to get an answer, and I'm still not sure what is going to be done to address that particular issue. So, I'm sorry, but I certainly feel like it here. Maxime
Attachment:
signature.asc
Description: PGP signature
