On Tue, Nov 01 2022, Taylor Blau wrote: > On Tue, Nov 01, 2022 at 04:01:18AM +0100, Ævar Arnfjörð Bjarmason wrote: >> > Yeah, I think it is crappy UX, too. It's just that I think the tests >> > should not _asserting_ the bad behavior. At most, they should tolerate >> > the bad behavior as a band-aid. So I think Dscho's patch is doing the >> > right thing (and I do agree that we should fix the immediate CI pain by >> > adjusting the tests, and letting the user-visible fix proceed >> > independently). >> >> The tests aren't just asserting the bad behavior, they're also ensuring >> that it doesn't get worse. 1 warning is ideal, 2-3 is bad, but >> tolerable, but if we start emitting 500 of these it would be nice to >> know. > > I admit that this kind of argument does not sway me. > > Is it likely that we would suddenly start spewing 500 such warnings? If > we did, are there no other tests that would catch it? And even if *that* > were the case, would nobody happen to notice it in the meantime either > during development or when we queue an affected topic onto 'next' for > wider testing? > > I guess the answer is that it's possible that we'd miss such a > regression in all of those above places, but to me it seems extremely > unlikely that we'd let such a regression through without noticing. Literally 500? Probably not, that was hyperbole to make a point, but several, low tens? Yeah, I know of at least a couple in-tree off the top of my head. The point, which I assumed was clear is that we literally wouldn't notice if it were 500, and that sort of thing is a common pattern in our tests. I.e. in most cases we'd ideally test_cmp known output (at least to the extent of assuring ourselves that we're getting it right). Instead we often just grep it, or don't test it at all. Sometimes for a good reason (e.g. the output containing absolute paths), but more often than not for no good reason.
