On Oct 7, 2024, at 03:29, Kent Overstreet <kent.overstreet@xxxxxxxxx> wrote: > > On Sun, Oct 06, 2024 at 12:04:45PM GMT, Linus Torvalds wrote: >> On Sat, 5 Oct 2024 at 21:33, Kent Overstreet <kent.overstreet@xxxxxxxxx> wrote: >>> >>> On Sun, Oct 06, 2024 at 12:30:02AM GMT, Theodore Ts'o wrote: >>>> >>>> You may believe that yours is better than anyone else's, but with >>>> respect, I disagree, at least for my own workflow and use case. And >>>> if you look at the number of contributors in both Luis and my xfstests >>>> runners[2][3], I suspect you'll find that we have far more >>>> contributors in our git repo than your solo effort.... >>> >>> Correct me if I'm wrong, but your system isn't available to the >>> community, and I haven't seen a CI or dashboard for kdevops? >>> >>> Believe me, I would love to not be sinking time into this as well, but >>> we need to standardize on something everyone can use. >> >> I really don't think we necessarily need to standardize. Certainly not >> across completely different subsystems. >> >> Maybe filesystem people have something in common, but honestly, even >> that is rather questionable. Different filesystems have enough >> different features that you will have different testing needs. >> >> And a filesystem tree and an architecture tree (or the networking >> tree, or whatever) have basically almost _zero_ overlap in testing - >> apart from the obvious side of just basic build and boot testing. >> >> And don't even get me started on drivers, which have a whole different >> thing and can generally not be tested in some random VM at all. > > Drivers are obviously a whole different ballgame, but what I'm after is > more > - tooling the community can use > - some level of common infrastructure, so we're not all rolling our own. > > "Test infrastructure the community can use" is a big one, because > enabling the community and making it easier for people to participate > and do real development is where our pipeline of new engineers comes > from. Yeah, the CI is really helpful, at least for those who want to get involved in the development of bcachefs. As a new comer, I’m not at all interested in setting up a separate testing environment at the very beginning, which might be time-consuming and costly. > > Over the past 15 years, I've seen the filesystem community get smaller > and older, and that's not a good thing. I've had some good success with > giving ktest access to people in the community, who then start using it > actively and contributing (small, so far) patches (and interesting, a > lot of the new activity is from China) - this means they can do > development at a reasonable pace and I don't have to look at their code > until it's actually passing all the tests, which is _huge_. > > And filesystem tests take overnight to run on a single machine, so > having something that gets them results back in 20 minutes is also huge. Exactly, I can verify some ideas very quickly with the help of the CI. So, a big thank you for all the effort you've put into it! > > The other thing I'd really like is to take the best of what we've got > for testrunner/CI dashboard (and opinions will vary, but of course I > like ktest the best) and make it available to other subsystems (mm, > block, kselftests) because not everyone has time to roll their own. > > That takes a lot of facetime - getting to know people's workflows, > porting tests - so it hasn't happened as much as I'd like, but it's > still an active interest of mine. > >> So no. People should *not* try to standardize on something everyone can use. >> >> But _everybody_ should participate in the basic build testing (and the >> basic boot testing we have, even if it probably doesn't exercise much >> of most subsystems). That covers a *lot* of stuff that various >> domain-specific testing does not (and generally should not). >> >> For example, when you do filesystem-specific testing, you very seldom >> have much issues with different compilers or architectures. Sure, >> there can be compiler version issues that affect behavior, but let's >> be honest: it's very very rare. And yes, there are big-endian machines >> and the whole 32-bit vs 64-bit thing, and that can certainly affect >> your filesystem testing, but I would expect it to be a fairly rare and >> secondary thing for you to worry about when you try to stress your >> filesystem for correctness. > > But - a big gap right now is endian /portability/, and that one is a > pain to cover with automated tests because you either need access to > both big and little endian hardware (at a minumm for creating test > images), or you need to run qemu in full-emulation mode, which is pretty > unbearably slow. > >> But build and boot testing? All those random configs, all those odd >> architectures, and all those odd compilers *do* affect build testing. >> So you as a filesystem maintainer should *not* generally strive to do >> your own basic build test, but very much participate in the generic >> build test that is being done by various bots (not just on linux-next, >> but things like the 0day bot on various patch series posted to the >> list etc). >> >> End result: one size does not fit all. But I get unhappy when I see >> some subsystem that doesn't seem to participate in what I consider the >> absolute bare minimum. > > So the big issue for me has been that with the -next/0day pipeline, I > have no visibility into when it finishes; which means it has to go onto > my mental stack of things to watch for and becomes yet another thing to > pipeline, and the more I have to pipeline the more I lose track of > things. > > (Seriously: when I am constantly tracking 5 different bug reports and > talking to 5 different users, every additional bit of mental state I > have to remember is death by a thousand cuts). > > Which would all be solved with a dashboard - which is why adding the > bulid testing to ktest (or ideally, stealing _all_ the 0day tests for > ktest) is becoming a bigger and bigger priority. > >> Btw, there are other ways to make me less unhappy. For example, a >> couple of years ago, we had a string of issues with the networking >> tree. Not because there was any particular maintenance issue, but >> because the networking tree is basically one of the biggest subsystems >> there are, and so bugs just happen more for that simple reason. Random >> driver issues that got found resolved quickly, but that kept happening >> in rc releases (or even final releases). >> >> And that was *despite* the networking fixes generally having been in linux-next. > > Yeah, same thing has been going on in filesystem land, which is why now > have fs-next that we're supposed to be targeting our testing automation > at. > > That one will likely come slower for me, because I need to clear out a > bunch of CI failing tests before I'll want to look at that, but it's on > my radar. > >> Now, the reason I mention the networking tree is that the one simple >> thing that made it a lot less stressful was that I asked whether the >> networking fixes pulls could just come in on Thursday instead of late >> on Friday or Saturday. That meant that any silly things that the bots >> picked up on (or good testers picked up on quickly) now had an extra >> day or two to get resolved. > > Ok, if fixes coming in on Saturday is an issue for you that's something > I can absolutely change. The only _critical_ one for rc2 was the > __wait_for_freeing_inode() fix (which did come in late), the rest > could've waited until Monday. > >> Now, it may be that the string of unfortunate networking issues that >> caused this policy were entirely just bad luck, and we just haven't >> had that. But the networking pull still comes in on Thursdays, and >> we've been doing it that way for four years, and it seems to have >> worked out well for both sides. I certainly feel a lot better about >> being able to do the (sometimes fairly sizeable) pull on a Thursday, >> knowing that if there is some last-minute issue, we can still fix just >> *that* before the rc or final release. >> >> And hey, that's literally just a "this was how we dealt with one >> particular situation". Not everybody needs to have the same rules, >> because the exact details will be different. I like doing releases on >> Sundays, because that way the people who do a fairly normal Mon-Fri >> week come in to a fresh release (whether rc or not). And people tend >> to like sending in their "work of the week" to me on Fridays, so I get >> a lot of pull requests on Friday, and most of the time that works just >> fine. >> >> So the networking tree timing policy ended up working quite well for >> that, but there's no reason it should be "The Rule" and that everybody >> should do it. But maybe it would lessen the stress on both sides for >> bcachefs too if we aimed for that kind of thing? > > Yeah, that sounds like the plan then.
