On Wed, Jan 17, 2024 at 08:03:35AM -0500, James Bottomley wrote: > On Wed, 2024-01-17 at 00:54 -0500, Theodore Ts'o wrote: > > On Tue, Jan 16, 2024 at 11:41:25PM -0500, Kent Overstreet wrote: > > > > > No, it's a leadership/mentorship thing. > > > > > > > > > > And this is something that's always been lacking in kernel > > > > > culture. Witness the kind of general grousing that goes on at > > > > > maintainer summits;maintainers complain about being overworked > > > > > and people not stepping up to help with the grungy > > > > > responsibilities, while simultaneously we still > > > > <blah blah blah> > > > > > > > Tests and test infrastructure fall into the necessary but not > > > > > fun category, so they languish. > > > > > > > > No, they fall into the "no company wants to pay someone to do the > > > > work" category, so it doesn't get done. > > > > > > > > It's not a "leadership" issue, what is the "leadership" supposed > > > > to do here, refuse to take any new changes unless someone ponys > > > > up and does the infrastructure and testing work first? That's > > > > not going to fly, for valid reasons. > > > > Greg is absolutely right about this. > > > > > But good tools are important beacuse they affect the rate of > > > everyday development; they're a multiplier on the money everone is > > > spending on salaries. > > > > Alas, companies don't see it that way. They take the value that get > > from Linux for granted, and they only care about the multipler effect > > of their employees salaries (and sometimes not even that). They most > > certainly care about the salutary effects on the entire ecosyustem. > > At least, I haven't seen any company make funding decisions on that > > basis. > > Actually, this is partly our fault. Companies behave exactly like a > selfish contributor does: > > https://archive.fosdem.org/2020/schedule/event/selfish_contributor/ > > The question they ask is "if I'm putting money into it, what am I > getting out of it". If the answer to that is that it benefits > everybody, it's basically charity to the entity being asked (and not > even properly tax deductible at that), which goes way back behind even > real charitable donations (which at least have a publicity benefit) and > you don't even get to speak to anyone about it when you go calling with > the collecting tin. If you can say it benefits these 5 tasks your > current employees are doing, you might have a possible case for the > engineering budget (you might get in the door but you'll still be > queuing behind every in-plan budget item). The best case is if you can > demonstrate some useful for profit contribution it makes to the actual > line of business (or better yet could be used to spawn a new line of > business), so when you're asking for a tool, it has to be usable > outside the narrow confines of the kernel and you need to be able to > articulate why it's generally useful (git is a great example, it was > designed to solve a kernel specific problem, but not it's in use pretty > much everywhere source control is a thing). > > Somewhere between 2000 and now we seem to have lost our ability to > frame the argument in the above terms, because the business quid pro > quo argument was what got us money for stuff we needed and the Linux > Foundation and the TAB formed, but we're not managing nearly as well > now. The environment has hardened against us (we're no longer the new > shiny) but that's not the whole explanation. I think this take is closer to the mark, yeah. The elephant in the room that I keep seeing is that MBA driven business culture in the U.S. has gotten _insane_, and we've all been stewing in the same pot together, collectively boiling, and not noticing or talking about just how bad it's gotten. Engineering culture really does matter; it's what makes the difference between working effectively or not. And by engineering culture I mean things like being able to set effective goals and deliver on them, and have a good balance between product based, end user focused development; exploratory, prototype-minded research product type stuff; and the "clean up your messes and eat your vegetables" type stuff that keeps tech debt from getting out of hand. Culturally, we in the kernel community are quite good on the last front, not so good on the first two, and I think a large part of the reason is people being immersed in corporate culture where everything is quarterly OKRs, "efficiency", et cetera - and everywhere I look, it's hard to find senior engineering involved in setting a roadmap. Instead we have a lot of "initiatives" and feifdoms, and if you ask me it's a direct result of MBA culture run amuck. Culturally, things seem to be a lot better in Europe - I've been seeing a _lot_ more willingness to fund grungy difficult long term projects there; the silicon valley mentality of "it must have the potential for a massive impact (and we have to get it done as quick as possible) or it's not worth looking at" is, thankfully, absent there. > I also have to say, that for all the complaints there's just not any > open source pull for test tools (there's no-one who's on a mission to > make them better). Demanding that someone else do it is proof of this > (if you cared enough you'd do it yourself). That's why all our testing > infrastructure is just some random set of scripts that mostly does what > I want, because it's the last thing I need to prove the thing I > actually care about works. It's awkward because the people with the greatest need, and therefore (in theory?) the greatest understanding for what kind of tools would be effective, are the people with massive other responsibilities. There are things we just can't do without delegating, and delegating is something we seem to be consistently not great at in the kernel community. And I don't think it needs to be that way, because younger engineers would really benefit from working closely with someone more senior, and in my experience the way to do a lot of these tooling things right is _not_ to build it all at once in a year of full time SWE salary time - it's much better to take your time, spend a lot of time learning the workflows, letting ideas percolate, and gradually build things up. Yet the way these projects all seem to go is we have one or a few people working full time mostly writing code, building things with a lot of _features_... and if you ask me, ending up with something where most of the features were things we didn't need or ask for and just make the end result harder to use. Tools are hard to get right; perhaps we should be spending more of our bikeshedding time on the lists bikeshedding our tools, and a little bit less on coding style minutia. Personally, I've tried to get the ball rolling multiple times with various people asking them what they want and need out of their testing tools and how they use them, and it often feels like pulling teeth. > Finally testing infrastructure is how OSDL (the precursor to the Linux > foundation) got started and got its initial funding, so corporations > have been putting money into it for decades with not much return (and > pretty much nothing to show for a unified testing infrastructure ... > ten points to the team who can actually name the test infrastructure > OSDL produced) and have finally concluded it's not worth it, making it > a 10x harder sell now. The circle of fail continues :)
