On Fri, 2022-04-15 at 16:39 -0400, Kent Overstreet wrote: > On Fri, Apr 15, 2022 at 09:31:24AM -0400, James Bottomley wrote: > > I think the solution to the problem is to try to maintain highly > > mobile reference implementations and also keep constantly > > considering what the requirements are for mobility (both on the > > part of the reference implementation and the kernel). I also think > > that if we ever gave impementors the magic ability just to dump any > > old code in the kernel, they'd use it to take the lazy way out > > because it's a lot easier than trying to keep the reference > > implementation separate. > > > > The fact that most "reference" implementations don't conform to the > > above isn't something we should be encouraging by supplying > > compatibility APIs that paper over the problem and encourage API > > bloat. > > I think it might help if we had a > - standard set of review guidelines > - standard workflow > > for pulling in (vendoring, however it gets done) code from external > repositories. But that's precisely what we shouldn't have. For most standard implementations it shouldn't be done. zlib is a case in point: even if we could pull in 74k lines from an external repo to try to extract the 300 we need that's not going to be via anything like a standard process. Now there might be something we could assist with in standardising how reference implementations should be done to make them mobile to environments like embedded and the kernel, but if the project isn't structured like this, it's already pretty much too late. > Generally what I see in the community is that people do want to > support the kernel better, but then people see things like Greg's > response of "don't do that" and it turns people off. Instead, we > could > - recognize that it's already being done (e.g. with zstd) It hasn't been done with zstd. What we have looks like a reference implementation based on RFC 8478, it's nothing like the facebook github for zstd and I'm sure more people than Greg (me included) would object greatly to trying to swap our 300 line implementation for the facebook 74k line one. > - put some things in writing about how it _should_ be done This is where I think we could help. As I said: libxfs (and librcu as you point out below) already exists as a blueprint. For rust async, it's a bit more complex because it depends on whether the async support in the compiler can truly be made compatible with the kernel threading model rather than simply vendoring the async crates. > This could help a lot with e.g. the way Facebook maintains ZSTD - if > we say "look, we want this code factored out better so we only have > to review the stuff that's relevant" - the engineers generally won't > mind doing that work, and now they'll have something to take to their > managers as justification. Well, they do, they just don't expect to do it by us vendoring the reference implementation: https://github.com/facebook/zstd/issues/1638 In the above, the facebook maintainer is willing to extract the reference core and add it to the linux kernel. That seems to be a great result for us. > Another thing I'd like to see is more of what the RCU folks have done > - liburcu is _amazing_ in that, at least for me, it's been a drop in > replacement for the kernel RCU implementation - and I would imagine > it's as much as possible the same code. A good candidate would be the > kernel workqueue code: workqueues are _great_ and I don't know of > anything like them in userspace. That code should be > seeing wider use! And if it was made an external library that was > consumed by both the kernel and userspace, it could be - but it would > require buy in from kernel people. That might work for , but most other languages need something different (and often have their own implementation, like python: https://docs.python.org/3/library/queue.html or rabbitMQ or something). > But as you were saying about Facebook (and I discovered when I was at > Google many moons ago, as a young engineer) - large organizations > tend to be insular, they like te pretend the outside world doesn't > exist. The kernel is one such organization :) We could be a better > citizen by encouraging and enabling efforts such as this. Well, I think they are. To be clear, I think the way zstd is supported in the kernel is great. If it works for facebook (having the maintainer extract it to the kernel and send us pull requests), I don't see a problem with keeping it the way it is rather than trying to vendor the existing code. Regards, James
