On 2025/02/08 2:14, Konstantin Ryabitsev wrote:
> On Fri, Feb 07, 2025 at 05:16:28AM +0900, Hector Martin wrote:
>> And what I see, is very little effort to improve that status quo, or at
>> least very little that yields any actual change that isn't just
>> band-aids (e.g. tooling like b4, which is nice and appreciated, but
>> doesn't fix any underlying issues). And that's not going to change no
>> matter how many long technical arguments we have on the MLs (or even off
>> MLs, since MLs are also not particularly good for this, and I've seen
>> multiple arguments only reach a resolution after being redirected to IRC).
>
> From my perspective, there are several camps clashing when it comes to the
> kernel development model. One is people who are (rightfully) pointing out that
> using the mailing lists was fine 20 years ago, but the world of software
> development has vastly moved on to forges.
>
> The other camp is people who (also rightfully) point out that kernel
> development has always been decentralized and we should resist all attempts to
> get ourselves into a position where Linux is dependent on any single
> Benevolent Entity (Github, Gitlab, LF, kernel.org, etc), because this would
> give that entity too much political or commercial control or, at the very
> least, introduce SPoFs.
>
> At best, I can hope to make both camps grumpily agree to coexist.
>
> I *am* very wary of Benevolent Entities, because we have too many very recent
> examples of companies "realigning priorities" when political winds shift.
> Programs and initiatives that have until recently been poster board examples
> of progress and benevolence are shuttered and defunded. I am concerned that
> we're only a couple of mood swings away from someone deciding that free
> software should not be allowed to exist because it benefits America's foes.
> Many of us remember all too well when large tech giants treated Linux as a
> "cancer" to be opposed, and I can certainly see that idea easily re-entering
> some Big Brain in Charge.
>
> From my perspective, I would like to ensure that Linux development can
> continue without a hard dependency on a single centralized forge -- whether
> controlled by a large commercial entity, or even a standalone one that is
> operated by kernel.org. It's becoming shockingly difficult to operate a public
> resource on the web unless you're willing to put it behind a large commercial
> CDN that will protect you from hostile bots (and if you do that, you're back
> to depending on the whims of a Benevolent Entity).
>
> We're trying to get lore.kernel.org to the point where it's like a global
> messaging bus that is indexed and searchable. Currently, you mostly have to
> send things to a mailing list for them to end up on lore, but it's gradually
> becoming less and less the case. We're already bridging with bugzilla and we
> should be able to bridge with forges soon, too (currently delayed again
> because I'm scrambling to move kernel.org frontends away from Equinix). Who
> knows, we may be actually leapfrogging the forge era of software development
> straight into "AI" agents era -- but that remains to be seen.
>
> Anyway, all of this is to say that I'm happy that you've found b4 useful, but
> I wouldn't view it as a band-aid -- it's just a very small and email-centric
> way to interact with the kernel lore.
>
The centralization concern is valid, but there are technical solutions
to this, such as forge federation. It's possible to set up a forge
environment to be less of a SPoF, such as by ensuring all data is open
and archiveable to allow for migration and backup restore, even by third
parties (you can make practically ~all forge data public except for
login passwords, and we have email-based reset processes for those).
It's also possible to simply shard, and let different subsystems choose
their own forge infrastructure, so downtime has a more limited effect.
Meanwhile, for better or worse, much of Linux infra *is* centralized -
for example, the mailing lists themselves, and a lot of the Git hosting.
There's also the fact that there is enough support for Linux that
finding a Benevolent Entity to provide stuff like DDoS protection is not
particularly hard, and such services are easily replaceable should
priorities change (and indeed are the kind of service that can only be
provided by a large entity, whether paid or sponsored, by nature). Heck,
even I have a Benevolent Entity sponsoring a CDN frontend for Asahi
sites and installer images (bunny.net, they're nice, highly recommended).
At the end of the day, I do not believe a theoretical breakdown of Linux
infra would be a major long-term setback to Linux kernel development.
The kernel already survived the great BitKeeper to Git migration, and
this time there's no proprietary software to be yanked out from under
the kernel (at least assuming the forge of choice is FOSS and
self-hostable, which seems like a no-brainer given the concerns). As
long as everyone has the Git trees, infrastructure can be rebuilt. As
long as everyone has email and git, development can continue ad-hoc.
Sure, it would cause a blip, but it's not like we would switch to a
forge, the forge would go down for a few days, and it would not be the
end of the world. Just ask the freedesktop folks (a good case study on
how infra is hard, but not impossible, and even if imperfect, even if at
times frustrating, it still functions, and well enough to still be
wildly positive compared to not having that infra at all).
But I'm afraid you'll find much if not most of the true opposition to
forges is not technical, it is philosophical or preference-based (even
though it may be presented as technical opposition, sometimes to
intentionally mislead). This is, in fact, quite a mirror of the R4L
situation, where technical arguments ("show me you can write a real
driver") quickly lead to non-technical arguments when solutions are
proposed ("it's cancer").
I actually considered moving soc/apple development to a forge personally
in the near future (obviously not my call to make any more), and I was
fully expecting a pile of pushback, "because that's not how we do things
here". Who knows, I might have gotten a "fuck you, either you accept
email patches or I remove you from MAINTAINTERS" from Linus.
All that said, thanks for b4. I might have given up much earlier, not
due to flamewars but due to exhaustion with poor tooling, if it didn't
exist.
- Hector
