Hi,
> I just wanted to reach out about these repos and the options and
> tradeoffs with porting them over to the backports.git project in a way
> that will benefit the larger community and our existing users.
Can you explain what you're trying to achieve by that?
And even, what is the reason to have these out-of-tree versions of the
drivers?
The backports project doesn't in itself work with out-of-tree drivers.
It _creates_ out-of-tree drivers, yes, but it takes the source of these
always from the latest upstream kernel.
So since I'm not sure what the github trees are even used for, let's say
1) these are used to provide additional fixes or hardware support over
what the upstream kernel already has
-> this is something backports will not support, the source driver
must be fully upstream
(OK, strictly speaking, that's not true - you could patch the
upstream driver before generating backports, but that seems
painful)
2) these are just used to provide newer drivers to users of
older kernels
-> this is exactly what backports pretty much already does
> I am able to begin the work on these drivers if we decide to try and
> merge them into backports.git,
>
They probably already ship in the current backports, since we generally
include all wifi drivers. However, it hasn't gotten a lot of attention
in the past few years other than our (Intel's) internal maintenance for
iwlwifi.
> but I need to understand a few things
> since my understanding is often the biggest limitation in all of this.
Sure :)
> On more than one occasion, I have gotten confused about how Linux
> backports does things so I want to make sure I am making the right
> assumptions about how it works with respect to commit history and a few
> other things for their respective drivers.
>
> My current understanding re: commit history is as follows:
>
> The main Linux backports project does not preserve the commit history of
> individual drivers as they exist in their original trees that they are
> taken from.
Correct.
> The main goal of the backports project is taking code from
> newer Linux kernel versions and adapting them so they can be compiled
> and run on older kernel versions.
Yes.
> This backporting process is automated
> by applying a series of patches to make the newer code compatible with
> the older kernel via automation defined in the backports.git repo
> through Makefile, Kconfig, dependent .c or .h files, etc. and of course
> the "semantic patch language" scripts of coccinelle.
Yes.
> Because the goal is to make new code work with old kernels, rather than
> to maintain a faithful, version-controlled history of that code, the
> commit history for individual drivers is not included in project's
> focus. The version control history in the backports.git repository
> serves to track changes to the backporting process itself, rather than
> to preserve the history of the backported code (I know that may be
> obvious, but I need to make sure I am not mistaken.)
Yes.
There _is_ a script (devel/git-tracker.py) that can "port" over history
from an upstream tree to a backport-generated git tree, but I'm not sure
I'd generally recommend using it. It's geared fairly specifically to our
internal development process which is actually that people work in such
a backport tree (we even publish a version of it [1]).
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/
> In a nutshell, the Linux backports project is geared towards
> compatibility and functionality, not towards historical preservation of
> codebase evolution.
It's just not relevant - the historical preservation of the code base
happens in the upstream Linux kernel project.
> The second thing I want to make sure I understand is how users can get
> backports for a driver via the Linux backports project:
>
> A happy path would look like them pulling down a tar of backports for
> their kernel version from
> https://cdn.kernel.org/pub/linux/kernel/projects/backports/stable/ and
> then running something like:
>
> ```sh
> # as a user
> make defconfig-rtlwifi
> make -j4
> # as root
> make install
> # reboot and enjoy
> ```
Yes, pretty much. It'd require someone actually maintaining backports
more reliably and creating packages though :)
> If a feature they need is not in a stable version, they will then have
> to execute a small series of extra steps to compile their own backports
> release and then use that; whether it's against linux-next or some other
> tree. This is a documented and relatively well-traveled path in kernel
> developer terms, but is likely out of reach for many "regular" users.
Well you could do that, but the backports maintainer could, in theory,
also offer a snapshot version.
This stuff is also fairly well scripted, I don't think it's _that_ much
more effort than "make ..." stuff above? Though it does require more
tooling (like spatch) to be installed.
> I
> think it is fair to say that this would require them to be a closer to a
> "developer" or experienced Linux user or administrator. They also may
> not know about or be able to find the docs for doing so. A GitHub repo
> with their device driver name is often what they find first.
If it exists ;-)
I mean, if you wanted that, you _could_ generate such a repo with
different tags/branches/etc. even with individual upstream commits using
the git-tracker tool. But it can take a long time to run and it's often
difficult to get the backports to even work for all individual commits,
so you'd probably care more about specific snapshots like RCs etc.
> The last thing I want to make sure I have straight is the value of
> backports.git for wireless drivers generally because there seem to be many:
>
> For maintainers/developers, backports.git offers a standardized,
> automated system that minimizes errors, streamlines many aspects of
> building, testing, and iterating, and allows for community-vetted (more
> eyes), secure code. This centralization frees maintainers to focus on
> other development priorities like fixing/reproducing bugs and landing
> patches upstream.
Well, I think that's overselling it a bit, given that it's been
languishing. We could also probably contribute more fixes than we
currently did, purely based on our internal version.
But yeah, it was _meant_ to basically make it easier to do backports, by
having everyone pitch in together ... did that happen? Not that much.
> For users, backports.git provides a stable way to access these new
> features and fixes. It also can be quite simple if you have the right
> kernel version, it lines up with the feature you need, or you have the
> requisite experience and can find then dive into more "developer-like"
> documentation.
I'm not sure it's really easy to use right now because there's no
regular maintenance.
> For the community, integrating the drivers into backports.git
>
I'm not sure I'd argue there's even a notion of "integrating the drivers
into backports.git". After all, backports.git doesn't contain drivers.
Though yeah, arguably having a driver actually work with backports takes
some work on the backports side, something you could call "integration".
> holds
> benefits for downstream projects like OpenWrt, which already rely on
> backports as a core design feature. Therefore OpenWrt and similar
> projects can more seamlessly and securely include them in their
> distributions, enhancing both device support and the end-user
> experience. In addition, the upstream drivers benefit due to increased
> visibility and accessibility, encouraging wider community engagement for
> testing, bug reporting, and code contributions.
:)
> This approach serves to unify efforts, improve security, and facilitate
> a solid, shared pipeline for maintainers, users, and the larger community.
>
> In conclusion, there are a few questions I am trying to answer for
> myself, but could use any feedback on.
>
> 1. How critical is the preservation of commit history for effective
> debugging and bisecting when issues arise in the backported drivers? We
> have the commit history in our GitHub repos, but they would not be
> present in drivers compiled from backports.git. Is there a way to have
> "the best of both worlds" or mitigate the effects of lacking that history?
Like I said above, you _could_ use git-tracker and create a new backport
for every (relevant) upstream commit, but then you end up fixing a lot
of intermittent backport issues:
Say some commit is introduced upstream that makes a patch in backports
no longer apply, but then let's say that commit is reverted pretty soon.
If you just follow the RCs, you might never notice. If you follow
individual commits, you'd have to adjust backports *twice* over it.
> 2. Are there methods to automate or simplify the process for users who
> need features not yet available in stable releases, to reduce the skill
> barrier?
It's not _that_ hard, basically just running "./gentree.py ..."?
Not sure what we could simplify there.
> 3. How can I help the backports project directly (rtw related or not)?
Depends how much work you want to put in? I think what'd help most right
now would be to actually have someone to do what Hauke used to do, which
was integrating patches from the list (and we'd probably have more of
them), and creating releases.
> 4. How can we best leverage the years of effort and modifications (#if
> #else #endif macros) made by Larry in these stand-alone repositories to
> inform the technical work of porting them to the shared backports framework?
That's tricky, I don't think there's a good way to leverage that
directly. Also, backports tries to _minimize_ the ifdefs needed in
favour of providing spatches and API changes.
Also, looking at the rtw89 repo, I notice that it _only_ contains the
driver and attempts to make it compatible with all mac80211 versions,
which we had previously though of as basically a losing proposition, and
so backports ships the _entire_ wireless stack so that the coupling
between cfg80211/mac80211/driver is maintained.
As a result, I think the only way to use this would be for inspiration
when looking at something that backports didn't (yet) handle (well).
> 5. Given the popularity and visibility of the rtw88 and rtw89 GitHub
> repositories, as indicated by their star counts, issue activity, and
> search engine ranking, what strategies can we employ to maintain this
> level of user engagement and support if we transition to backports.git?
> Would mirroring the code and providing custom-tailored instructions in
> the READMEs serve dual purposes of user support and elevating the
> visibility of backports.git in the larger community? Is this a welcomed
> proposition?
Given that nobody really seems to have much time to spend on the
backports project ... I think that'd be up to you.
You could generate backports for all Linux RCs and commit them to those
repos for example, in ascending order? You could even try to use the
git-tracker.py to generate individual commits, but as I described that
has some caveats. Or even provide linux-next snapshots in a separate
branch?
As long as you don't think of "backport outputs a tarball", I think you
can see a lot of choices, although not all of them will preserve a
patch-by-patch history of the driver.
> I understand that many of these questions ultimately require hands-on
> work and investigation on my part. However, I believe now is an
> opportune time to engage the community in these discussions.
>
> I will be attending the Linux Plumbers Conference next month on behalf
> of Larry and myself and would like to use this as a start to further
> explore potential collaborations and conversations around these topics.
Maybe Luis will be there? I won't.
> Lastly, I want to thank the backports.git maintainers for providing an
> invaluable tool for the Linux community. Also, thank you Luis for
> getting me started on having these conversations.
:)
johannes
--
To unsubscribe from this list: send the line "unsubscribe backports" in
