On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote:
On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote:
>
> I'm shocked that these are the statistics you use to claim the current AUTOSEL
> process is working. I think they actually show quite the opposite!
>
> First, since many AUTOSEL commits aren't actually fixes but nearly all
> stable-tagged commits *are* fixes, the rate of regressions per commit would need
> to be lower for AUTOSEL commits than for stable-tagged commits in order for
> AUTOSEL commits to have the same rate of regressions *per fix*. Your numbers
> suggest a similar regression rate *per commit*. Thus, AUTOSEL probably
> introduces more regressions *per fix* than stable-tagged commits.
Interesting claim. How many of the AUTOSEL commits are "actual" fixes?
How do you know if a commit is a fix for anything or not?
Could you try and back claims with some evidence?
Yes, in a perfect world where we know if a commit is a fix we could
avoid introducing regressions into the stable trees. Heck, maybe we could
even stop writing buggy code to begin with?
Are you seriously trying to claim that a random commit your neural network
picked up is just as likely to be a fix as a commit that the author explicitly
tagged as a fix and/or for stable?
I'd like to think that this is the case after the initial selection and
multiple rounds of reviews, yes.
That's quite an extraordinary claim, and it's not true from my experience. Lots
of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to
understand fairly well whether the patch is a "fix", are not actually fixes. Or
are very borderline "fixes" that don't meet stable criteria. (Note, I generally
only bother responding to AUTOSEL if I think a patch is actually going to cause
a problem. So a lack of response isn't necessarily agreement that a patch is
really suitable for stable...)
Oh sorry, personal experience is not "evidence". Please disregard my invalid
non-evidence-based opinion.
> (Of course, stable-tagged commits sometimes have missing prerequisite bugs too.
> But it's expected to be at a lower rate, since the original developers and
> maintainers are directly involved in adding the stable tags. These are the
> people who are more familiar than anyone else with prerequisites.)
You'd be surprised. There is documentation around how one would annotate
dependencies for stable tagged commits, something along the lines of:
cc: stable@xxxxxxxxxx # dep1 dep2
Grep through the git log and see how often this is actually used.
Well, probably more common is that prerequisites are in the same patchset, and
the prerequisites are tagged for stable too. Whereas AUTOSEL often just picks
patch X of N. Also, developers and maintainers who tag patches for stable are
probably more likely to help with the stable process in general and make sure
patches are backported correctly...
Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately
cherry-picking patch X of N so often.
That's a fair point.
> a multi-patch series, and if so are earlier patches needed as prerequisites".
> There also needs to be more soak time in mainline, and more review time.
Tricky bit with mainline/review time is that very few of our users
actually run -rc trees.
We end up hitting many of the regressions because the commits actually
end up in stable trees. Should it work that way? No, but our testing
story around -rc releases is quite lacking.
Well, in the bug that affected me, it *was* found on mainline almost
immediately. It just took a bit longer than the extremely aggressive 7-day
AUTOSEL period to be fixed.
Oh sorry again, one example is not "evidence". Please disregard my invalid
non-evidence-based opinion.
I'm happy that we're in agreement that significant process changes can't
happen because of opinions or anecdotal examples.
In all seriousness, I will work on addressing the issues that happened
around the commit(s) you've pointed out and improve our existing
process.
--
Thanks,
Sasha
