On Wed, Jun 14, 2023 at 2:48 PM Alexei Starovoitov <alexei.starovoitov@xxxxxxxxx> wrote: > > On Wed, Jun 14, 2023 at 1:57 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote: > > > > On Wed, Jun 14, 2023 at 11:23:52AM -0700, Robert Kolchmeyer wrote: > > > Hi all, > > > > > > I believe 5.10.168 and 5.15.93 introduced a regression that impacts > > > the Cilium project. Some information on the nature of the regression > > > is available at https://github.com/cilium/cilium/issues/25500. The > > > primary symptom seems to be the error `BPF program is too large.` > > > > > > My colleague has found that reverting the following two commits: > > > > > > 8de8c4a "bpf: Support <8-byte scalar spill and refill" > > > 9ff2beb "bpf: Fix incorrect state pruning for <8B spill/fill" > > > > > > resolves the regression. > > > > > > If we revert these in the stable tree, there may be a few changes that > > > depend on those that also need to be reverted, but I'm not sure yet. > > > > > > Would it make sense to revert these changes (and any dependent ones) > > > in the 5.10 and 5.15 trees? If anyone has other ideas, I can help test > > > possible solutions. > > > > Can you actually test if those reverts work properly for you and if > > there are other dependencies involved? > > > > And is this issue also in 6.1.y and Linus's tree? If not, why not, are > > we just missing a commit? We can't revert something from a stable > > release if you are going to hit the same issue when moving to a new > > release, right? > > > > thanks, > > > > greg k-h > > Before jumping to reverts.. > how is it fixed in 6.0+ kernels? > > "BPF program is too large" can probably be worked around on the cilium side. > The kernel cannot guarantee that a particular program will > always be verifiable. We find safety bugs in the verifier and often > enough the fixes to such issues make the verifier work harder to prove > the safety of the program. > This is one of such cases. These two commits are necessary. > Reverting them will prevent loading of valid programs. > So reverts is a dangerous path. > The best is to identify the other patches from 6.0+ and backport them. > The second best path is to bump 1M limit to something higher to > mitigate "more work by the verifier". Thanks all for the pointers, this is super helpful. I'm working on getting more details, and I'll post here when I have them. -Robert
