On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@xxxxxx> wrote: > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@xxxxxx> wrote: > >> > >> > >> > >> On 7/25/21 8:22 AM, Vincent Li wrote: > >>> > >>> > >>> > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > >>> > >>>> > >>>> > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > >>>> > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > >>>>>> > >>>>>> > >>>>>> Hi BPF experts, > >>>>>> > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > >>>>>> > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > >>>>>> (7)!" subsys=datapath-loader > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > >>>>>> subsys=datapath-loader > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>>>> > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > >>>>> > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > >>>>> no clue which commits in 5.8-rc1 resolved the issue > >>>>> > >>>>>> > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > >>>>>> bytes (9 tries)!" subsys=datapath-loader > >> > >> The error message is "Log buffer too small to dump verifier log 16777215 > >> bytes (9 tries)!". > >> > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > >> much shorter. So it fixed the above log buffer too small issue. > >> > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > the problem went away for kernel 5.x- 5.8, but the > > "Prog section '2/7' rejected: Argument list too long.." issue > > persisted even after I remove the "verbose" logging > > for kernel version 4.19, any clue on that? > > No, I don't. > > You need to have detailed verifier log. In verifier, there are quite > some places which returns -E2BIG. > I will do another round of bisect, correct myself, the "The argument list too long" occurred in 5.1, but not in 5.2 > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>>>> > >>>>>> 5.8 works fine. > >>>>>> > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > >>>>>> would be appreciated! > >>>> > >>>> I have git bisected to this commit: > [...]
