On Sun, Jul 25, 2021 at 8:49 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@xxxxxx> wrote: > > > > > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@xxxxxx> wrote: > > >> > > >> > > >> > > >> On 7/25/21 8:22 AM, Vincent Li wrote: > > >>> > > >>> > > >>> > > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > > >>> > > >>>> > > >>>> > > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > > >>>> > > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > >>>>>> > > >>>>>> > > >>>>>> Hi BPF experts, > > >>>>>> > > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > > >>>>>> > > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > > >>>>>> (7)!" subsys=datapath-loader > > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > > >>>>>> subsys=datapath-loader > > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > >>>>>> > > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > > >>>>> > > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > > >>>>> no clue which commits in 5.8-rc1 resolved the issue > > >>>>> > > >>>>>> > > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > > >>>>>> bytes (9 tries)!" subsys=datapath-loader > > >> > > >> The error message is "Log buffer too small to dump verifier log 16777215 > > >> bytes (9 tries)!". > > >> > > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > > >> much shorter. So it fixed the above log buffer too small issue. > > >> > > > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > > the problem went away for kernel 5.x- 5.8, but the > > > "Prog section '2/7' rejected: Argument list too long.." issue > > > persisted even after I remove the "verbose" logging > > > for kernel version 4.19, any clue on that? > > > > No, I don't. > > > > You need to have detailed verifier log. In verifier, there are quite > > some places which returns -E2BIG. > > > I will do another round of bisect, correct myself, the "The argument > list too long" occurred in 5.1, but not in 5.2 It looks to be this commit fixed the issue commit c04c0d2b968ac45d6ef020316808ef6c82325a82 (HEAD) Author: Alexei Starovoitov <ast@xxxxxxxxxx> Date: Mon Apr 1 21:27:45 2019 -0700 bpf: increase complexity limit and maximum program size > > > > > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > >>>>>> > > >>>>>> 5.8 works fine. > > >>>>>> > > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > > >>>>>> would be appreciated! > > >>>> > > >>>> I have git bisected to this commit: > > [...]
