On Mon, Jul 26, 2021 at 10:35 AM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > On Sun, Jul 25, 2021 at 8:49 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > > > On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@xxxxxx> wrote: > > > > > > > > > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@xxxxxx> wrote: > > > >> > > > >> > > > >> > > > >> On 7/25/21 8:22 AM, Vincent Li wrote: > > > >>> > > > >>> > > > >>> > > > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > > > >>>> > > > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> Hi BPF experts, > > > >>>>>> > > > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > > > >>>>>> > > > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > > > >>>>>> (7)!" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > > > >>>>>> subsys=datapath-loader > > > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > > >>>>>> > > > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > > > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > > > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > > > >>>>> > > > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > > > >>>>> no clue which commits in 5.8-rc1 resolved the issue > > > >>>>> > > > >>>>>> > > > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > > > >>>>>> bytes (9 tries)!" subsys=datapath-loader > > > >> > > > >> The error message is "Log buffer too small to dump verifier log 16777215 > > > >> bytes (9 tries)!". > > > >> > > > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > > > >> much shorter. So it fixed the above log buffer too small issue. > > > >> > > > > > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > > > the problem went away for kernel 5.x- 5.8, but the > > > > "Prog section '2/7' rejected: Argument list too long.." issue > > > > persisted even after I remove the "verbose" logging > > > > for kernel version 4.19, any clue on that? > > > > > > No, I don't. > > > > > > You need to have detailed verifier log. In verifier, there are quite > > > some places which returns -E2BIG. > > > > > I will do another round of bisect, correct myself, the "The argument > > list too long" occurred in 5.1, but not in 5.2 > > It looks to be this commit fixed the issue > commit c04c0d2b968ac45d6ef020316808ef6c82325a82 (HEAD) > Author: Alexei Starovoitov <ast@xxxxxxxxxx> > Date: Mon Apr 1 21:27:45 2019 -0700 > bpf: increase complexity limit and maximum program size > > > > > > > > > > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > > >>>>>> > > > >>>>>> 5.8 works fine. > > > >>>>>> > > > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > > > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > > > >>>>>> would be appreciated! > > > >>>> > > > >>>> I have git bisected to this commit: > > > [...] Finally bring an end to my long time mystery issue of using eth_store_daddr() result in "invalid mem access", I need to initialize the mac address variable with 0, uninitialized variable also cause "invalid read from stack off" in kernel 4.9, but not version above 4.9. https://github.com/cilium/cilium/pull/17370#issuecomment-922396415
