On Wed, Jul 26, 2023 at 2:00 PM Ian Kumlien <ian.kumlien@xxxxxxxxx> wrote: > > On Wed, Jul 26, 2023 at 10:03 AM Pablo Neira Ayuso <pablo@xxxxxxxxxxxxx> wrote: > > > > Hi, > > > > On Mon, Jul 24, 2023 at 02:24:15PM -0700, Jakub Kicinski wrote: > > > Adding netfilter to CC. > > > > > > On Sun, 23 Jul 2023 16:44:50 +0200 Ian Kumlien wrote: > > > > Running vanilla 6.4.4 with cherry picked: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.4.5&id=7a59f29961cf97b98b02acaadf5a0b1f8dde938c > > > > > > [...] > > > > [108431.305700] RSP: 0018:ffffac250ade7e28 EFLAGS: 00010206 > > > > [108431.311107] RAX: 0000000000000081 RBX: ffff9ebc413b42f8 RCX: > > > > 0000000000000001 > > > > [108431.318420] RDX: 00000001067200c0 RSI: ffff9ebeda71ce58 RDI: > > > > ffff9ebeda71ce58 > > > > [108431.325735] RBP: ffff9ebc413b4250 R08: ffff9ebc413b4250 R09: > > > > ffff9ebe3d7fad58 > > > > [108431.333068] R10: 0000000000000000 R11: 0000000000000003 R12: > > > > ffff9ebfafab0000 > > > > [108431.340415] R13: 0000000000000000 R14: ffff9ebfafab0005 R15: > > > > ffff9ebd79a0f780 > > > > [108431.347764] FS: 0000000000000000(0000) GS:ffff9ebfafa80000(0000) > > > > knlGS:0000000000000000 > > > > [108431.356069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [108431.362012] CR2: 0000000000000081 CR3: 000000045e99e000 CR4: > > > > 00000000003526e0 > > > > [108431.369361] Call Trace: > > > > [108431.371999] <TASK> > > > > [108431.374296] ? __die (arch/x86/kernel/dumpstack.c:421 > > > > arch/x86/kernel/dumpstack.c:434) > > > > [108431.377553] ? page_fault_oops (arch/x86/mm/fault.c:707) > > > > [108431.381850] ? load_balance (kernel/sched/fair.c:10926) > > > > [108431.385884] ? exc_page_fault (arch/x86/mm/fault.c:1279 > > > > arch/x86/mm/fault.c:1486 arch/x86/mm/fault.c:1542) > > > > [108431.390094] ? asm_exc_page_fault (./arch/x86/include/asm/idtentry.h:570) > > > > [108431.394482] ? flow_offload_teardown > > > > (./arch/x86/include/asm/bitops.h:75 > > > > ./include/asm-generic/bitops/instrumented-atomic.h:42 > > > > net/netfilter/nf_flow_table_core.c:362) > > > > [108431.399036] nf_flow_offload_gc_step > > > > (./arch/x86/include/asm/bitops.h:207 > > > > ./arch/x86/include/asm/bitops.h:239 > > > > ./include/asm-generic/bitops/instrumented-non-atomic.h:142 > > > > net/netfilter/nf_flow_table_core.c:436) > > > > This crash points here. > > > > static void nf_flow_offload_gc_step(struct nf_flowtable *flow_table, > > struct flow_offload *flow, void *data) > > { > > if (nf_flow_has_expired(flow) || > > nf_ct_is_dying(flow->ct) || > > nf_flow_is_outdated(flow)) > > flow_offload_teardown(flow); > > > > if (test_bit(NF_FLOW_TEARDOWN, &flow->flags)) { <-- > > > > Is this always reproducible on your testbed? > > That's a bit unknown, I don't quite know what triggers it... I only > know it's happened twice :/ > (That i've noticed - the fw runs with a watchdog and it's always been > a "uhuh... uptime is less than expected" kind of thing) I should add that i do: for interface in eno1 eno2 eno3 eno4 ; do for offload in ntuple hw-tc-offload rx-udp-gro-forwarding rx-gro-list ; do ethtool -K $interface $offload on > /dev/null done done And that some interfaces are directly attached to a bridge while others are more normal lspci |grep Ethernet 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11) 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11) 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11) 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 1GbE (rev 11) This is since i added NET_SCHED etc support back in to the kernel tc qdisc show qdisc noqueue 0: dev lo root refcnt 2 qdisc mq 0: dev eno1 root qdisc fq 0: dev eno1 parent :c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :b limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :a limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :9 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno1 parent :1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc mq 0: dev eno2 root qdisc fq 0: dev eno2 parent :c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :b limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :a limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :9 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno2 parent :1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 18028b initial_quantum 90140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc mq 0: dev eno3 root qdisc fq 0: dev eno3 parent :c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :b limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :a limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :9 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno3 parent :1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc mq 0: dev eno4 root qdisc fq 0: dev eno4 parent :c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :b limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :a limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :9 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc fq 0: dev eno4 parent :1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop qdisc noqueue 0: dev external-net root refcnt 2 qdisc noqueue 0: dev local-lan root refcnt 2 qdisc noqueue 0: dev virtual-net root refcnt 2 qdisc noqueue 8001: dev vnet0 root refcnt 2 qdisc noqueue 8002: dev vnet1 root refcnt 2 qdisc noqueue 8003: dev vnet2 root refcnt 2 qdisc noqueue 8004: dev vnet3 root refcnt 2 qdisc noqueue 8005: dev vnet4 root refcnt 2 qdisc noqueue 8006: dev vnet5 root refcnt 2 qdisc noqueue 8007: dev vnet6 root refcnt 2 qdisc noqueue 8008: dev vnet7 root refcnt 2 qdisc noqueue 0: dev int root refcnt 2 > > Thanks.
