Good day.
Had an issue with FLOW OFFLOAD on nftables based NAT server. NAT
crashes after a few hours of running with the error:
[Пт мар 5 11:30:12 2021] rcu: INFO: rcu_sched self-detected stall on CPU
[Пт мар 5 11:30:12 2021] rcu: 10-....: (5251 ticks this GP)
idle=73e/1/0x4000000000000002 softirq=4099632/4099633 fqs=2625
[Пт мар 5 11:30:12 2021] rcu: (t=5251 jiffies g=14930289 q=91701)
[Пт мар 5 11:30:12 2021] NMI backtrace for cpu 10
[Пт мар 5 11:30:12 2021] CPU: 10 PID: 64 Comm: ksoftirqd/10 Tainted:
G OE 4.19.0-14-amd64 #1 Debian 4.19.171-2
What could be the problem? What can you check?
Kernel: Linux nat8 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2
(2021-01-30) x86_64 GNU/Linux
Version nftables:
nft -v nftables v0.9.0 (Fearless Fosdick)
Server: CPU Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Memory 64Gb
Network card: Intel 82599ES (520 chipset), driver ixgbe 5.10.2
Configuration nftables (nft list ruleset):
table ip nat {
chain postrouting {
type nat hook postrouting priority 100; policy accept;
counter packets 0 bytes 0
oif "ens1f1" ip saddr 10.0.0.0/8 snat to 31.43.223.160-31.43.223.176 persistent
oif "ens1f1" ip saddr 192.168.0.0/16 snat to
31.43.223.160-31.43.223.176 persistent
oif "ens1f1" ip saddr 172.16.0.0/12 snat to
31.43.223.160-31.43.223.176 persistent
counter packets 0 bytes 0
}
}
table inet filter {
flowtable fastnat {
hook ingress priority 0
devices = { ens1f0, ens1f1 }
}
chain forward {
type filter hook forward priority 0; policy accept;
ip protocol { tcp, udp } flow offload @fastnat counter packets 3 bytes 323
counter packets 3 bytes 323
}
}
table ip raw {
ct helper pptp-gre {
type "pptp" protocol tcp
l3proto ip
}
chain prerouting {
type filter hook prerouting priority -300; policy accept;
tcp dport 1723 ct helper set "pptp-gre"
counter packets 84 bytes 5147
}
}
More detailed error log:
[Пт мар 5 11:30:12 2021] rcu: INFO: rcu_sched self-detected stall on CPU
[Пт мар 5 11:30:12 2021] rcu: 10-....: (5251 ticks this GP)
idle=73e/1/0x4000000000000002 softirq=4099632/4099633 fqs=2625
[Пт мар 5 11:30:12 2021] rcu: (t=5251 jiffies g=14930289 q=91701)
[Пт мар 5 11:30:12 2021] NMI backtrace for cpu 10
[Пт мар 5 11:30:12 2021] CPU: 10 PID: 64 Comm: ksoftirqd/10 Tainted:
G OE 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:30:12 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:30:12 2021] Call Trace:
[Пт мар 5 11:30:12 2021] <IRQ>
[Пт мар 5 11:30:12 2021] dump_stack+0x66/0x81
[Пт мар 5 11:30:12 2021] nmi_cpu_backtrace.cold.4+0x13/0x50
[Пт мар 5 11:30:12 2021] ? lapic_can_unplug_cpu+0x80/0x80
[Пт мар 5 11:30:12 2021] nmi_trigger_cpumask_backtrace+0xf9/0x100
[Пт мар 5 11:30:12 2021] rcu_dump_cpu_stacks+0x9b/0xcb
[Пт мар 5 11:30:12 2021] rcu_check_callbacks.cold.81+0x1db/0x335
[Пт мар 5 11:30:12 2021] ? tick_sched_do_timer+0x60/0x60
[Пт мар 5 11:30:12 2021] update_process_times+0x28/0x60
[Пт мар 5 11:30:12 2021] tick_sched_handle+0x22/0x60
[Пт мар 5 11:30:12 2021] tick_sched_timer+0x37/0x70
[Пт мар 5 11:30:12 2021] __hrtimer_run_queues+0x100/0x280
[Пт мар 5 11:30:12 2021] hrtimer_interrupt+0x100/0x220
[Пт мар 5 11:30:12 2021] ? handle_irq_event+0x47/0x60
[Пт мар 5 11:30:12 2021] smp_apic_timer_interrupt+0x6a/0x140
[Пт мар 5 11:30:12 2021] apic_timer_interrupt+0xf/0x20
[Пт мар 5 11:30:12 2021] </IRQ>
[Пт мар 5 11:30:12 2021] RIP:
0010:nf_conntrack_tuple_taken+0xad/0x240 [nf_conntrack]
[Пт мар 5 11:30:12 2021] Code: 48 8d 04 f5 00 00 00 00 48 89 f1 48 29
f0 48 c1 e0 03 48 29 c7 48 01 df 4c 39 ef 74 1b 48 8b 05 a9 16 bb d5
8b b7 88 00 00 00 <29> c6 85 f6 7e 2f 8b 43 10 41 39 06 74 36 48 8b 1b
48 89 d8 f6 c3
[Пт мар 5 11:30:12 2021] RSP: 0018:ffffbd004652f680 EFLAGS: 00000202
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:30:12 2021] RAX: 000000010164f9f0 RBX: ffff9422a30af410
RCX: 0000000000000000
[Пт мар 5 11:30:12 2021] RDX: 000000000000b89a RSI: 00000000029915c5
RDI: ffff9422a30af400
[Пт мар 5 11:30:12 2021] RBP: ffffffff962deb80 R08: ffffbd004652f6d8
R09: 000000008029bca6
[Пт мар 5 11:30:12 2021] R10: ffff942252148dc0 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:30:12 2021] R13: ffff942251345cc0 R14: ffffbd004652f6c0
R15: fffffffffffffff0
[Пт мар 5 11:30:12 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:30:12 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:30:12 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:30:12 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:30:12 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:30:12 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:30:12 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:30:12 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:30:12 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:30:12 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:30:12 2021] ? nf_ct_ext_add+0x90/0x160 [nf_conntrack]
[Пт мар 5 11:30:12 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:30:12 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:30:12 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:30:12 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:30:12 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:30:12 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:30:12 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:30:12 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:30:12 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:30:12 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:30:12 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:30:12 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:30:12 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:30:12 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:30:12 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:30:12 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:30:12 2021] ? sort_range+0x20/0x20
[Пт мар 5 11:30:12 2021] run_ksoftirqd+0x26/0x40
[Пт мар 5 11:30:12 2021] smpboot_thread_fn+0xc5/0x160
[Пт мар 5 11:30:12 2021] kthread+0x112/0x130
[Пт мар 5 11:30:12 2021] ? kthread_bind+0x30/0x30
[Пт мар 5 11:30:12 2021] ret_from_fork+0x35/0x40
[Пт мар 5 11:33:18 2021] rcu: INFO: rcu_sched self-detected stall on CPU
[Пт мар 5 11:33:18 2021] rcu: 10-....: (1 GPs behind)
idle=6ba/1/0x4000000000000002 softirq=4101971/4101972 fqs=2624
[Пт мар 5 11:33:18 2021] rcu: (t=5251 jiffies g=14930473 q=89249)
[Пт мар 5 11:33:18 2021] NMI backtrace for cpu 10
[Пт мар 5 11:33:18 2021] CPU: 10 PID: 64 Comm: ksoftirqd/10 Tainted:
G OE 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:33:18 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:33:18 2021] Call Trace:
[Пт мар 5 11:33:18 2021] <IRQ>
[Пт мар 5 11:33:18 2021] dump_stack+0x66/0x81
[Пт мар 5 11:33:18 2021] nmi_cpu_backtrace.cold.4+0x13/0x50
[Пт мар 5 11:33:18 2021] ? lapic_can_unplug_cpu+0x80/0x80
[Пт мар 5 11:33:18 2021] nmi_trigger_cpumask_backtrace+0xf9/0x100
[Пт мар 5 11:33:18 2021] rcu_dump_cpu_stacks+0x9b/0xcb
[Пт мар 5 11:33:18 2021] rcu_check_callbacks.cold.81+0x1db/0x335
[Пт мар 5 11:33:18 2021] ? tick_sched_do_timer+0x60/0x60
[Пт мар 5 11:33:18 2021] update_process_times+0x28/0x60
[Пт мар 5 11:33:18 2021] tick_sched_handle+0x22/0x60
[Пт мар 5 11:33:18 2021] tick_sched_timer+0x37/0x70
[Пт мар 5 11:33:18 2021] __hrtimer_run_queues+0x100/0x280
[Пт мар 5 11:33:18 2021] hrtimer_interrupt+0x100/0x220
[Пт мар 5 11:33:18 2021] ? handle_irq_event+0x47/0x60
[Пт мар 5 11:33:18 2021] smp_apic_timer_interrupt+0x6a/0x140
[Пт мар 5 11:33:18 2021] apic_timer_interrupt+0xf/0x20
[Пт мар 5 11:33:18 2021] </IRQ>
[Пт мар 5 11:33:18 2021] RIP:
0010:nf_conntrack_tuple_taken+0x80/0x240 [nf_conntrack]
[Пт мар 5 11:33:18 2021] Code: 89 0c 24 e8 f2 f3 ff ff 89 da 89 c3 48
0f af d3 48 c1 ea 20 48 8b 0c 24 48 8d 04 d1 48 8b 00 48 89 c3 a8 01
75 4a 0f b6 73 37 <4c> 89 ff 48 8d 04 f5 00 00 00 00 48 89 f1 48 29 f0
48 c1 e0 03 48
[Пт мар 5 11:33:18 2021] RSP: 0018:ffffbd004652f680 EFLAGS: 00000246
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:33:18 2021] RAX: ffff94227d8b03d0 RBX: ffff94227d8b03d0
RCX: 0000000000000001
[Пт мар 5 11:33:18 2021] RDX: 0000000000001916 RSI: 0000000000000000
RDI: ffff94227ae76780
[Пт мар 5 11:33:18 2021] RBP: ffffffff962deb80 R08: ffffbd004652f6d8
R09: 000000002531dd40
[Пт мар 5 11:33:18 2021] R10: ffff94224f1a17c0 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:33:18 2021] R13: ffff942a748d0140 R14: ffffbd004652f6c0
R15: fffffffffffffff0
[Пт мар 5 11:33:18 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:33:18 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:33:18 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:33:18 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:33:18 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:33:18 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:33:18 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:33:18 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:33:18 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:33:18 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:33:18 2021] ? nf_nat_ipv4_csum_recalc+0xb0/0xb0 [nf_nat_ipv4]
[Пт мар 5 11:33:18 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:33:18 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:33:18 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:33:18 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:33:18 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:33:18 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:33:18 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:33:18 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:33:18 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:33:18 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:33:18 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:33:18 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:33:18 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:33:18 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:33:18 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:33:18 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:33:18 2021] ? sort_range+0x20/0x20
[Пт мар 5 11:33:18 2021] run_ksoftirqd+0x26/0x40
[Пт мар 5 11:33:18 2021] smpboot_thread_fn+0xc5/0x160
[Пт мар 5 11:33:18 2021] kthread+0x112/0x130
[Пт мар 5 11:33:18 2021] ? kthread_bind+0x30/0x30
[Пт мар 5 11:33:18 2021] ret_from_fork+0x35/0x40
[Пт мар 5 11:33:46 2021] rcu: INFO: rcu_sched self-detected stall on CPU
[Пт мар 5 11:33:46 2021] rcu: 10-....: (5250 ticks this GP)
idle=6c2/1/0x4000000000000002 softirq=4101976/4101977 fqs=2301
[Пт мар 5 11:33:46 2021] rcu: (t=5251 jiffies g=14930485 q=55206)
[Пт мар 5 11:33:46 2021] NMI backtrace for cpu 10
[Пт мар 5 11:33:46 2021] CPU: 10 PID: 64 Comm: ksoftirqd/10 Tainted:
G OE 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:33:46 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:33:46 2021] Call Trace:
[Пт мар 5 11:33:46 2021] <IRQ>
[Пт мар 5 11:33:46 2021] dump_stack+0x66/0x81
[Пт мар 5 11:33:46 2021] nmi_cpu_backtrace.cold.4+0x13/0x50
[Пт мар 5 11:33:46 2021] ? lapic_can_unplug_cpu+0x80/0x80
[Пт мар 5 11:33:46 2021] nmi_trigger_cpumask_backtrace+0xf9/0x100
[Пт мар 5 11:33:46 2021] rcu_dump_cpu_stacks+0x9b/0xcb
[Пт мар 5 11:33:46 2021] rcu_check_callbacks.cold.81+0x1db/0x335
[Пт мар 5 11:33:46 2021] ? tick_sched_do_timer+0x60/0x60
[Пт мар 5 11:33:46 2021] update_process_times+0x28/0x60
[Пт мар 5 11:33:46 2021] tick_sched_handle+0x22/0x60
[Пт мар 5 11:33:46 2021] tick_sched_timer+0x37/0x70
[Пт мар 5 11:33:46 2021] __hrtimer_run_queues+0x100/0x280
[Пт мар 5 11:33:46 2021] hrtimer_interrupt+0x100/0x220
[Пт мар 5 11:33:46 2021] ? handle_irq_event+0x47/0x60
[Пт мар 5 11:33:46 2021] smp_apic_timer_interrupt+0x6a/0x140
[Пт мар 5 11:33:46 2021] apic_timer_interrupt+0xf/0x20
[Пт мар 5 11:33:46 2021] </IRQ>
[Пт мар 5 11:33:46 2021] RIP:
0010:nf_conntrack_tuple_taken+0xad/0x240 [nf_conntrack]
[Пт мар 5 11:33:46 2021] Code: 48 8d 04 f5 00 00 00 00 48 89 f1 48 29
f0 48 c1 e0 03 48 29 c7 48 01 df 4c 39 ef 74 1b 48 8b 05 a9 16 bb d5
8b b7 88 00 00 00 <29> c6 85 f6 7e 2f 8b 43 10 41 39 06 74 36 48 8b 1b
48 89 d8 f6 c3
[Пт мар 5 11:33:46 2021] RSP: 0018:ffffbd004652f680 EFLAGS: 00000206
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:33:46 2021] RAX: 000000010165cae1 RBX: ffff942a8887be10
RCX: 0000000000000000
[Пт мар 5 11:33:46 2021] RDX: 0000000000003061 RSI: 0000000002aecad6
RDI: ffff942a8887be00
[Пт мар 5 11:33:46 2021] RBP: ffffffff962deb80 R08: ffffbd004652f6d8
R09: 0000000061909bca
[Пт мар 5 11:33:46 2021] R10: ffff942a7548eb40 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:33:46 2021] R13: ffff942a74eb5400 R14: ffffbd004652f6c0
R15: fffffffffffffff0
[Пт мар 5 11:33:46 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:33:46 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:33:46 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:33:46 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:33:46 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:33:46 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:33:46 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:33:46 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:33:46 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:33:46 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:33:46 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:33:46 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:33:46 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:33:46 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:33:46 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:33:46 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:33:46 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:33:46 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:33:46 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:33:46 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:33:46 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:33:46 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:33:46 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:33:46 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:33:46 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:33:46 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:33:46 2021] ? sort_range+0x20/0x20
[Пт мар 5 11:33:46 2021] run_ksoftirqd+0x26/0x40
[Пт мар 5 11:33:46 2021] smpboot_thread_fn+0xc5/0x160
[Пт мар 5 11:33:46 2021] kthread+0x112/0x130
[Пт мар 5 11:33:46 2021] ? kthread_bind+0x30/0x30
[Пт мар 5 11:33:46 2021] ret_from_fork+0x35/0x40
[Пт мар 5 11:35:29 2021] watchdog: BUG: soft lockup - CPU#6 stuck for
22s! [swapper/6:0]
[Пт мар 5 11:35:29 2021] Modules linked in: nft_objref nft_ct
nf_conntrack_pptp nf_conntrack_proto_gre ipmi_ssif intel_rapl
nft_flow_offload nf_flow_table_inet nf_flow_table sb_edac
nf_tables_set x86_pkg_temp_thermal intel_powerclamp coretemp mgag200
crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel intel_cstate
drm_kms_helper intel_uncore pcc_cpufreq intel_rapl_perf nft_nat
serio_raw pcspkr drm evdev joydev sg i2c_algo_bit iTCO_wdt hpilo hpwdt
iTCO_vendor_support ioatdma wmi ipmi_si nft_counter ipmi_devintf
ipmi_msghandler acpi_tad acpi_power_meter button nft_chain_nat_ipv4
nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2
fscrypto ecb raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor hid_generic
[Пт мар 5 11:35:29 2021] usbhid hid raid6_pq libcrc32c
crc32c_generic raid0 multipath linear raid1 md_mod sd_mod crc32c_intel
ahci libahci libata aesni_intel xhci_pci aes_x86_64 uhci_hcd
crypto_simd ehci_pci xhci_hcd psmouse cryptd ehci_hcd glue_helper
scsi_mod ixgbe(OE) tg3 usbcore lpc_ich i2c_i801 mfd_core libphy dca
usb_common
[Пт мар 5 11:35:29 2021] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G
OE 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:35:29 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:35:29 2021] RIP:
0010:nf_conntrack_tuple_taken+0x80/0x240 [nf_conntrack]
[Пт мар 5 11:35:29 2021] Code: 89 0c 24 e8 f2 f3 ff ff 89 da 89 c3 48
0f af d3 48 c1 ea 20 48 8b 0c 24 48 8d 04 d1 48 8b 00 48 89 c3 a8 01
75 4a 0f b6 73 37 <4c> 89 ff 48 8d 04 f5 00 00 00 00 48 89 f1 48 29 f0
48 c1 e0 03 48
[Пт мар 5 11:35:29 2021] RSP: 0018:ffff942b1f803760 EFLAGS: 00000246
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:35:29 2021] RAX: ffff94226f4f57d0 RBX: ffff94226f4f57d0
RCX: 0000000000000001
[Пт мар 5 11:35:29 2021] RDX: 0000000000001f5f RSI: 0000000000000000
RDI: ffff942a891b0f00
[Пт мар 5 11:35:29 2021] RBP: ffffffff962deb80 R08: ffff942b1f8037b8
R09: 00000000ca74ee0f
[Пт мар 5 11:35:29 2021] R10: ffff94224e4e8780 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:35:29 2021] R13: ffff942a74346a00 R14: ffff942b1f8037a0
R15: fffffffffffffff0
[Пт мар 5 11:35:29 2021] FS: 0000000000000000(0000)
GS:ffff942b1f800000(0000) knlGS:0000000000000000
[Пт мар 5 11:35:29 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Пт мар 5 11:35:29 2021] CR2: 00007fff73454038 CR3: 0000000b7c80a003
CR4: 00000000001606e0
[Пт мар 5 11:35:29 2021] Call Trace:
[Пт мар 5 11:35:29 2021] <IRQ>
[Пт мар 5 11:35:29 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:35:29 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:35:29 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:35:29 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:35:29 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:35:29 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:35:29 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:35:29 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:35:29 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:35:29 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:35:29 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:35:29 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:35:29 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:35:29 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:35:29 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:35:29 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:35:29 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:35:29 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:35:29 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:35:29 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:35:29 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:35:29 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:35:29 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:35:29 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:35:29 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:35:29 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:35:29 2021] irq_exit+0xba/0xc0
[Пт мар 5 11:35:29 2021] do_IRQ+0x7f/0xe0
[Пт мар 5 11:35:29 2021] common_interrupt+0xf/0xf
[Пт мар 5 11:35:29 2021] </IRQ>
[Пт мар 5 11:35:29 2021] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[Пт мар 5 11:35:29 2021] Code: e8 7c 85 b2 ff 80 7c 24 0b 00 74 17 9c
58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 1e 0d b8 ff fb
66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff
ff 7f 48 39 c3
[Пт мар 5 11:35:29 2021] RSP: 0018:ffffbd0046317e90 EFLAGS: 00000246
ORIG_RAX: ffffffffffffffdc
[Пт мар 5 11:35:29 2021] RAX: ffff942b1f822140 RBX: 000055a73f206ba4
RCX: 000000000000001f
[Пт мар 5 11:35:29 2021] RDX: 000055a73f206ba4 RSI: 000000003342629e
RDI: 0000000000000000
[Пт мар 5 11:35:29 2021] RBP: ffff942b1f82a748 R08: 0000000000000002
R09: 0000000000021a00
[Пт мар 5 11:35:29 2021] R10: 0000d8c97090db52 R11: ffff942b1f821128
R12: 0000000000000004
[Пт мар 5 11:35:29 2021] R13: ffffffff962b71d8 R14: 0000000000000004
R15: 0000000000000000
[Пт мар 5 11:35:29 2021] do_idle+0x228/0x270
[Пт мар 5 11:35:29 2021] cpu_startup_entry+0x6f/0x80
[Пт мар 5 11:35:29 2021] start_secondary+0x1a4/0x200
[Пт мар 5 11:35:29 2021] secondary_startup_64+0xa4/0xb0
[Пт мар 5 11:36:01 2021] watchdog: BUG: soft lockup - CPU#6 stuck for
21s! [ksoftirqd/6:43]
[Пт мар 5 11:36:01 2021] Modules linked in: nft_objref nft_ct
nf_conntrack_pptp nf_conntrack_proto_gre ipmi_ssif intel_rapl
nft_flow_offload nf_flow_table_inet nf_flow_table sb_edac
nf_tables_set x86_pkg_temp_thermal intel_powerclamp coretemp mgag200
crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel intel_cstate
drm_kms_helper intel_uncore pcc_cpufreq intel_rapl_perf nft_nat
serio_raw pcspkr drm evdev joydev sg i2c_algo_bit iTCO_wdt hpilo hpwdt
iTCO_vendor_support ioatdma wmi ipmi_si nft_counter ipmi_devintf
ipmi_msghandler acpi_tad acpi_power_meter button nft_chain_nat_ipv4
nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2
fscrypto ecb raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor hid_generic
[Пт мар 5 11:36:01 2021] usbhid hid raid6_pq libcrc32c
crc32c_generic raid0 multipath linear raid1 md_mod sd_mod crc32c_intel
ahci libahci libata aesni_intel xhci_pci aes_x86_64 uhci_hcd
crypto_simd ehci_pci xhci_hcd psmouse cryptd ehci_hcd glue_helper
scsi_mod ixgbe(OE) tg3 usbcore lpc_ich i2c_i801 mfd_core libphy dca
usb_common
[Пт мар 5 11:36:01 2021] CPU: 6 PID: 43 Comm: ksoftirqd/6 Tainted: G
OEL 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:36:01 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:36:01 2021] RIP:
0010:nf_conntrack_tuple_taken+0xad/0x240 [nf_conntrack]
[Пт мар 5 11:36:01 2021] Code: 48 8d 04 f5 00 00 00 00 48 89 f1 48 29
f0 48 c1 e0 03 48 29 c7 48 01 df 4c 39 ef 74 1b 48 8b 05 a9 16 bb d5
8b b7 88 00 00 00 <29> c6 85 f6 7e 2f 8b 43 10 41 39 06 74 36 48 8b 1b
48 89 d8 f6 c3
[Пт мар 5 11:36:01 2021] RSP: 0018:ffffbd0046487680 EFLAGS: 00000283
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:36:01 2021] RAX: 0000000101664eb1 RBX: ffff9422964bfb90
RCX: 0000000000000000
[Пт мар 5 11:36:01 2021] RDX: 000000000000840f RSI: 00000000029a6128
RDI: ffff9422964bfb80
[Пт мар 5 11:36:01 2021] RBP: ffffffff962deb80 R08: ffffbd00464876d8
R09: 00000000ffebaa15
[Пт мар 5 11:36:01 2021] R10: ffff942a74641900 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:36:01 2021] R13: ffff942a75d90b40 R14: ffffbd00464876c0
R15: fffffffffffffff0
[Пт мар 5 11:36:01 2021] FS: 0000000000000000(0000)
GS:ffff942b1f800000(0000) knlGS:0000000000000000
[Пт мар 5 11:36:01 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Пт мар 5 11:36:01 2021] CR2: 00007fff73454038 CR3: 0000000b7c80a003
CR4: 00000000001606e0
[Пт мар 5 11:36:01 2021] Call Trace:
[Пт мар 5 11:36:01 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:36:01 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:36:01 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:36:01 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:36:01 2021] ? ip_route_output_flow+0x19/0x50
[Пт мар 5 11:36:01 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:36:01 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:36:01 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:36:01 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:36:01 2021] ? eth_header+0x26/0xc0
[Пт мар 5 11:36:01 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:36:01 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:36:01 2021] ? nf_nat_ipv4_csum_recalc+0xb0/0xb0 [nf_nat_ipv4]
[Пт мар 5 11:36:01 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:36:01 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:36:01 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:36:01 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:36:01 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:36:01 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:36:01 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:36:01 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:36:01 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:36:01 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:36:01 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:36:01 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:36:01 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:36:01 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:36:01 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:36:01 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:36:01 2021] ? sort_range+0x20/0x20
[Пт мар 5 11:36:01 2021] run_ksoftirqd+0x26/0x40
[Пт мар 5 11:36:01 2021] smpboot_thread_fn+0xc5/0x160
[Пт мар 5 11:36:01 2021] kthread+0x112/0x130
[Пт мар 5 11:36:01 2021] ? kthread_bind+0x30/0x30
[Пт мар 5 11:36:01 2021] ret_from_fork+0x35/0x40
[Пт мар 5 11:36:44 2021] rcu: INFO: rcu_sched self-detected stall on CPU
[Пт мар 5 11:36:44 2021] rcu: 4-....: (1 GPs behind) idle=32a/0/0x3
softirq=3874694/3874694 fqs=2602
[Пт мар 5 11:36:44 2021] rcu: (t=5251 jiffies g=14930609 q=81134)
[Пт мар 5 11:36:44 2021] NMI backtrace for cpu 4
[Пт мар 5 11:36:44 2021] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G
OEL 4.19.0-14-amd64 #1 Debian 4.19.171-2
[Пт мар 5 11:36:44 2021] Hardware name: HP ProLiant DL360
Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
[Пт мар 5 11:36:44 2021] Call Trace:
[Пт мар 5 11:36:44 2021] <IRQ>
[Пт мар 5 11:36:44 2021] dump_stack+0x66/0x81
[Пт мар 5 11:36:44 2021] nmi_cpu_backtrace.cold.4+0x13/0x50
[Пт мар 5 11:36:44 2021] ? lapic_can_unplug_cpu+0x80/0x80
[Пт мар 5 11:36:44 2021] nmi_trigger_cpumask_backtrace+0xf9/0x100
[Пт мар 5 11:36:44 2021] rcu_dump_cpu_stacks+0x9b/0xcb
[Пт мар 5 11:36:44 2021] rcu_check_callbacks.cold.81+0x1db/0x335
[Пт мар 5 11:36:44 2021] ? tick_sched_do_timer+0x60/0x60
[Пт мар 5 11:36:44 2021] update_process_times+0x28/0x60
[Пт мар 5 11:36:44 2021] tick_sched_handle+0x22/0x60
[Пт мар 5 11:36:44 2021] tick_sched_timer+0x37/0x70
[Пт мар 5 11:36:44 2021] __hrtimer_run_queues+0x100/0x280
[Пт мар 5 11:36:44 2021] hrtimer_interrupt+0x100/0x220
[Пт мар 5 11:36:44 2021] smp_apic_timer_interrupt+0x6a/0x140
[Пт мар 5 11:36:44 2021] apic_timer_interrupt+0xf/0x20
[Пт мар 5 11:36:44 2021] RIP:
0010:nf_conntrack_tuple_taken+0x80/0x240 [nf_conntrack]
[Пт мар 5 11:36:44 2021] Code: 89 0c 24 e8 f2 f3 ff ff 89 da 89 c3 48
0f af d3 48 c1 ea 20 48 8b 0c 24 48 8d 04 d1 48 8b 00 48 89 c3 a8 01
75 4a 0f b6 73 37 <4c> 89 ff 48 8d 04 f5 00 00 00 00 48 89 f1 48 29 f0
48 c1 e0 03 48
[Пт мар 5 11:36:44 2021] RSP: 0018:ffff94231f903760 EFLAGS: 00000246
ORIG_RAX: ffffffffffffff13
[Пт мар 5 11:36:44 2021] RAX: ffff942aa5f50f10 RBX: ffff942aa5f50f10
RCX: 0000000000000001
[Пт мар 5 11:36:44 2021] RDX: 0000000000000436 RSI: 0000000000000000
RDI: ffff94229ee78000
[Пт мар 5 11:36:44 2021] RBP: ffffffff962deb80 R08: ffff94231f9037b8
R09: 00000000db6085b2
[Пт мар 5 11:36:44 2021] R10: ffff94224da82780 R11: 0000000000000001
R12: ffffffffc0670a84
[Пт мар 5 11:36:44 2021] R13: ffff94224cb28140 R14: ffff94231f9037a0
R15: fffffffffffffff0
[Пт мар 5 11:36:44 2021] ? apic_timer_interrupt+0xa/0x20
[Пт мар 5 11:36:44 2021] nf_nat_used_tuple+0x33/0x50 [nf_nat]
[Пт мар 5 11:36:44 2021] nf_nat_l4proto_unique_tuple+0xba/0x1d0 [nf_nat]
[Пт мар 5 11:36:44 2021] get_unique_tuple+0x1ee/0x550 [nf_nat]
[Пт мар 5 11:36:44 2021] nf_nat_setup_info+0x8c/0x290 [nf_nat]
[Пт мар 5 11:36:44 2021] nft_nat_eval+0xcd/0x120 [nft_nat]
[Пт мар 5 11:36:44 2021] nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:36:44 2021] ? ixgbe_xmit_frame_ring+0x611/0xf10 [ixgbe]
[Пт мар 5 11:36:44 2021] ? nft_counter_obj_dump+0x20/0x20 [nft_counter]
[Пт мар 5 11:36:44 2021] ? nft_do_chain+0xea/0x420 [nf_tables]
[Пт мар 5 11:36:44 2021] ? nf_conntrack_tuple_taken+0x5e/0x240 [nf_conntrack]
[Пт мар 5 11:36:44 2021] ? get_unique_tuple+0x265/0x550 [nf_nat]
[Пт мар 5 11:36:44 2021] nft_nat_do_chain+0x66/0x80 [nft_chain_nat_ipv4]
[Пт мар 5 11:36:44 2021] nf_nat_inet_fn+0xea/0x230 [nf_nat]
[Пт мар 5 11:36:44 2021] nf_nat_ipv4_out+0x14/0xa0 [nf_nat_ipv4]
[Пт мар 5 11:36:44 2021] nf_hook_slow+0x44/0xc0
[Пт мар 5 11:36:44 2021] ip_output+0xd2/0xe0
[Пт мар 5 11:36:44 2021] ? ip_fragment.constprop.50+0x80/0x80
[Пт мар 5 11:36:44 2021] ip_forward+0x3bf/0x480
[Пт мар 5 11:36:44 2021] ? ip4_key_hashfn+0xb0/0xb0
[Пт мар 5 11:36:44 2021] ip_rcv+0xbc/0xd0
[Пт мар 5 11:36:44 2021] ? ip_rcv_finish_core.isra.18+0x360/0x360
[Пт мар 5 11:36:44 2021] __netif_receive_skb_one_core+0x5a/0x80
[Пт мар 5 11:36:44 2021] netif_receive_skb_internal+0x2f/0xa0
[Пт мар 5 11:36:44 2021] napi_gro_receive+0xba/0xe0
[Пт мар 5 11:36:44 2021] ixgbe_poll+0x530/0x1170 [ixgbe]
[Пт мар 5 11:36:44 2021] ? tick_sched_do_timer+0x60/0x60
[Пт мар 5 11:36:44 2021] net_rx_action+0x149/0x3b0
[Пт мар 5 11:36:44 2021] __do_softirq+0xde/0x2d8
[Пт мар 5 11:36:44 2021] irq_exit+0xba/0xc0
[Пт мар 5 11:36:44 2021] do_IRQ+0x7f/0xe0
[Пт мар 5 11:36:44 2021] common_interrupt+0xf/0xf
[Пт мар 5 11:36:44 2021] </IRQ>
[Пт мар 5 11:36:44 2021] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[Пт мар 5 11:36:44 2021] Code: e8 7c 85 b2 ff 80 7c 24 0b 00 74 17 9c
58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 1e 0d b8 ff fb
66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff
ff 7f 48 39 c3
[Пт мар 5 11:36:44 2021] RSP: 0018:ffffbd0046307e90 EFLAGS: 00000246
ORIG_RAX: ffffffffffffffd9
[Пт мар 5 11:36:44 2021] RAX: ffff94231f922140 RBX: 000055b8713511c3
RCX: 000000000000001f
[Пт мар 5 11:36:44 2021] RDX: 000055b8713511c3 RSI: 000000003342629e
RDI: 0000000000000000
[Пт мар 5 11:36:44 2021] RBP: ffff94231f92a748 R08: 0000000000000002
R09: 0000000000021a00
[Пт мар 5 11:36:44 2021] R10: 0000d8f461084355 R11: ffff94231f921128
R12: 0000000000000001
[Пт мар 5 11:36:44 2021] R13: ffffffff962b70b8 R14: 0000000000000001
R15: 0000000000000000
[Пт мар 5 11:36:44 2021] do_idle+0x228/0x270
[Пт мар 5 11:36:44 2021] cpu_startup_entry+0x6f/0x80
[Пт мар 5 11:36:44 2021] start_secondary+0x1a4/0x200
[Пт мар 5 11:36:44 2021] secondary_startup_64+0xa4/0xb0
