This patchset fixes bug in ipt_CLUSTERIP when network namespace is destroyed.
First patch fixes deadlock.
Second patch removes wrong warning message.
test commands:
test commands:
%ip netns add vm1
%ip netns exec vm1 bash
%ip link set lo up
%iptables -A INPUT -p tcp -i lo -d 192.168.0.5 --dport 80 \
-j CLUSTERIP --new --hashmode sourceip \
--clustermac 01:00:5e:00:00:20 --total-nodes 2 --local-node 1
%exit
%ip netns del vm1
splat looks like:
[ 341.184508] WARNING: CPU: 1 PID: 87 at net/ipv4/netfilter/ipt_CLUSTERIP.c:840 clusterip_net_exit+0x319/0x380 [ipt_CLUSTERIP]
[ 341.184850] Modules linked in: ipt_CLUSTERIP nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp iptable_filter bpfilter ip_tables x_tables
[ 341.184850] CPU: 1 PID: 87 Comm: kworker/u4:2 Not tainted 4.19.0-rc5+ #16
[ 341.184850] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Aptio CRB, BIOS 5.6.5 07/08/2015
[ 341.227509] Workqueue: netns cleanup_net
[ 341.227509] RIP: 0010:clusterip_net_exit+0x319/0x380 [ipt_CLUSTERIP]
[ 341.227509] Code: 0f 85 7f fe ff ff 48 c7 c2 80 64 2c c0 be a8 02 00 00 48 c7 c7 a0 63 2c c0 c6 05 18 6e 00 00 01 e8 bc 38 ff f5 e9 5b fe ff ff <0f> 0b e9 33 ff ff ff e8 4b 90 50 f6 e9 2d fe ff ff 48 89 df e8 de
[ 341.227509] RSP: 0018:ffff88011086f408 EFLAGS: 00010202
[ 341.227509] RAX: dffffc0000000000 RBX: 1ffff1002210de85 RCX: 0000000000000000
[ 341.227509] RDX: 1ffff1002210de85 RSI: ffff880110813be8 RDI: ffffed002210de58
[ 341.227509] RBP: ffff88011086f4d0 R08: 0000000000000000 R09: 0000000000000000
[ 341.227509] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1002210de81
[ 341.227509] R13: ffff880110625a48 R14: ffff880114cec8c8 R15: 0000000000000014
[ 341.227509] FS: 0000000000000000(0000) GS:ffff880116600000(0000) knlGS:0000000000000000
[ 341.227509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 341.227509] CR2: 00007f11fd38e000 CR3: 000000013ca16000 CR4: 00000000001006e0
[ 341.227509] Call Trace:
[ 341.227509] ? __clusterip_config_find+0x460/0x460 [ipt_CLUSTERIP]
[ 341.227509] ? default_device_exit+0x1ca/0x270
[ 341.227509] ? remove_proc_entry+0x1cd/0x390
[ 341.227509] ? dev_change_net_namespace+0xd00/0xd00
[ 341.227509] ? __init_waitqueue_head+0x130/0x130
[ 341.227509] ops_exit_list.isra.10+0x94/0x140
[ 341.227509] cleanup_net+0x45b/0x900
[ 341.227509] ? net_drop_ns+0x110/0x110
[ 341.227509] ? swapgs_restore_regs_and_return_to_usermode+0x3c/0x80
[ 341.227509] ? save_trace+0x300/0x300
[ 341.227509] ? lock_acquire+0x1e3/0x470
[ 341.227509] ? _raw_spin_unlock_irq+0x29/0x40
[ 341.227509] ? _raw_spin_unlock_irq+0x29/0x40
[ 341.227509] ? __lock_acquire+0x4500/0x4500
[ 341.227509] ? __lock_is_held+0xb4/0x140
[ 341.227509] process_one_work+0xc13/0x1de0
[ 341.227509] ? pwq_dec_nr_in_flight+0x3c0/0x3c0
[ 341.227509] ? set_load_weight+0x270/0x270
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x30/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x30/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x30/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __switch_to_asm+0x30/0x60
[ 341.227509] ? __switch_to_asm+0x24/0x60
[ 341.227509] ? __schedule+0x695/0x1e40
[ 341.227509] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 341.227509] ? __sched_text_start+0x8/0x8
[ 341.227509] ? cyc2ns_read_end+0x10/0x10
[ 341.227509] ? save_trace+0x300/0x300
[ 341.227509] ? sched_clock_local+0xd4/0x140
[ 341.227509] ? find_held_lock+0x39/0x1c0
[ 341.227509] ? worker_thread+0x353/0x1120
[ 341.227509] ? lock_contended+0xdb0/0xdb0
[ 341.227509] ? __lock_acquire+0x4500/0x4500
[ 341.227509] ? do_raw_spin_trylock+0x101/0x1a0
[ 341.227509] ? do_raw_spin_lock+0x1f0/0x1f0
[ 341.227509] worker_thread+0x15d/0x1120
[ 341.227509] ? process_one_work+0x1de0/0x1de0
[ 341.227509] ? cyc2ns_read_end+0x10/0x10
[ 341.227509] ? save_trace+0x300/0x300
[ 341.227509] ? cyc2ns_read_end+0x10/0x10
[ 341.227509] ? kasan_kmalloc+0xa0/0xd0
[ 341.227509] ? sched_clock_local+0xd4/0x140
[ 341.227509] ? find_held_lock+0x39/0x1c0
[ 341.227509] ? check_flags.part.36+0x450/0x450
[ 341.227509] ? _raw_spin_unlock_irqrestore+0x32/0x50
[ 341.227509] ? __kthread_parkme+0x44/0x180
[ 341.227509] ? __bpf_trace_preemptirq_template+0x10/0x10
[ 341.227509] ? __kthread_parkme+0xb6/0x180
[ 341.227509] ? process_one_work+0x1de0/0x1de0
[ 341.227509] kthread+0x322/0x3e0
[ 341.227509] ? kthread_create_worker_on_cpu+0xc0/0xc0
[ 341.227509] ret_from_fork+0x3a/0x50
[ 341.227509] irq event stamp: 61102
[ 341.227509] hardirqs last enabled at (61101): [<ffffffffb63408da>] __call_rcu.constprop.72+0x40a/0x1010
[ 341.227509] hardirqs last disabled at (61102): [<ffffffffb6004979>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 341.227509] softirqs last enabled at (61062): [<ffffffffb82006db>] __do_softirq+0x6db/0xa11
[ 341.227509] softirqs last disabled at (61045): [<ffffffffb61801ae>] irq_exit+0x19e/0x1d0
[ 341.227509] ---[ end trace f9d42961412cbe53 ]---
[ 399.083884] INFO: task kworker/u4:2:87 blocked for more than 30 seconds.
[ 399.091797] Tainted: G W 4.19.0-rc5+ #16
[ 399.098707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 399.107640] kworker/u4:2 D17096 87 2 0x80000000
[ 399.113957] Workqueue: netns cleanup_net
[ 399.118474] Call Trace:
[ 399.121347] ? __schedule+0x68d/0x1e40
[ 399.125688] ? __sched_text_start+0x8/0x8
[ 399.130301] ? dump_stack+0x13a/0x16b
[ 399.134524] ? trace_hardirqs_off+0x6b/0x220
[ 399.139438] ? save_trace+0x300/0x300
[ 399.143669] ? ret_from_fork+0x3a/0x50
[ 399.148001] ? dump_stack+0x13a/0x16b
[ 399.152237] ? find_held_lock+0x13f/0x1c0
[ 399.156868] ? debug_locks_off+0x11/0x80
[ 399.161395] schedule+0xe7/0x380
[ 399.165133] ? __schedule+0x1e40/0x1e40
[ 399.169559] ? _raw_spin_unlock_irq+0x29/0x40
[ 399.174659] ? trace_hardirqs_on+0x93/0x210
[ 399.179470] ? __bpf_trace_preemptirq_template+0x10/0x10
[ 399.185646] __rwsem_down_write_failed_common+0x786/0x1540
[ 399.191932] ? rwsem_spin_on_owner+0x600/0x600
[ 399.197057] ? debug_show_all_locks+0x500/0x500
[ 399.202257] ? debug_show_all_locks+0x500/0x500
[ 399.207455] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 399.212754] ? __lock_is_held+0xb4/0x140
[ 399.217275] ? ftrace_regs_caller_op_ptr+0x6b/0x6b
[ 399.222765] ? ftrace_caller_op_ptr+0xe/0xe
[ 399.227628] ? iptable_filter_net_exit+0x42/0x80 [iptable_filter]
[ 399.234635] ? stop_critical_timings+0x420/0x420
[ 399.239933] ? rcu_read_lock_sched_held+0x114/0x130
[ 399.245528] ? print_irqtrace_events+0x270/0x270
[ 399.250836] ? retint_kernel+0x10/0x10
[ 399.255155] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 399.260449] ? retint_kernel+0x10/0x10
[ 399.264765] ? trace_hardirqs_on_caller+0x9a/0x220
[ 399.270262] ? cyc2ns_read_end+0x10/0x10
[ 399.274790] ? save_trace+0x300/0x300
[ 399.279030] ? sched_clock_local+0xd4/0x140
[ 399.283861] ? sched_clock_cpu+0x126/0x170
[ 399.288624] ? find_held_lock+0x39/0x1c0
[ 399.293141] ? unregister_netdevice_notifier+0x8c/0x460
[ 399.299130] ? lock_acquire+0x1e3/0x470
[ 399.303627] ? __lock_acquire+0x4500/0x4500
[ 399.308445] ? call_rwsem_down_write_failed+0x13/0x20
[ 399.314228] call_rwsem_down_write_failed+0x13/0x20
[ 399.319825] ? cleanup_net+0x119/0x900
[ 399.324152] down_write+0xd3/0x160
[ 399.328086] ? down_read+0x1e0/0x1e0
[ 399.332197] ? sched_clock_cpu+0x126/0x170
[ 399.336906] ? find_held_lock+0x39/0x1c0
[ 399.341437] unregister_netdevice_notifier+0x8c/0x460
[ 399.347235] ? register_netdevice_notifier+0x790/0x790
[ 399.353118] ? __local_bh_enable_ip+0xe9/0x1b0
[ 399.358216] ? __local_bh_enable_ip+0xe9/0x1b0
[ 399.363322] ? clusterip_tg_destroy+0x372/0x650 [ipt_CLUSTERIP]
[ 399.370081] ? trace_hardirqs_on+0x93/0x210
[ 399.374884] ? __bpf_trace_preemptirq_template+0x10/0x10
[ 399.380981] ? clusterip_tg_destroy+0x372/0x650 [ipt_CLUSTERIP]
[ 399.387759] clusterip_tg_destroy+0x3ad/0x650 [ipt_CLUSTERIP]
[ 399.394344] ? clusterip_net_init+0x3d0/0x3d0 [ipt_CLUSTERIP]
[ 399.400924] ? cleanup_match+0x17d/0x200 [ip_tables]
[ 399.406681] ? xt_unregister_table+0x215/0x300 [x_tables]
[ 399.412860] ? kfree+0xe2/0x2a0
[ 399.416510] cleanup_entry+0x1d5/0x2f0 [ip_tables]
[ 399.422015] ? cleanup_match+0x200/0x200 [ip_tables]
[ 399.427726] __ipt_unregister_table+0x9b/0x1a0 [ip_tables]
[ 399.434014] iptable_filter_net_exit+0x43/0x80 [iptable_filter]
[ 399.440788] ops_exit_list.isra.10+0x94/0x140
[ 399.445813] cleanup_net+0x45b/0x900
[ 399.449955] ? net_drop_ns+0x110/0x110
[ 399.454281] ? swapgs_restore_regs_and_return_to_usermode+0x3c/0x80
[ 399.461453] ? save_trace+0x300/0x300
[ 399.465676] ? lock_acquire+0x1e3/0x470
[ 399.470105] ? _raw_spin_unlock_irq+0x29/0x40
[ 399.475102] ? _raw_spin_unlock_irq+0x29/0x40
[ 399.480109] ? __lock_acquire+0x4500/0x4500
[ 399.484930] ? __lock_is_held+0xb4/0x140
[ 399.489468] process_one_work+0xc13/0x1de0
[ 399.494206] ? pwq_dec_nr_in_flight+0x3c0/0x3c0
[ 399.499411] ? set_load_weight+0x270/0x270
[ 399.504112] ? __switch_to_asm+0x24/0x60
[ 399.508674] ? __switch_to_asm+0x30/0x60
[ 399.513183] ? __switch_to_asm+0x24/0x60
[ 399.517696] ? __switch_to_asm+0x24/0x60
[ 399.522201] ? __switch_to_asm+0x30/0x60
[ 399.526711] ? __switch_to_asm+0x24/0x60
[ 399.531228] ? __switch_to_asm+0x24/0x60
[ 399.535739] ? __switch_to_asm+0x30/0x60
[ 399.540246] ? __switch_to_asm+0x24/0x60
[ 399.544761] ? __switch_to_asm+0x24/0x60
[ 399.549263] ? __switch_to_asm+0x30/0x60
[ 399.553775] ? __switch_to_asm+0x24/0x60
[ 399.558298] ? __schedule+0x695/0x1e40
[ 399.562652] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 399.567963] ? __sched_text_start+0x8/0x8
[ 399.572624] ? cyc2ns_read_end+0x10/0x10
[ 399.577146] ? save_trace+0x300/0x300
[ 399.581379] ? sched_clock_local+0xd4/0x140
[ 399.586198] ? find_held_lock+0x39/0x1c0
[ 399.590730] ? worker_thread+0x353/0x1120
[ 399.595355] ? lock_contended+0xdb0/0xdb0
[ 399.599974] ? __lock_acquire+0x4500/0x4500
[ 399.604792] ? do_raw_spin_trylock+0x101/0x1a0
[ 399.609894] ? do_raw_spin_lock+0x1f0/0x1f0
[ 399.614728] worker_thread+0x15d/0x1120
[ 399.619177] ? process_one_work+0x1de0/0x1de0
[ 399.624187] ? cyc2ns_read_end+0x10/0x10
[ 399.628699] ? save_trace+0x300/0x300
[ 399.632922] ? cyc2ns_read_end+0x10/0x10
[ 399.637434] ? kasan_kmalloc+0xa0/0xd0
[ 399.641766] ? sched_clock_local+0xd4/0x140
[ 399.646638] ? find_held_lock+0x39/0x1c0
[ 399.651177] ? check_flags.part.36+0x450/0x450
[ 399.656284] ? _raw_spin_unlock_irqrestore+0x32/0x50
[ 399.661968] ? __kthread_parkme+0x44/0x180
[ 399.666688] ? __bpf_trace_preemptirq_template+0x10/0x10
[ 399.672787] ? __kthread_parkme+0xb6/0x180
[ 399.677504] ? process_one_work+0x1de0/0x1de0
[ 399.682501] kthread+0x322/0x3e0
[ 399.686238] ? kthread_create_worker_on_cpu+0xc0/0xc0
[ 399.692027] ret_from_fork+0x3a/0x50
[ 399.696210] INFO: lockdep is turned off.
Taehee Yoo (2):
netfilter: ipt_CLUSTERIP: fix deadlock in netns exit routine
netfilter: ipt_CLUSTERIP: remove wrong WARN_ON_ONCE in netns
exit routine
net/ipv4/netfilter/ipt_CLUSTERIP.c | 72 +++++++++++++++++-------------
1 file changed, 40 insertions(+), 32 deletions(-)
--
2.17.1
