Hello,
First, thank you for all the work you did and are still doing around
Netfilter!
I'm writing you this email, because when I run the MPTCP test suite with
a VM running a kernel built with a debug config including
CONFIG_PROVE_RCU_LIST=y (and CONFIG_RCU_EXPERT=y), I get the following
warning:
> =============================
> WARNING: suspicious RCU usage
> 6.12.0-rc3+ #7 Not tainted
> -----------------------------
> net/netfilter/nf_tables_api.c:3420 RCU-list traversed in non-reader section!!
>
> other info that might help us debug this:
>
>
> rcu_scheduler_active = 2, debug_locks = 1
> 1 lock held by iptables/134:
> #0: ffff888008c4fcc8 (&nft_net->commit_mutex){+.+.}-{3:3}, at: nf_tables_valid_genid (include/linux/jiffies.h:101) nf_tables
>
> stack backtrace:
> CPU: 1 UID: 0 PID: 134 Comm: iptables Not tainted 6.12.0-rc3+ #7
> Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> Call Trace:
> <TASK>
> dump_stack_lvl (lib/dump_stack.c:123)
> lockdep_rcu_suspicious (kernel/locking/lockdep.c:6822)
> __nft_rule_lookup (net/netfilter/nf_tables_api.c:3420 (discriminator 7)) nf_tables
> nf_tables_delrule (net/netfilter/nf_tables_api.c:4300 (discriminator 1)) nf_tables
> ? __pfx_nf_tables_delrule (net/netfilter/nf_tables_api.c:4262) nf_tables
> ? __mutex_unlock_slowpath (arch/x86/include/asm/atomic64_64.h:101 (discriminator 1))
> ? __nla_validate_parse (lib/nlattr.c:638)
> nfnetlink_rcv_batch (net/netfilter/nfnetlink.c:524)
> ? __pfx_nfnetlink_rcv_batch (net/netfilter/nfnetlink.c:373)
> ? rcu_read_lock_any_held (kernel/rcu/update.c:386 (discriminator 1))
> ? validate_chain (kernel/locking/lockdep.c:3797 (discriminator 1))
> ? rcu_read_lock_any_held (kernel/rcu/update.c:386 (discriminator 1))
> ? validate_chain (kernel/locking/lockdep.c:3797 (discriminator 1))
> ? __pfx_validate_chain (kernel/locking/lockdep.c:3860)
> ? __nla_validate_parse (lib/nlattr.c:638)
> nfnetlink_rcv (net/netfilter/nfnetlink.c:647)
> ? __pfx___netlink_lookup (net/netlink/af_netlink.c:512)
> ? __pfx_nfnetlink_rcv (net/netfilter/nfnetlink.c:651)
> netlink_unicast (net/netlink/af_netlink.c:1331)
> ? __pfx_netlink_unicast (net/netlink/af_netlink.c:1342)
> ? find_held_lock (kernel/locking/lockdep.c:5315 (discriminator 1))
> ? __might_fault (mm/memory.c:6700 (discriminator 5))
> netlink_sendmsg (net/netlink/af_netlink.c:1901)
> ? __pfx_netlink_sendmsg (net/netlink/af_netlink.c:1820)
> ? __import_iovec (lib/iov_iter.c:1433 (discriminator 1))
> ____sys_sendmsg (net/socket.c:729 (discriminator 1))
> ? __pfx_____sys_sendmsg (net/socket.c:2553)
> ? __pfx_copy_msghdr_from_user (net/socket.c:2533)
> ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4347)
> ? sk_setsockopt (net/core/sock.c:1129)
> ? __local_bh_enable_ip (arch/x86/include/asm/irqflags.h:42)
> ___sys_sendmsg (net/socket.c:2663)
> ? __pfx_sk_setsockopt (net/core/sock.c:1163)
> ? __pfx____sys_sendmsg (net/socket.c:2650)
> ? mark_lock (kernel/locking/lockdep.c:4703 (discriminator 1))
> ? fdget (include/linux/atomic/atomic-arch-fallback.h:479 (discriminator 2))
> ? find_held_lock (kernel/locking/lockdep.c:5315 (discriminator 1))
> __sys_sendmsg (net/socket.c:2690 (discriminator 1))
> ? __pfx___sys_sendmsg (net/socket.c:2678)
> ? __sys_setsockopt (include/linux/file.h:35)
> do_syscall_64 (arch/x86/entry/common.c:52 (discriminator 1))
> entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
> RIP: 0033:0x7ff6a026e004
It is very easy for me to reproduce it: simply by adding and removing an
IPTables rule. Just in case, here are the steps that can be used to have
the same behaviour:
$ cd [kernel source code]
$ echo "iptables -A OUTPUT -j REJECT; iptables -D OUTPUT -j REJECT" \
> .virtme-exec-run
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-debug -e RCU_EXPERT -e PROVE_RCU_LIST
I looked a bit at the code in net/netfilter/nf_tables_api.c, and I can
see that rcu_read_(un)lock() are probably missing, but I'm a bit
confused by how the chain->rules list is protected, and modified using
or not the RCU helpers.
Do you mind looking at this issue please?
(No urgency on my side.)
Cheers,
Matt
--
Sponsored by the NGI0 Core fund.
