Hello!
Getting the following git kernel trace on boot with rc.local having :
ipset create sshguard4 hash:net
iptables -A INPUT -p tcp --dport 22 -m set --match-set sshguard4 src -j DROP
current git kernel:
$ uname -a
Linux ttip 5.2.0-rc5 #981 SMP Mon Jun 17 09:52:04 MSK 2019 sparc64 GNU/Linux
linux-2.6$ git desc
v5.2-rc5
$ dmesg
<cut>
[ 10.356388] Adding 787176k swap on /dev/vdiska4. Priority:-2
extents:1 across:787176k FS
[ 10.471900] EXT4-fs (vdiska1): mounting ext3 file system using the
ext4 subsystem
[ 10.487226] EXT4-fs (vdiska1): mounted filesystem with ordered data
mode. Opts: (null)
[ 11.158102] random: crng init done
[ 11.158155] random: 7 urandom warning(s) missed due to ratelimiting
[ 11.697866] ======================================================
[ 11.697875] WARNING: possible circular locking dependency detected
[ 11.697886] 5.2.0-rc5 #981 Not tainted
[ 11.697894] ------------------------------------------------------
[ 11.697902] iptables/732 is trying to acquire lock:
[ 11.697913] 000000004f61aa56 (&table[i].mutex){+.+.}, at:
nfnl_lock+0x24/0x40 [nfnetlink]
[ 11.697937]
but task is already holding lock:
[ 11.697946] 000000000d652829 (&net->nft.commit_mutex){+.+.}, at:
nf_tables_valid_genid+0x18/0x60 [nf_tables]
[ 11.697973]
which lock already depends on the new lock.
[ 11.697983]
the existing dependency chain (in reverse order) is:
[ 11.697992]
-> #1 (&net->nft.commit_mutex){+.+.}:
[ 11.698012] __mutex_lock+0x48/0x920
[ 11.698021] mutex_lock_nested+0x1c/0x40
[ 11.698033] nf_tables_valid_genid+0x18/0x60 [nf_tables]
[ 11.698043] nfnetlink_rcv_batch+0x24c/0x620 [nfnetlink]
[ 11.698053] nfnetlink_rcv+0x110/0x140 [nfnetlink]
[ 11.698067] netlink_unicast+0x12c/0x1e0
[ 11.698076] netlink_sendmsg+0x324/0x360
[ 11.698091] sock_sendmsg+0x34/0x80
[ 11.698099] ___sys_sendmsg+0x228/0x240
[ 11.698108] __sys_sendmsg+0x4c/0x80
[ 11.698116] sys_sendmsg+0x18/0x40
[ 11.698131] linux_sparc_syscall+0x34/0x44
[ 11.698138]
-> #0 (&table[i].mutex){+.+.}:
[ 11.698157] lock_acquire+0x1a4/0x1c0
[ 11.698165] __mutex_lock+0x48/0x920
[ 11.698173] mutex_lock_nested+0x1c/0x40
[ 11.698181] nfnl_lock+0x24/0x40 [nfnetlink]
[ 11.698196] ip_set_nfnl_get_byindex+0x19c/0x280 [ip_set]
[ 11.698207] set_match_v1_checkentry+0x14/0xc0 [xt_set]
[ 11.698222] xt_check_match+0x238/0x260 [x_tables]
[ 11.698234] __nft_match_init+0x160/0x180 [nft_compat]
[ 11.698244] nft_match_init+0x18/0x40 [nft_compat]
[ 11.698256] nf_tables_newrule+0x57c/0x7a0 [nf_tables]
[ 11.698266] nfnetlink_rcv_batch+0x3f8/0x620 [nfnetlink]
[ 11.698275] nfnetlink_rcv+0x110/0x140 [nfnetlink]
[ 11.698284] netlink_unicast+0x12c/0x1e0
[ 11.698292] netlink_sendmsg+0x324/0x360
[ 11.698300] sock_sendmsg+0x34/0x80
[ 11.698309] ___sys_sendmsg+0x228/0x240
[ 11.698317] __sys_sendmsg+0x4c/0x80
[ 11.698325] sys_sendmsg+0x18/0x40
[ 11.698334] linux_sparc_syscall+0x34/0x44
[ 11.698340]
other info that might help us debug this:
[ 11.698351] Possible unsafe locking scenario:
[ 11.698359] CPU0 CPU1
[ 11.698366] ---- ----
[ 11.698372] lock(&net->nft.commit_mutex);
[ 11.698381] lock(&table[i].mutex);
[ 11.698390] lock(&net->nft.commit_mutex);
[ 11.698400] lock(&table[i].mutex);
[ 11.698408]
*** DEADLOCK ***
[ 11.698418] 1 lock held by iptables/732:
[ 11.698424] #0: 000000000d652829 (&net->nft.commit_mutex){+.+.},
at: nf_tables_valid_genid+0x18/0x60 [nf_tables]
[ 11.698444]
stack backtrace:
[ 11.698454] CPU: 6 PID: 732 Comm: iptables Not tainted 5.2.0-rc5 #981
[ 11.698463] Call Trace:
[ 11.698471] [00000000004cfde0] print_circular_bug+0x2e0/0x320
[ 11.698480] [00000000004d4bd8] __lock_acquire+0x1d38/0x2900
[ 11.698489] [00000000004d6084] lock_acquire+0x1a4/0x1c0
[ 11.698498] [0000000000a06508] __mutex_lock+0x48/0x920
[ 11.698506] [0000000000a06dfc] mutex_lock_nested+0x1c/0x40
[ 11.698516] [000000001071c024] nfnl_lock+0x24/0x40 [nfnetlink]
[ 11.698527] [00000000107568dc] ip_set_nfnl_get_byindex+0x19c/0x280 [ip_set]
[ 11.698537] [000000001078e5d4] set_match_v1_checkentry+0x14/0xc0 [xt_set]
[ 11.698549] [0000000010310ed8] xt_check_match+0x238/0x260 [x_tables]
[ 11.698559] [000000001077cc00] __nft_match_init+0x160/0x180 [nft_compat]
[ 11.698569] [000000001077ccb8] nft_match_init+0x18/0x40 [nft_compat]
[ 11.698582] [0000000010731c3c] nf_tables_newrule+0x57c/0x7a0 [nf_tables]
[ 11.698592] [000000001071d238] nfnetlink_rcv_batch+0x3f8/0x620 [nfnetlink]
[ 11.698602] [000000001071d570] nfnetlink_rcv+0x110/0x140 [nfnetlink]
[ 11.698611] [000000000093e82c] netlink_unicast+0x12c/0x1e0
[ 11.698620] [000000000093f484] netlink_sendmsg+0x324/0x360
Full kernel configuration file as well full dmesg messages are
available at https://github.com/mator/sparc64-dmesg/
system info:
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/sparc64-linux-gnu/8/lto-wrapper
Target: sparc64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian
8.3.0-7' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs
--enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr
--with-gcc-major-version-only --program-suffix=-8
--program-prefix=sparc64-linux-gnu- --enable-shared
--enable-linker-build-id --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --libdir=/usr/lib
--enable-nls --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-libquadmath
--disable-libquadmath-support --enable-plugin --enable-default-pie
--with-system-zlib --disable-libphobos --enable-objc-gc=auto
--enable-multiarch --disable-werror --with-cpu-32=ultrasparc
--enable-targets=all --with-long-double-128 --enable-multilib
--enable-checking=release --build=sparc64-linux-gnu
--host=sparc64-linux-gnu --target=sparc64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-7)
# ldconfig -V
ldconfig (Debian GLIBC 2.28-10) 2.28
# ld -V
GNU ld (GNU Binutils for Debian) 2.31.1
PS: i wasn't able to trace which kernel version introduced this
possible deadlock... but tried (from top git tag v5.2-rc1 to bottom)
up to 4.13 kernel version...
