There is a deadlock on linux stable kernels (tested 4.9 and 4.14) when
running an executable from an NFS mount when CONFIG_IMA is enabled
(default IMA policy used).
Repro steps (tested on debian 9 with 4.9 kernel and with 4.14 kernel):
$ sudo mkdir -p /mnt/nfs
$ sudo mount -o ro,exec,nolock <ip>:/vol /mnt/nfs/
# "test.sh" is an executable on the nfs volume
$ /mnt/nfs/test.sh
<deadlock>
The following stack trace shows the deadlock.
[ 2479.260272] bash D 0 25187 25140 0x00000080
[ 2479.267680] ffff93f4d9eec000 0000000000000000 ffff93f4dc211040
ffff93f4ffd18980
[ 2479.278602] ffff93f4dd8370c0 ffffbc2ac386b8f0 ffffffffbde0fed9
0000000000000000
[ 2479.288154] 0000000000000046 ffff93f4ffd18980 ffffffffbd8a2cda
ffff93f4dc211040
[ 2479.297712] Call Trace:
[ 2479.300311] [<ffffffffbde0fed9>] ? __schedule+0x239/0x6f0
[ 2479.306005] [<ffffffffbd8a2cda>] ? check_preempt_curr+0x7a/0x90
[ 2479.313524] [<ffffffffbde103c2>] ? schedule+0x32/0x80
[ 2479.318801] [<ffffffffbde12d30>] ? rwsem_down_read_failed+0xf0/0x150
[ 2479.326855] [<ffffffffbdb3f054>] ? call_rwsem_down_read_failed+0x14/0x30
[ 2479.333802] [<ffffffffbde125ec>] ? down_read+0x1c/0x30
[ 2479.340542] [<ffffffffc050b6b9>] ? nfs_start_io_read+0x19/0x70 [nfs]
[ 2479.348529] [<ffffffffc0502c6f>] ? nfs_file_read+0x2f/0xa0 [nfs]
[ 2479.354779] [<ffffffffbda06fed>] ? new_sync_read+0xdd/0x130
[ 2479.361981] [<ffffffffbdae2a43>] ? integrity_kernel_read+0x43/0x60
[ 2479.368606] [<ffffffffbdae484c>] ? ima_calc_file_hash+0x36c/0x710
[ 2479.376312] [<ffffffffbdae453b>] ? ima_calc_file_hash+0x5b/0x710
[ 2479.382550] [<ffffffffbd98aa86>] ? __alloc_pages_nodemask+0xf6/0x260
[ 2479.389124] [<ffffffffbdae5485>] ? ima_collect_measurement+0x145/0x1b0
[ 2479.397258] [<ffffffffbda2e7fb>] ? vfs_getxattr_alloc+0x5b/0x110
[ 2479.403490] [<ffffffffbdae3c73>] ? process_measurement+0x393/0x550
[ 2479.409913] [<ffffffffc04a177f>] ? generic_hash_cred+0x2f/0x60 [sunrpc]
[ 2479.418138] [<ffffffffc04a19c0>] ?
rpc_lookup_cred_nonblock+0x20/0x20 [sunrpc]
[ 2479.426972] [<ffffffffc04a058a>] ?
rpcauth_lookup_credcache+0x9a/0x2c0 [sunrpc]
[ 2479.434520] [<ffffffffc0465230>] ? nfs3_proc_commit_setup+0x10/0x10 [nfsv3]
[ 2479.443082] [<ffffffffc050717f>] ?
nfs_attribute_cache_expired+0x2f/0x80 [nfs]
[ 2479.450672] [<ffffffffc050749a>] ? nfs_revalidate_inode_rcu+0x1a/0x40 [nfs]
[ 2479.459215] [<ffffffffc05002de>] ? nfs_do_access+0x29e/0x350 [nfs]
[ 2479.467010] [<ffffffffbda0e13a>] ? search_binary_handler+0x2a/0x1c0
[ 2479.473651] [<ffffffffbda0f8fa>] ? do_execveat_common.isra.37+0x5aa/0x790
[ 2479.482055] [<ffffffffbda0fd05>] ? SyS_execve+0x35/0x40
[ 2479.487508] [<ffffffffbd803b7d>] ? do_syscall_64+0x8d/0xf0
[ 2479.494586] [<ffffffffbde14c4e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
Both process_measurement() and nfs_start_io_read() try to acquire the
inode-lock:
process_measurement() {
...
inode_lock(inode); {
down_write(&inode->i_rwsem);
}
...
nfs_start_io_read {
...
down_read(&inode->i_rwsem);
...
This deadlock is primarily fixed by following commit in linux master
(since 4.15):
0d73a55208e94fc9fb6deaeea61438cd3280d4c0 ima: re-introduce own
integrity cache lock
The complete list of commits required (with clean cherry-picks) to fix
the issue in 4.14 stable kernel is below:
e2598077dc6a26c9644393e5c21f22a90dbdccdb ima: re-initialize iint->atomic_flags
0d73a55208e94fc9fb6deaeea61438cd3280d4c0 ima: re-introduce own
integrity cache lock
50b977481fce90aa5fbda55e330b9d722733e358 EVM: Add support for portable
signature format
f3cc6b25dcc5616f0d5c720009b2ac66f97df2ff ima: always measure and audit
files in policy
For 4.9 kernel, one additional commit is required and there is a minor
conflict resolution needed:
19339c251607a3defc7f089511ce8561936fee45 Revert "evm: Translate
user/group ids relative to s_user_ns when computing HMAC"
Since the kernel deadlock is trivial to trigger for common default
configurations, I would like to request cherry-picking the above
commits to linux-stable branches 4.14 and 4.9.
Thanks,
--
Aditya
