We found a following problem in kernel 5.10, and the same problem should
exist in mainline:
During NFS mount using 'soft' option over RoCE network, we observed kernel
crash with below trace when network issues occur (congestion/disconnect):
nfs: server 10.10.253.211 not responding, timed out
BUG: kernel NULL pointer dereference, address: 00000000000000a0
RIP: 0010:frwr_unmap_sync+0x77/0x200 [rpcrdma]
Call Trace:
? __die_body.cold+0x8/0xd
? no_context+0x155/0x230
? __bad_area_nosemaphore+0x52/0x1a0
? exc_page_fault+0x2dc/0x550
? asm_exc_page_fault+0x1e/0x30
? frwr_unmap_sync+0x77/0x200 [rpcrdma]
xprt_release+0x9e/0x1a0 [sunrpc]
rpc_release_resources_task+0xe/0x50 [sunrpc]
rpc_release_task+0x19/0xa0 [sunrpc]
rpc_async_schedule+0x29/0x40 [sunrpc]
process_one_work+0x1b2/0x350
worker_thread+0x49/0x310
? rescuer_thread+0x380/0x380
kthread+0xfb/0x140
Problem analysis:
The crash happens in frwr_unmap_sync() when accessing req->rl_registered
list, caused by either NULL pointer or accessing freed MR resources.
There's a race condition between:
T1
__ib_process_cq
wc->wr_cqe->done (frwr_wc_localinv)
rpcrdma_flush_disconnect
rpcrdma_force_disconnect
xprt_force_disconnect
xprt_autoclose
xprt_rdma_close
rpcrdma_xprt_disconnect
rpcrdma_reqs_reset
frwr_reset
rpcrdma_mr_pop(&req->rl_registered)
T2
rpc_async_schedule
rpc_release_task
rpc_release_resources_task
xprt_release
xprt_rdma_free
frwr_unmap_sync
rpcrdma_mr_pop(&req->rl_registered)
This problem also exists in function rpcrdma_mrs_destroy().
--
Thanks,
Nan
