Hi,
I came a across a machine recently which has multiple threads blocked in
nfs_writedata_alloc(). They were waiting for mempool_alloc to provide an
allocation but it never did.
Memory was tight and all the pre-allocations were in use by pending requests.
These requests were queued on "NFS client" which means they were waiting for
the state manager to do something.
But there was no state manager. Presumably kthread_run failed when it tried
to allocate some memory.
I cannot see anything that would retry the attempt to start the thread, and
even if there was, we probably need to complete some NFS writes before more
memory comes available.
In this particular case the main problem was quite separate. Too many large
processes and not enough swap space, and the OOM killer missed its target.
So even if NFS had worked perfectly the machine would still have locked up.
But it does suggest that there is a weakness here.
As kthread_create used GFP_KERNEL to allocate a thread, and as writes can
block waiting for the thread to be created, there appears to be room for a
deadlock.
My thought is that this could be fixed by using a WQ_MEM_RECLAIM work queue.
The WQ_MEM_RECLAIM flag ensures there is always at least one thread running
so no allocation is needed. Before diving in and trying to implement this I
thought it would be safest to ask as there are two issues that I'm not
certain of.
1/ nfs4_run_state_manager() explicitly allows SIGKILL. Why is this?
Is there some situation where it might be appropriate to kill the manager
thread?
2/ would it be reasonable to have a single work queue for all nfs clients?
In the worst case this could serialise reclaim across all clients so we
wouldn't want any reclaim attempt to block indefinitely. Is that likely
to be a big problem do you think?
Thanks for any hints or suggestions,
NeilBrown
Attachment:
signature.asc
Description: PGP signature
