On Thu, 2012-06-28 at 10:51 -0400, Jeff Layton wrote: > We've had some reports of a deadlock where rpciod ends up with a stack > trace like this: > > PID: 2507 TASK: ffff88103691ab40 CPU: 14 COMMAND: "rpciod/14" > #0 [ffff8810343bf2f0] schedule at ffffffff814dabd9 > #1 [ffff8810343bf3b8] nfs_wait_bit_killable at ffffffffa038fc04 [nfs] > #2 [ffff8810343bf3c8] __wait_on_bit at ffffffff814dbc2f > #3 [ffff8810343bf418] out_of_line_wait_on_bit at ffffffff814dbcd8 > #4 [ffff8810343bf488] nfs_commit_inode at ffffffffa039e0c1 [nfs] > #5 [ffff8810343bf4f8] nfs_release_page at ffffffffa038bef6 [nfs] > #6 [ffff8810343bf528] try_to_release_page at ffffffff8110c670 > #7 [ffff8810343bf538] shrink_page_list.clone.0 at ffffffff81126271 > #8 [ffff8810343bf668] shrink_inactive_list at ffffffff81126638 > #9 [ffff8810343bf818] shrink_zone at ffffffff8112788f > #10 [ffff8810343bf8c8] do_try_to_free_pages at ffffffff81127b1e > #11 [ffff8810343bf958] try_to_free_pages at ffffffff8112812f > #12 [ffff8810343bfa08] __alloc_pages_nodemask at ffffffff8111fdad > #13 [ffff8810343bfb28] kmem_getpages at ffffffff81159942 > #14 [ffff8810343bfb58] fallback_alloc at ffffffff8115a55a > #15 [ffff8810343bfbd8] ____cache_alloc_node at ffffffff8115a2d9 > #16 [ffff8810343bfc38] kmem_cache_alloc at ffffffff8115b09b > #17 [ffff8810343bfc78] sk_prot_alloc at ffffffff81411808 > #18 [ffff8810343bfcb8] sk_alloc at ffffffff8141197c > #19 [ffff8810343bfce8] inet_create at ffffffff81483ba6 > #20 [ffff8810343bfd38] __sock_create at ffffffff8140b4a7 > #21 [ffff8810343bfd98] xs_create_sock at ffffffffa01f649b [sunrpc] > #22 [ffff8810343bfdd8] xs_tcp_setup_socket at ffffffffa01f6965 [sunrpc] > #23 [ffff8810343bfe38] worker_thread at ffffffff810887d0 > #24 [ffff8810343bfee8] kthread at ffffffff8108dd96 > #25 [ffff8810343bff48] kernel_thread at ffffffff8100c1ca > > The problem is pretty clear. rpciod is trying to allocate memory for a new > socket to talk to the server. The VM ends up calling ->releasepage to get > more memory, and it tries to do a blocking commit. That commit can't > succeed however without a connected socket, so we deadlock. > > Fix this by setting PF_FSTRANS on the workqueue task prior to doing the > socket allocation, and having nfs_release_page check for that flag when > deciding whether to do a blocking commit call. Also, set PF_FSTRANS > unconditionally in rpc_async_schedule since that function can also do > allocations sometimes. > > Also, cc'ing Mel since I borrowed his tsk_restore_flags helper here, and > this patch may cause merge conflicts with some of his Swap-over-NFS > patches. > > Cc: Mel Gorman <mgorman@xxxxxxx> > Signed-off-by: Jeff Layton <jlayton@xxxxxxxxxx> > --- > fs/nfs/file.c | 8 ++++++-- > include/linux/sched.h | 7 +++++++ > net/sunrpc/sched.c | 4 ++++ > net/sunrpc/xprtrdma/transport.c | 4 +++- > net/sunrpc/xprtsock.c | 13 +++++++++++++ > 5 files changed, 33 insertions(+), 3 deletions(-) > > diff --git a/fs/nfs/file.c b/fs/nfs/file.c > index a6708e6b..80cc621 100644 > --- a/fs/nfs/file.c > +++ b/fs/nfs/file.c > @@ -463,8 +463,12 @@ static int nfs_release_page(struct page *page, gfp_t gfp) > if (mapping && (gfp & GFP_KERNEL) == GFP_KERNEL) { > int how = FLUSH_SYNC; > > - /* Don't let kswapd deadlock waiting for OOM RPC calls */ > - if (current_is_kswapd()) > + /* > + * Don't let kswapd deadlock waiting for OOM RPC calls, and > + * don't recurse back into the fs if we know that we're trying > + * to reclaim memory for an fs-related allocation. > + */ > + if (current_is_kswapd() || current->flags & PF_FSTRANS) > how = 0; > nfs_commit_inode(mapping->host, how); This will still try to do a direct reclaim. Even if we make that reclaim non-blocking, I'm thinking that we should perhaps just skip the nfs_commit_inode() in the case where PF_FSTRANS is set: if we're low on memory, then it is better to skip the nfs_commit_inode rather than tie up more resources in an attempt to free up memory that we _know_ is going to be in vain. > } > diff --git a/include/linux/sched.h b/include/linux/sched.h > index 4059c0f..d71b523 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -1889,6 +1889,13 @@ static inline void rcu_switch_from(struct task_struct *prev) > > #endif > > +static inline void tsk_restore_flags(struct task_struct *p, > + unsigned long pflags, unsigned long mask) > +{ > + p->flags &= ~mask; > + p->flags |= pflags & mask; > +} Do we need this? rpciod is a workqueue thread, so in principle we already _know_ that PF_FSTRANS is not going to be set. -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@xxxxxxxxxx www.netapp.com ��.n��������+%������w��{.n�����{��w���jg��������ݢj����G�������j:+v���w�m������w�������h�����٥