On Tue, 2021-09-28 at 23:25 +0800, Miklos Szeredi wrote:
> On Fri, Sep 24, 2021 at 09:52:35AM +0200, Miklos Szeredi wrote:
> > On Fri, 24 Sept 2021 at 05:52, Ed Tsai <ed.tsai@xxxxxxxxxxxx>
> > wrote:
> > >
> > > On Wed, 2021-08-18 at 17:24 +0800, Miklos Szeredi wrote:
> > > > On Tue, 13 Jul 2021 at 04:42, Ed Tsai <ed.tsai@xxxxxxxxxxxx>
> > > > wrote:
> > > > >
> > > > > On Tue, 2021-06-08 at 17:30 +0200, Miklos Szeredi wrote:
> > > > > > On Thu, 3 Jun 2021 at 14:52, chenguanyou <
> > > > > > chenguanyou9338@xxxxxxxxx>
> > > > > > wrote:
> > > > > > >
> > > > > > > ABA deadlock
> > > > > > >
> > > > > > > PID: 17172 TASK: ffffffc0c162c000 CPU: 6 COMMAND:
> > > > > > > "Thread-21"
> > > > > > > 0 [ffffff802d16b400] __switch_to at ffffff8008086a4c
> > > > > > > 1 [ffffff802d16b470] __schedule at ffffff80091ffe58
> > > > > > > 2 [ffffff802d16b4d0] schedule at ffffff8009200348
> > > > > > > 3 [ffffff802d16b4f0] bit_wait at ffffff8009201098
> > > > > > > 4 [ffffff802d16b510] __wait_on_bit at ffffff8009200a34
> > > > > > > 5 [ffffff802d16b5b0] inode_wait_for_writeback at
> > > > > > > ffffff800830e1e8
> > > > > > > 6 [ffffff802d16b5e0] evict at ffffff80082fb15c
> > > > > > > 7 [ffffff802d16b620] iput at ffffff80082f9270
> > > > > > > 8 [ffffff802d16b680] dentry_unlink_inode at
> > > > > > > ffffff80082f4c90
> > > > > > > 9 [ffffff802d16b6a0] __dentry_kill at ffffff80082f1710
> > > > > > > 10 [ffffff802d16b6d0] shrink_dentry_list at
> > > > > > > ffffff80082f1c34
> > > > > > > 11 [ffffff802d16b750] prune_dcache_sb at ffffff80082f18a8
> > > > > > > 12 [ffffff802d16b770] super_cache_scan at
> > > > > > > ffffff80082d55ac
> > > > > > > 13 [ffffff802d16b860] shrink_slab at ffffff8008266170
> > > > > > > 14 [ffffff802d16b900] shrink_node at ffffff800826b420
> > > > > > > 15 [ffffff802d16b980] do_try_to_free_pages at
> > > > > > > ffffff8008268460
> > > > > > > 16 [ffffff802d16ba60] try_to_free_pages at
> > > > > > > ffffff80082680d0
> > > > > > > 17 [ffffff802d16bbe0] __alloc_pages_nodemask at
> > > > > > > ffffff8008256514
> > > > > > > 18 [ffffff802d16bc60] fuse_copy_fill at ffffff8008438268
> > > > > > > 19 [ffffff802d16bd00] fuse_dev_do_read at
> > > > > > > ffffff8008437654
> > > > > > > 20 [ffffff802d16bdc0] fuse_dev_splice_read at
> > > > > > > ffffff8008436f40
> > > > > > > 21 [ffffff802d16be60] sys_splice at ffffff8008315d18
> > > > > > > 22 [ffffff802d16bff0] __sys_trace at ffffff8008084014
> > > > > > >
> > > > > > > PID: 9652 TASK: ffffffc0c9ce0000 CPU: 4 COMMAND:
> > > > > > > "kworker/u16:8"
> > > > > > > 0 [ffffff802e793650] __switch_to at ffffff8008086a4c
> > > > > > > 1 [ffffff802e7936c0] __schedule at ffffff80091ffe58
> > > > > > > 2 [ffffff802e793720] schedule at ffffff8009200348
> > > > > > > 3 [ffffff802e793770] __fuse_request_send at
> > > > > > > ffffff8008435760
> > > > > > > 4 [ffffff802e7937b0] fuse_simple_request at
> > > > > > > ffffff8008435b14
> > > > > > > 5 [ffffff802e793930] fuse_flush_times at ffffff800843a7a0
> > > > > > > 6 [ffffff802e793950] fuse_write_inode at ffffff800843e4dc
> > > > > > > 7 [ffffff802e793980] __writeback_single_inode at
> > > > > > > ffffff8008312740
> > > > > > > 8 [ffffff802e793aa0] writeback_sb_inodes at
> > > > > > > ffffff80083117e4
> > > > > > > 9 [ffffff802e793b00] __writeback_inodes_wb at
> > > > > > > ffffff8008311d98
> > > > > > > 10 [ffffff802e793c00] wb_writeback at ffffff8008310cfc
> > > > > > > 11 [ffffff802e793d00] wb_workfn at ffffff800830e4a8
> > > > > > > 12 [ffffff802e793d90] process_one_work at
> > > > > > > ffffff80080e4fac
> > > > > > > 13 [ffffff802e793e00] worker_thread at ffffff80080e5670
> > > > > > > 14 [ffffff802e793e60] kthread at ffffff80080eb650
> > > > > >
> > > > > > The issue is real.
> > > > > >
> > > > > > The fix, however, is not the right one. The fundamental
> > > > > > problem
> > > > > > is
> > > > > > that fuse_write_inode() blocks on a request to userspace.
> > > > > >
> > > > > > This is the same issue that fuse_writepage/fuse_writepages
> > > > > > face. In
> > > > > > that case the solution was to copy the page contents to a
> > > > > > temporary
> > > > > > buffer and return immediately as if the writeback already
> > > > > > completed.
> > > > > >
> > > > > > Something similar needs to be done here: send the
> > > > > > FUSE_SETATTR
> > > > > > request
> > > > > > asynchronously and return immediately from
> > > > > > fuse_write_inode(). The
> > > > > > tricky part is to make sure that multiple time updates for
> > > > > > the
> > > > > > same
> > > > > > inode aren't mixed up...
> > > > > >
> > > > > > Thanks,
> > > > > > Miklos
> > > > >
> > > > > Dear Szeredi,
> > > > >
> > > > > Writeback thread calls fuse_write_inode() and wait for user
> > > > > Daemon
> > > > > to
> > > > > complete this write inode request. The user daemon will
> > > > > alloc_page()
> > > > > after taking this request, and a deadlock could happen when
> > > > > we try
> > > > > to
> > > > > shrink dentry list under memory pressure.
> > > > >
> > > > > We (Mediatek) glad to work on this issue for mainline and
> > > > > also LTS.
> > > > > So
> > > > > another problem is that we should not change the protocol or
> > > > > feature
> > > > > for stable kernel.
> > > > >
> > > > > Use GFP_NOFS | __GFP_HIGHMEM can really avoid this by skip
> > > > > the
> > > > > dentry
> > > > > shirnker. It works but degrade the alloc_page success rate.
> > > > > In a
> > > > > more
> > > > > fundamental way, we could cache the contents and return
> > > > > immediately.
> > > > > But how to ensure the request will be done successfully,
> > > > > e.g.,
> > > > > always
> > > > > retry if it fails from daemon.
> > > >
> > > > Key is where the the dirty metadata is flushed. To prevent
> > > > deadlock
> > > > it must not be flushed from memory reclaim, so must make sure
> > > > that it
> > > > is flushed on close(2) and munmap(2) and not dirtied after
> > > > that.
> > > >
> > > > I'm working on this currently and hope to get it ready for the
> > > > next
> > > > merge window.
> > > >
> > > > Thanks,
> > > > Miklos
> > >
> > > Hi Miklos,
> > >
> > > I'm not sure whether it has already been resolved in mainline.
> > > If it still WIP, please cc me on future emails.
> >
> > Hi,
> >
> > This is taking a bit longer, unfortunately, but I already have
> > something in testing and currently cleaning it up for review. Hope
> > to
> > post a series today or early next week.
>
>
> Here's a minimal patch. It's been through some iterations and some
> testing, but
> more review and testing is definitely welcome.
>
> Chenguanyou, can you please verify that it fixes the deadlock?
>
> Thanks,
> Miklos
>
> ---
> From: Miklos Szeredi <mszeredi@xxxxxxxxxx>
> Subject: fuse: make sure reclaim doesn't write the inode
>
> In writeback cache mode mtime/ctime updates are cached, and flushed
> to the
> server using the ->write_inode() callback.
>
> Closing the file will result in a dirty inode being immediately
> written,
> but in other cases the inode can remain dirty after all references
> are
> dropped. This result in the inode being written back from reclaim,
> which
> can deadlock on a regular allocation while the request is being
> served.
>
> The usual mechanisms (GFP_NOFS/PF_MEMALLOC*) don't work for FUSE,
> because
> serving a request involves unrelated userspace process(es).
>
> Instead do the same as for dirty pages: make sure the inode is
> written
> before the last reference is gone.
>
> - fuse_vma_close(): flush times in addition to the dirty pages
>
> - fallocate(2)/copy_file_range(2): these call file_update_time() or
> file_modified(), so flush the inode before returning from the call
>
> - unlink(2), link(2) and rename(2): these call fuse_update_ctime(),
> so
> flush the ctime directly from this helper
>
> Reported-by: chenguanyou <chenguanyou@xxxxxxxxxx>
> Signed-off-by: Miklos Szeredi <mszeredi@xxxxxxxxxx>
> ---
> fs/fuse/dir.c | 8 ++++++++
> fs/fuse/file.c | 24 +++++++++++++++++++++---
> fs/fuse/fuse_i.h | 1 +
> 3 files changed, 30 insertions(+), 3 deletions(-)
>
> --- a/fs/fuse/dir.c
> +++ b/fs/fuse/dir.c
> @@ -738,12 +738,20 @@ static int fuse_symlink(struct user_name
> return create_new_entry(fm, &args, dir, entry, S_IFLNK);
> }
>
> +void fuse_flush_time_update(struct inode *inode)
> +{
> + int err = sync_inode_metadata(inode, 1);
> +
> + mapping_set_error(inode->i_mapping, err);
> +}
> +
> void fuse_update_ctime(struct inode *inode)
> {
> fuse_invalidate_attr(inode);
> if (!IS_NOCMTIME(inode)) {
> inode->i_ctime = current_time(inode);
> mark_inode_dirty_sync(inode);
> + fuse_flush_time_update(inode);
> }
> }
>
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -1847,6 +1847,17 @@ int fuse_write_inode(struct inode *inode
> struct fuse_file *ff;
> int err;
>
> + /*
> + * Inode is always written before the last reference is dropped
> and
> + * hence this should not be reached from reclaim.
> + *
> + * Writing back the inode from reclaim can deadlock if the
> request
> + * processing itself needs an allocation. Allocations
> triggering
> + * reclaim while serving a request can't be prevented, because
> it can
> + * involve any number of unrelated userspace processes.
> + */
> + WARN_ON(wbc->for_reclaim);
> +
> ff = __fuse_write_file_get(fi);
> err = fuse_flush_times(inode, ff);
> if (ff)
> @@ -2339,12 +2350,15 @@ static int fuse_launder_page(struct page
> }
>
> /*
> - * Write back dirty pages now, because there may not be any suitable
> - * open files later
> + * Write back dirty data/metadata now (there may not be any suitable
> + * open files later for data)
> */
> static void fuse_vma_close(struct vm_area_struct *vma)
> {
> - filemap_write_and_wait(vma->vm_file->f_mapping);
> + int err;
> +
> + err = write_inode_now(vma->vm_file->f_mapping->host, 1);
> + mapping_set_error(vma->vm_file->f_mapping, err);
> }
>
> /*
> @@ -3001,6 +3015,8 @@ static long fuse_file_fallocate(struct f
> if (lock_inode)
> inode_unlock(inode);
>
> + fuse_flush_time_update(inode);
> +
> return err;
> }
>
> @@ -3110,6 +3126,8 @@ static ssize_t __fuse_copy_file_range(st
> inode_unlock(inode_out);
> file_accessed(file_in);
>
> + fuse_flush_time_update(inode_out);
> +
> return err;
> }
>
> --- a/fs/fuse/fuse_i.h
> +++ b/fs/fuse/fuse_i.h
> @@ -1145,6 +1145,7 @@ int fuse_allow_current_process(struct fu
>
> u64 fuse_lock_owner_id(struct fuse_conn *fc, fl_owner_t id);
>
> +void fuse_flush_time_update(struct inode *inode);
> void fuse_update_ctime(struct inode *inode);
>
> int fuse_update_attributes(struct inode *inode, struct file *file);
Hi Mikloz, Greg,
This deadlock issue could be raised in high memory pressure and the
patch has been merged in commit 5c791fe ("fuse: make sure reclaim
doesn't write the inode").
Can we take it to the LTS version?
Best,
Ed Tsai
