Re: [ext4 io hang] buffered write io hang in balance_dirty_pages

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2023/4/27 18:01, Ming Lei wrote:
On Thu, Apr 27, 2023 at 02:36:51PM +0800, Baokun Li wrote:
On 2023/4/27 12:50, Ming Lei wrote:
Hello Matthew,

On Thu, Apr 27, 2023 at 04:58:36AM +0100, Matthew Wilcox wrote:
On Thu, Apr 27, 2023 at 10:20:28AM +0800, Ming Lei wrote:
Hello Guys,

I got one report in which buffered write IO hangs in balance_dirty_pages,
after one nvme block device is unplugged physically, then umount can't
succeed.
That's a feature, not a bug ... the dd should continue indefinitely?
Can you explain what the feature is? And not see such 'issue' or 'feature'
on xfs.

The device has been gone, so IMO it is reasonable to see FS buffered write IO
failed. Actually dmesg has shown that 'EXT4-fs (nvme0n1): Remounting
filesystem read-only'. Seems these things may confuse user.

The reason for this difference is that ext4 and xfs handle errors
differently.

ext4 remounts the filesystem as read-only or even just continues, vfs_write
does not check for these.
vfs_write may not find anything wrong, but ext4 remount could see that
disk is gone, which might happen during or after remount, however.

xfs shuts down the filesystem, so it returns a failure at
xfs_file_write_iter when it finds an error.


``` ext4
ksys_write
  vfs_write
   ext4_file_write_iter
    ext4_buffered_write_iter
     ext4_write_checks
      file_modified
       file_modified_flags
        __file_update_time
         inode_update_time
          generic_update_time
           __mark_inode_dirty
            ext4_dirty_inode ---> 2. void func, No propagating errors out
             __ext4_journal_start_sb
              ext4_journal_check_start ---> 1. Error found, remount-ro
     generic_perform_write ---> 3. No error sensed, continue
      balance_dirty_pages_ratelimited
       balance_dirty_pages_ratelimited_flags
        balance_dirty_pages
         // 4. Sleeping waiting for dirty pages to be freed
         __set_current_state(TASK_KILLABLE)
         io_schedule_timeout(pause);
```

``` xfs
ksys_write
  vfs_write
   xfs_file_write_iter
    if (xfs_is_shutdown(ip->i_mount))
      return -EIO;    ---> dd fail
```
Thanks for the info which is really helpful for me to understand the
problem.

balance_dirty_pages() is sleeping in KILLABLE state, so kill -9 of
the dd process should succeed.
Yeah, dd can be killed, however it may be any application(s), :-)

Fortunately it won't cause trouble during reboot/power off, given
userspace will be killed at that time.



Thanks,
Ming

Don't worry about that, we always set the current thread to TASK_KILLABLE

while waiting in balance_dirty_pages().
I have another concern, if 'dd' isn't killed, dirty pages won't be cleaned, and
these (big amount)memory becomes not usable, and typical scenario could be USB HDD
unplugged.


thanks,
Ming
Yes, it is unreasonable to continue writing data with the previously opened fd after
the file system becomes read-only, resulting in dirty page accumulation.

I provided a patch in another reply.
Could you help test if it can solve your problem?
If it can indeed solve your problem, I will officially send it to the email list.

--
With Best Regards,
Baokun Li
.



[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux