RE: [EXT] Re: FYI, fsnotify contention with aio and io_uring.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Micron Confidential

Hi Jeff and Jens,

About +ACI-FAN+AF8-MODIFY fsnotify watch set on /dev+ACI-.

Was using Fedora34 distro (with 6.3.9 kernel), and fio. Without any particular/specific setting.
I tried to see what could watch /dev but failed at that.
I used the inotify-info tool, but that display watchers using the inotify interface. And nothing was watching /dev via inotify.
Need to figure out how to do the same but for the fanotify interface.
I'll look at it again and let you know.



Micron Confidential
+AD4- -----Original Message-----
+AD4- From: Jens Axboe
+AD4- Sent: Tuesday, August 8, 2023 2:41 PM
+AD4- To: Jeff Moyer Pierre Labat
+AD4- Cc: ''
+AD4- Subject: +AFs-EXT+AF0- Re: FYI, fsnotify contention with aio and io+AF8-uring.
+AD4- CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless you
+AD4- recognize the sender and were expecting this message.
+AD4- On 8/7/23 2:11?PM, Jeff Moyer wrote:
+AD4- +AD4- Hi, Pierre,
+AD4- +AD4-
+AD4- +AD4- Pierre Labat writes:
+AD4- +AD4-
+AD4- +AD4APg- Hi,
+AD4- +AD4APg-
+AD4- +AD4APg- This is FYI, may be you already knows about that, but in case you
+AD4- don't....
+AD4- +AD4APg-
+AD4- +AD4APg- I was pushing the limit of the number of nvme read IOPS, the FIO +-
+AD4- +AD4APg- the Linux OS can handle. For that, I have something special under the
+AD4- +AD4APg- Linux nvme driver. As a consequence I am not limited by whatever the
+AD4- +AD4APg- NVME SSD max IOPS or IO latency would be.
+AD4- +AD4APg-
+AD4- +AD4APg- As I cranked the number of system cores and FIO jobs doing direct 4k
+AD4- +AD4APg- random read on /dev/nvme0n1, I hit a wall. The IOPS scaling slows
+AD4- +AD4APg- (less than linear) and around 15 FIO jobs on 15 core threads, the
+AD4- +AD4APg- overall IOPS, in fact, goes down as I add more FIO jobs. For example
+AD4- +AD4APg- on a system with 24 cores/48 threads, when I goes beyond 15 FIO jobs,
+AD4- +AD4APg- the overall IOPS starts to go down.
+AD4- +AD4APg-
+AD4- +AD4APg- This happens the same for io+AF8-uring and aio. Was using kernel version
+AD4- 6.3.9. Using one namespace (/dev/nvme0n1).
+AD4- +AD4-
+AD4- +AD4- +AFs-snip+AF0-
+AD4- +AD4-
+AD4- +AD4APg- As you can see 76+ACU- of the cpu on the box is sucked up by
+AD4- +AD4APg- lockref+AF8-get+AF8-not+AF8-zero() and lockref+AF8-put+AF8-return().  Looking at the
+AD4- +AD4APg- code, there is contention when IO+AF8-uring call fsnotify+AF8-access().
+AD4- +AD4-
+AD4- +AD4- Is there a FAN+AF8-MODIFY fsnotify watch set on /dev?  If so, it might be
+AD4- +AD4- a good idea to find out what set it and why.
+AD4- This would be my guess too, some distros do seem to do that. The
+AD4- notification bits scale horribly, nobody should use it for anything high
+AD4- performance...
+AD4- --
+AD4- Jens Axboe

[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux