One thing I've learned from optimizing I/O performance of userland applications is that a surprising number of applications are effectively "fsync() benchmarks." What I mean by that is many applications call fsync() - and similar functionality incurring a flush to non-volatile storage - surprisingly often and that overall I/O performance is effectively bottlenecked by how quickly the underlying storage device can process repeated flushes. You can easily confirm the validity of this statement by running sync/flush heavy software (like pretty much any database software) with eatmydata [1], an LD_PRELOAD library / utility that effectively short-circuits libc functions like fsync(), fdatasync(), and msync() so they no-op instead of triggering a sync/flush [via a syscall]. The kernel exposes various statistics about block devices, filesystems, the page cache, etc via procfs. However, as far as I can tell there are no direct or very limited statistics on sync/flush operations. I've often wanted to capture/view system/kernel-level activity for these sync/flush operations because they are often strong contributors to I/O bottlenecks and it can be extremely useful to correlate these operations against other metrics. I've long considered sending kernel patches to add procfs observability for sync/flushing activity so common system monitoring tools can easily consume those metrics. I'm emailing linux-block@ and linux-fsdevel@ to a) try to assess whether these patches would be well-received b) gain feedback on the appropriate activity to track (I'm not a kernel expert). I think a reasonable response is "you can already observe this activity via syscall tracing and using tools like systemtap, so no additional procfs counters are needed." I'd counter by saying, yes, you can get some valuable information this way. But I believe there are some large holes, such as limited visibility into block layer activity outside of debugfs and when kernel subsystems (such as background flusher threads) incur work independently of an observable syscall. Do others feel there is an observability gap in the kernel around filesystem syncing and storage flushing? If so, how would you recommend improving matters? Gregory [1] https://www.flamingspork.com/projects/libeatmydata/
