On Fri, Aug 27, 2021 at 11:57 PM Christoph Hellwig <hch@xxxxxx> wrote: > > On Fri, Aug 27, 2021 at 05:20:44PM -0700, Neeraj Singh wrote: > > You're right. On re-read of the man page, sync_file_range is listed as > > an "extremely dangerous" > > system call. The opportunity in the linux kernel is to offer an > > alternative set of flags or separate > > API that allows for an application like Git to separate a metadata > > writeback request from the disk flush. > > How do you want to do that? I metadata writeback without a cache flush > is worse than useless, in fact it is generally actively harmful. > > To take XFS as an example: fsync and fdatasync do the following thing: > > 1) writeback all dirty data for file to the data device > 2) flush the write cache of the data device to ensure they are really > on disk before writing back the metadata referring to them > 3) write out the log up until the log sequence that contained the last > modifications to the file > 4) flush the cache for the log device. > If the data device and the log device are the same (they usually are > for common setups) and the log device support the FUA bit that writes > through the cache, the log writes use that bit and this step can > be skipped. > > So in general there are very few metadata writes, and it is absolutely > essential to flush the cache before that, because otherwise your metadata > could point to data that might not actually have made it to disk. > > The best way to optimize such a workload is by first batching all the > data writeout for multiple fils in step one, then only doing one cache > flush and one log force (as we call it) to cover all the files. syncfs > will do that, but without a good way to pick individual files. Yes, I think we want to do step (1) of your sequence for all of the files, then issue steps (2-4) for all files as a group. Of course, if the log fills up then we can flush the intermediate steps. The unfortunate thing is that there's no Linux interface to do step (1) and to also ensure that the relevant data is in the log stream or is otherwise available to be part of the durable metadata. It seems to me that XFS would be compatible with this sequence if the appropriate kernel API exists. > > > Separately, I'm hoping I can push from the Windows filesystem side to > > get a barrier primitive put into > > the NVME standard so that we can offer more useful behavior to > > applications rather than these painful > > hardware flushes. > > I'm not sure what you mean with barriers, but if you mean the concept > of implying a global ordering on I/Os as we did in Linux back in the > bad old days the barrier bio flag, or badly reinvented by this paper: > > https://www.usenix.org/conference/fast18/presentation/won > > they might help a little bit with single threaded operations, but will > heavily degrade I/O performance for multithreaded workloads. As an > active member of (but not speaking for) the NVMe technical working group > with a bit of knowledge of SSD internals I also doubt it will be very > well received there. I looked at that paper and definitely agree with you about the questionable implementation strategy they picked. I don't (yet) have detailed knowledge of SSD internals, but it's surprising to me that there is little value to barrier semantics within the drive as opposed to a full durability sync. At least for Windows, we have a database (the Registry) for which any single-threaded latency improvement would be welcome.
