On Sat, Jun 01, 2019 at 11:01:42AM +0300, Amir Goldstein wrote: > On Sat, Jun 1, 2019 at 2:28 AM Dave Chinner <david@xxxxxxxxxxxxx> wrote: > > > > On Sat, Jun 01, 2019 at 08:45:49AM +1000, Dave Chinner wrote: > > > Given that we can already use AIO to provide this sort of ordering, > > > and AIO is vastly faster than synchronous IO, I don't see any point > > > in adding complex barrier interfaces that can be /easily implemented > > > in userspace/ using existing AIO primitives. You should start > > > thinking about expanding libaio with stuff like > > > "link_after_fdatasync()" and suddenly the whole problem of > > > filesystem data vs metadata ordering goes away because the > > > application directly controls all ordering without blocking and > > > doesn't need to care what the filesystem under it does.... > > > > And let me point out that this is also how userspace can do an > > efficient atomic rename - rename_after_fdatasync(). i.e. on > > completion of the AIO_FSYNC, run the rename. This guarantees that > > the application will see either the old file of the complete new > > file, and it *doesn't have to wait for the operation to complete*. > > Once it is in flight, the file will contain the old data until some > > point in the near future when will it contain the new data.... > > What I am looking for is a way to isolate the effects of "atomic rename/link" > from the rest of the users. Sure there is I/O bandwidth and queued > bios, but at least isolate other threads working on other files or metadata > from contending with the "atomic rename" thread of journal flushes and > the like. That's not a function of the kernel API. That's a function of the implementation behind the kernel API. i.e. The API requires data to be written before the rename/link is committed, how that is achieved is up to the filesystem. And some filesystems will not be able to isolate the API behavioural requirement from other users.... > Actually, one of my use cases is "atomic rename" of files with > no data (looking for atomicity w.r.t xattr and mtime), so this "atomic rename" > thread should not be interfering with other workloads at all. Which should already guaranteed because a) rename is supposed to be atomic, and b) metadata ordering requirements in journalled filesystems. If they lose xattrs across rename, there's something seriously wrong with the filesystem implementation. I'm really not sure what you think filesystems are actually doing with metadata across rename operations.... > > Seriously, sit down and work out all the "atomic" data vs metadata > > behaviours you want, and then tell me how many of them cannot be > > implemented as "AIO_FSYNC w/ completion callback function" in > > userspace. This mechanism /guarantees ordering/ at the application > > level, the application does not block waiting for these data > > integrity operations to complete, and you don't need any new kernel > > side functionality to implement this. > > So I think what I could have used is AIO_BATCH_FSYNC, an interface > that was proposed by Ric Wheeler and discussed on LSF: > https://lwn.net/Articles/789024/ > Ric was looking for a way to efficiently fsync a "bunch of files". > Submitting several AIO_FSYNC calls is not the efficient way of doing that. /me sighs. That's not what I just suggested, and I've already addressed this "AIO_FSYNC sucks" FUD in multiple separate threads. You do realise you can submit multiple AIO operations with a single io_submit() call, right? struct iocb ioc[10]; struct io_event ev[10]; for (i = 0; i < 10; i++) { io_prep_fsync(&ioc[i], fd[i]); ioc[i]->data = callback_arg[i]; } io_submit(aio_ctx, 10, &ioc); io_getevents(aio_ctx, 10, 10, ev, NULL); for (i = 0; i < 10; i++) post_fsync_callback(&ev[i]); There's your single syscall AIO_BATCH_FSYNC functionality, and it implements a per-fd post-fsync callback function. This isn't rocket science.... [snip] > I am trying to reduce the number of fsyncs from applications > and converting fsync to AIO_FSYNC is not going to help with that. Your whole argument is "fsync is inefficient because cache flushes, therefore AIO_FSYNC must be inefficient." IOWs, you've already decided what is wrong, how it can and can't be fixed and the solution you want regardless of whether your assertions are correct or not. You haven't provided any evidence that a new kernel API is the only viable solution, nor that the existing ones cannot provide the functionality you require. So, in the interests of /informed debate/, please implement what you want using batched AIO_FSYNC + rename/linkat completion callback and measure what it acheives. Then implement a sync_file_range/linkat thread pool that provides the same functionality to the application (i.e. writeback concurrency in userspace) and measure it. Then we can discuss what the relative overhead is with numbers and can perform analysis to determine what the cause of the performance differential actually is. Neither of these things require kernel modifications, but you need to provide the evidence that existing APIs are insufficient. Indeed, we now have the new async ioring stuff that can run async sync_file_range calls, so you probably need to benchmark replacing AIO_FSYNC with that interface as well. This new API likely does exactly what you want without the journal/device cache flush overhead of AIO_FSYNC.... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
