On Tue, Nov 03, 2015 at 04:04:13PM +1100, Dave Chinner wrote: > On Mon, Nov 02, 2015 at 07:53:27PM -0800, Dan Williams wrote: > > On Mon, Nov 2, 2015 at 1:44 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote: <> > > > This comes back to the comments I made w.r.t. the pmem driver > > > implementation doing synchronous IO by immediately forcing CPU cache > > > flushes and barriers. it's obviously correct, but it looks like > > > there's going to be a major performance penalty associated with it. > > > This is why I recently suggested that a pmem driver that doesn't do > > > CPU cache writeback during IO but does it on REQ_FLUSH is an > > > architecture we'll likely have to support. > > > > > > > The only thing we can realistically delay is wmb_pmem() i.e. the final > > sync waiting for data that has *left* the cpu cache. Unless/until we > > get a architecturally guaranteed method to write-back the entire > > cache, or flush the cache by physical-cache-way we're stuck with > > either non-temporal cycles or looping on potentially huge virtual > > address ranges. > > I'm missing something: why won't flushing the address range returned > by bdev_direct_access() during a fsync operation work? i.e. we're > working with exactly the same address as dax_clear_blocks() and > dax_do_io() use, so why can't we look up that address and flush it > from fsync? I could be wrong, but I don't see a reason why DAX can't use the strategy of writing data and marking it dirty in one step and then flushing later in response to fsync/msync. I think this could be used everywhere we write or zero data - dax_clear_blocks(), dax_io() etc. (I believe that lots of the block zeroing code will go away once we have the XFS and ext4 patches in that guarantee we will only get written and zeroed extents from the filesystem in response to get_block().) I think the PMEM driver, lacking the ability to mark things as dirty in the radix tree, etc, will need to keep doing things synchronously. Hmm...if we go this path, though, is that an argument against moving the zeroing from DAX down into the driver? True, with BRD it makes things nice and efficient because you can zero and never flush, and the driver knows there's nothing else to do. For PMEM, though, you lose the ability to zero the data and then queue the flushing for later, as you would be able to do if you left the zeroing code in DAX. The benefit of this is that if you are going to immediately re-write the newly zeroed data (which seems common), PMEM will end up doing an extra cache flush of the zeroes, only to have them overwritten and marked as dirty by DAX. If we leave the zeroing to DAX we can mark it dirty once, zero it once, write it once, and flush it once. This would make us lose the ability to do hardware-assisted flushing in the future that requires driver specific knowledge, though I don't think that exists yet. Perhaps we should leave the zeroing in DAX for now to take advantage of the single flush, and then move it down if a driver can improve performance with hardware assisted PMEM zeroing? _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
