On Fri, Aug 21, 2009 at 04:24:59PM +0100, Jamie Lokier wrote: > In measurements I've done, disabling a disk's write cache results in > much slower ext3 filesystem writes than using barriers. Others report > similar results. This is with disks that don't have NCQ; good NCQ may > be better. On a scsi disk and a SATA SSD with NCQ I get different results. Most worksloads, in particular metadata-intensive ones and large streaming writes are noticably better just turning off the write cache. The only onces that benefit from it are relatively small writes witout O_SYNC or much fsyncs. This is however using XFS which tends to issue much more barriers than ext3. > Using FUA for all writes should be equivalent to writing with write > cache disabled. > > A journalling filesystem or database tends to write like this: > > (guest) WRITE > (guest) WRITE > (guest) WRITE > (guest) WRITE > (guest) WRITE > (guest) CACHE FLUSH > (guest) WRITE > (guest) CACHE FLUSH > (guest) WRITE > (guest) WRITE > (guest) WRITE In the optimal case, yeah. > Assuming that WRITE FUA is equivalent to disabling write cache, we may > expect the WRITE FUA version to run much slower than the CACHE FLUSH > version. For a workload that only does FUA writes, yeah. That is however the use case for virtual machines. As I'm looking into those issues I will run some benchmarks comparing both variants. > It's also too weak, of course, on drives which don't support FUA. > Then you have to use CACHE FLUSH anyway, so the code should support > that (or disable the write cache entirely, which also performs badly). > If you don't handle drives without FUA, then you're back to "integrity > sometimes, user must check type of hardware", which is something we're > trying to get away from. Integrity should not be a surprise when the > application requests it. As mentioned in the previous mails FUA would only be an optimization (if it ends up helping) we do need to support the cache flush case. > > I thought about this alot . It would be sensible to only require > > the FUA semantics if O_SYNC is specified. But from looking around at > > users of O_DIRECT no one seems to actually specify O_SYNC with it. > > O_DIRECT with true POSIX O_SYNC is a bad idea, because it flushes > inode metadata (like mtime) too. O_DIRECT|O_DSYNC is better. O_SYNC above is the Linux O_SYNC aka Posix O_DYNC. > O_DIRECT without O_SYNC, O_DSYNC, fsync or fdatasync is asking for > integrity problems when direct writes are converted to buffered writes > - which applies to all or nearly all OSes according to their > documentation (I've read a lot of them). It did not happen on IRIX where O_DIRECT originated that did not happen, neither does it happen on Linux when using XFS. Then again at least on Linux we provide O_SYNC (that is Linux O_SYNC, aka Posix O_DYSC) semantics for that case. > Imho, integrity should not be something which depends on the user > knowing the details of their hardware to decide application > configuration options - at least, not out of the box. That is what I meant. Only doing cache flushes/FUA for O_DIRECT|O_DSYNC is not what users naively expect. And the wording in hour manpages also suggests this behaviour, although it is not entirely clear: O_DIRECT (Since Linux 2.4.10) Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user space buffers. The I/O is synchronous, that is, at the completion of a read(2) or write(2), data is guaranteed to have been transferred. See NOTES below forfurther discussion. (And yeah, the whole wording is horrible, I will send an update once we've sorted out the semantics, including caveats about older kernels) > > And on Linux where O_SYNC really means O_DYSNC that's pretty sensible - > > if O_DIRECT bypasses the filesystem cache there is nothing else > > left to sync for a non-extending write. > > Oh, O_SYNC means O_DSYNC? I thought it was the other way around. > Ugh, how messy. Yes. Except when using XFS and using the "osyncisosync" mount option :) > > The fallback was a relatively recent addition to the O_DIRECT semantics > > for broken filesystems that can't handle holes very well. Fortunately > > enough we do force O_SYNC (that is Linux O_SYNC aka Posix O_DSYNC) > > semantics for that already. > > Ok, so you're saying there's no _harm_ in specifying O_DSYNC with > O_DIRECT either? :-) No. In the generic code and filesystems I looked at it simply has no effect at all. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
