Hi Stefan, Christoph is bang on right. To further elaborate upon this, here is what is happening in the above case : By using DIRECT, SYNC/DSYNC flags on a block device (i.e. bypassing the file-systems layer), essentially you are enforcing a CMD_FLUSH on each I/O command sent to the disk. This is by design of the block-device driver in the Linux kernel. This severely degrades the performance. A detailed walk-through of the various I/O scenarios is available at thecodeartist.blogspot.com/2012/08/hdd-filesystems-osync.html Note that SYNC/DSYNC on a filesystem(eg. ext2/3/4) does NOT issue a CMD_FLUSH. The "SYNC" via filesystem, simply guarantees that the data is sent to the disk and not really flushed to the disk. It will continue to reside in the internal cache on the disk, waiting to be written to the disk platter in a optimum manner (bunch of writes re-ordered to be sequential on-disk and clubbed together in one go). This can affect performance to a large extent on modern HDDs with NCQ support (CMD_FLUSH simply cancels all performance benefits of NCQ). In case of SSDs, the huge IOPS number for the disk (40,000 in case of Crucial M4) is again typically observed with write-cache enabled. For Crucial M4 SSDs, http://www.crucial.com/pdf/tech_specs-letter_crucial_m4_ssd_v3-11-11_online.pdf Footnote1 - "Typical I/O performance numbers as measured using Iometer with a queue depth of 32 and write cache enabled. Iometer measurements are performed on a 8GB span. 4k transfers used for Read/Write latency values." To simply disable this behaviour and make the SYNC/DSYNC behaviour and performance on raw block-device I/O resemble the standard filesystem I/O you may want to apply the following patch to your kernel - https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba The above patch simply disables the CMD_FLUSH command support even on disks that claim to support it. regards ChinmayVS On Wed, Nov 20, 2013 at 6:24 PM, Christoph Hellwig <hch@xxxxxxxxxxxxx> wrote: > On Wed, Nov 20, 2013 at 01:12:43PM +0100, Stefan Priebe - Profihost AG wrote: >> Can anyone explain to me why O_DSYNC for my app on linux is so slow? > > Because FreeBSD ignores O_DSYNC on block devices, it never sends a FLUSH > to the device. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
