On Tue, Nov 3, 2009 at 5:11 PM, Michael Evans <mjevans1983@xxxxxxxxx> wrote: > On Tue, Nov 3, 2009 at 11:29 AM, Chris Worley <worleys@xxxxxxxxx> wrote: >> On Tue, Nov 3, 2009 at 9:31 AM, Michael Evans <mjevans1983@xxxxxxxxx> wrote: >>> Is it possible the data-set you are testing with is too small? Modern >>> drives have caches of around 32MB, your test might only be executing >>> within the drive's caches. >> >> Cache is disabled on the drives, and disabled on the system by using >> O_DIRECT on the benchmark. I am seeing proper behavior using >> synchronous writes, but that slows down performance too much. >> > > I'm not familiar with O_DIRECT so I looked it up: > http://www.kernel.org/doc/man-pages/online/pages/man2/open.2.html > > It -appears- that O_DIRECT merely attempts to efficiently map > transfers as full blocks/allocation units. I see no mention, within > the context of O_DIRECT of buffering or not (and presume it -is- > buffered by default). > > This page provides additional detail: > http://www.ukuug.org/events/linux2001/papers/html/AArcangeli-o_direct.html > > It seems that O_DIRECT disables any kernel-side caches for operations, > but does not modify any caching that might be performed by the > underlying block devices. O_DIRECT might not even be applicable with > an MD/DM device. > > > O_DIRECT (Since Linux 2.4.10) > Try to minimize cache effects of the I/O to and from > this file. In > general this will degrade performance, but it is useful in special > situations, such as when applications do their own > caching. File I/O > is done directly to/from user space buffers. The > O_DIRECT flag on its > own makes at an effort to transfer data synchronously, > but does not > give the guarantees of the O_SYNC that data and > necessary metadata are > transferred. To guarantee synchronous I/O the O_SYNC > must be used in > addition to O_DIRECT. See NOTES below for further discussion. > > A semantically similar (but deprecated) interface for > block devices is > described in raw(8). > > As that quote from the manual states, to get the effect you desire you > must use -both- O_DIRECT and O_SYNC. > So, you're saying that the MD layer is doing it's own buffering? Are you sure? With the system cache disabled and the drive (and block device driver) cache disabled, there should be no reason to require synchronous I/O, unless, as you suggest, the MD layer is broken. You're saying that both O_DIRECT and O_SYNC must be used to disable cache effects. Why then are there two separate flags and not just one? Synchronous is a very different behavior that is not necessary for this test and put additional requirements that are not needed for this test. Chris -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
