> The filesystem can still choose to do that for O_DIRECT if it wants - look at > all the filesystems that have a "fall back to buffered IO because this is too > hard to implement in the direct Io path". Yes, I agree that the filesystem can still decide to buffer IO even with O_DIRECT, but the application's intent is that the effects of caching are minimized. Whereas with O_CONCURRENT_WRITE, the intent is to maximize caching. > IOWs, you've got another set of custom userspace APIs that are needed to make > proper use of this open flag? Yes and no. Applications can make ioctls to the filesystem to query or set layout details but don't have to. Directory level default layout attributes can be set up by an admin to meet the requirements of the application. > > In panfs, a well behaved CONCURRENT_WRITE application will consider > > the file's layout on storage. Access from different machines will not > > overlap within the same RAID stripe so as not to cause distributed > > stripe lock contention. Writes to the file that are page aligned can > > be cached and the filesystem can aggregate multiple such writes before > > writing out to storage. Conversely, a CONCURRENT_WRITE application > > that ends up colliding on the same stripe will see worse performance. > > Non page aligned writes are treated by panfs as write-through and > > non-cachable, as the filesystem will have to assume that the region of > > the page that is untouched by this machine might in fact be written to > > on another machine. Caching such a page and writing it out later might lead to data corruption. > That seems to fit the expected behaviour of O_DIRECT pretty damn closely - if > the app doesn't do correctly aligned and sized IO then performance is going to > suck, and if the apps doesn't serialize access to the file correctly it can and > will corrupt data in the file.... I make the same case as above, that O_DIRECT and O_CONCURRENT_WRITE have opposite intents with respect to caching. Our filesystem handles them differently, so we need to distinguish between the two. > > The benefit of CONCURRENT_WRITE is that unlike O_DIRECT, the > > application does not have to implement any caching to see good performance. > Sure, but it has to be aware of layout and where/how it can write, which is > exactly the same constraints that local filesystems place on O_DIRECT access. > Not convinced. The use case fits pretty neatly into expected O_DIRECT semantics > and behaviour, IMO. I'd like to make a slight adjustment to my proposal. The HPC community had talked about extensions to POSIX to include O_LAZY as a way for filesystems to relax data coherency requirements. There is code in the ceph filesystem that uses that flag if defined. Can we get O_LAZY defined? HEC POSIX extension: http://www.pdsw.org/pdsw06/resources/hec-posix-extensions-sc2006-workshop.pdf Ceph usage of O_LAZY: https://github.com/ceph/ceph-client/blob/1e37f2f84680fa7f8394fd444b6928e334495ccc/net/ceph/ceph_fs.c#L78 Regards, Rodney
