On Thu, 2012-02-09 at 16:51 +1100, Harshula wrote:
> Hi Trond,
>
> Thanks for the reply. Could you please elaborate on the subtleties
> involved that require an application to be rewritten if forcedirectio
> mount option was available?
Firstly, we don't support O_DIRECT+O_APPEND (since the NFS protocol
itself doesn't support atomic appends), so that would break a bunch of
applications.
Secondly, uncached I/O means that read() and write() requests need to be
serialised by the application itself, since there are no atomicity or
ordering guarantees at the VFS, NFS or RPC call level. Normally, the
page cache services read() requests if there are outstanding writes, and
so provides the atomicity guarantees that POSIX requires.
IOW: if a write() occurs while you are reading, the application may end
up retrieving part of the old data, and part of the new data instead of
either one or the other.
IOW: your application still needs to be aware of the fact that it is
using O_DIRECT, and you are better of adding explicit support for it
rather than hacky cluges such as a forcedirectio option.
> On Thu, 2012-02-09 at 04:12 +0000, Myklebust, Trond wrote:
> > On Thu, 2012-02-09 at 14:56 +1100, Harshula wrote:
> > >
> > > The "sync" option, depending on the NFS server, may impact the NFS
> > > server's performance when serving many NFS clients. But still worth a
> > > try.
> >
> > What on earth makes you think that directio would be any different?
>
> Like I said, sync is still worth a try. I will do O_DIRECT Vs sync mount
> option runs and see what the numbers look like. A while back the numbers
> for cached Vs direct small random writes showed as the number of threads
> increased the cached performance fell well below direct performance. In
> this case I'll be looking at large streaming writes, so completely
> different scenario, but I'd like to verify the numbers first.
>
> Just to be clear, I am not disagreeing with you. "sync" maybe sufficient
> for the scenario I described earlier.
>
> > If
> > your performance requirements can't cope with 'sync', then they sure as
> > hell won't deal well with 'fsc'.
>
> "fsc"?
>
> > Directio is _synchronous_ just like 'sync'. The big difference is that
> > with 'sync' then at least those reads are still cached.
>
> There's another scenario, which we talked about a while back, where the
> cached async reads of a slowly growing file (tail) was spitting out
> non-exist NULLs to user space. The forcedirectio mount option should
> prevent that. Furthermore, the "sync" mount option will not help anymore
> because you removed nfs_readpage_sync().
No. See the points about O_APPEND and serialisation of read() and
write() above. You may still end up seeing NUL characters (and indeed
worse forms of corruption).
> > > The other hack that seems to work is periodically triggering an
> > > nfs_getattr(), via ls -l, to force the dirty pages to be flushed to the
> > > NFS server. Not exactly elegant ...
> >
> > ????????????????????????????????
>
> int nfs_getattr(struct vfsmount *mnt, struct dentry *dentry, struct kstat *stat)
> {
> struct inode *inode = dentry->d_inode;
> int need_atime = NFS_I(inode)->cache_validity & NFS_INO_INVALID_ATIME;
> int err;
>
> /* Flush out writes to the server in order to update c/mtime. */
> if (S_ISREG(inode->i_mode)) {
> err = filemap_write_and_wait(inode->i_mapping);
> if (err)
> goto out;
> }
I'm aware of that code. The point is that '-osync' does that for free.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@xxxxxxxxxx
www.netapp.com
��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
