I agree completely that the write() returning only means it's in the page cache. I agree completely that fsync() result is the only way to know your data is safe. Neither of those is what I, or the original poster (and what other posters in the past) on this subject are disputing or concerned about. The issue is, the write() call (in my case - read() in the original posters case) does NOT return. We both expect that a soft mounted NFS filesystem should propagate i/o errors back to the application when the retrans/timeo fails (without the filesystem being mounted sync). But that doesn't happen. And thus the application blocks indefinitely (or certainly longer than useful). Why repeated umount -f's eventually get the i/o error back to the caller and thus "unblock" the application, I'm not sure. But I'd guess it has something to do with having to get entries pending to be written off the queue until it eventually works its way back to the last write() that blocked b/c the cache was full (or something like that). -----Original Message----- From: Ric Wheeler <rwheeler@xxxxxxxxxx> Date: Thu, 06 Mar 2014 21:29:16 To: <bhawley@xxxxxxxxxxx>; Trond Myklebust<trond.myklebust@xxxxxxxxxxxxxxx> Cc: Andrew Martin<amartin@xxxxxxxxxxx>; Jim Rees<rees@xxxxxxxxx>; Brown Neil<neilb@xxxxxxx>; <linux-nfs-owner@xxxxxxxxxxxxxxx>; <linux-nfs@xxxxxxxxxxxxxxx> Subject: Re: Optimal NFS mount options to safely allow interrupts and timeouts on newer kernels On 03/06/2014 09:14 PM, Brian Hawley wrote: > Trond, > > In this case, it isn't fsync or close that are not getting the i/o error. It is the write(). > > And we check the return value of every i/o related command. Checking write() return status means we wrote to the page cache - you must also fsync() that file to push it out to the target. Do that when it counts, leaving data in the page cache until you actually need persistence and your performance should be reasonable. Doing it the safe way is not free, you will see a performance hit (less so if you can do batching, etc). ric > > We aren't using synchronous because the performance becomes abysmal. > > Repeated umount -f does eventually result in the i/o error getting propagated back to the write() call. I suspect the repeated umount -f's are working their way through blocks in the cache/queue and eventually we get back to the blocked write. > > As I mentioned previously, if we mount with sync or direct i/o type options, we will get the i/o error, but for performance reasons, this isn't an option. > > -----Original Message----- > From: Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx> > Date: Thu, 6 Mar 2014 14:06:24 > To: <bhawley@xxxxxxxxxxx> > Cc: Andrew Martin<amartin@xxxxxxxxxxx>; Jim Rees<rees@xxxxxxxxx>; Brown Neil<neilb@xxxxxxx>; <linux-nfs-owner@xxxxxxxxxxxxxxx>; <linux-nfs@xxxxxxxxxxxxxxx> > Subject: Re: Optimal NFS mount options to safely allow interrupts and timeouts on newer kernels > > > On Mar 6, 2014, at 14:00, Brian Hawley <bhawley@xxxxxxxxxxx> wrote: > >> Even with small timeo and retrans, you won't get i/o errors back to the reads/writes. That's been our experience anyway. > Read caching, and buffered writes mean that the I/O errors often do not occur during the read()/write() system call itself. > > We do try to propagate I/O errors back to the application as soon as the do occur, but if that application isn’t using synchronous I/O, and it isn’t checking the return values of fsync() or close(), then there is little the kernel can do... > >> With soft, you may end up with lost data (data that had already been written to the cache but not yet to the storage). You'd have that same issue with 'hard' too if it was your appliance that failed. If the appliance never comes back, those blocks can never be written. >> >> In your case though, you're not writing. >> >> >> -----Original Message----- >> From: Andrew Martin <amartin@xxxxxxxxxxx> >> Date: Thu, 6 Mar 2014 10:43:42 >> To: Jim Rees<rees@xxxxxxxxx> >> Cc: <bhawley@xxxxxxxxxxx>; NeilBrown<neilb@xxxxxxx>; <linux-nfs-owner@xxxxxxxxxxxxxxx>; <linux-nfs@xxxxxxxxxxxxxxx> >> Subject: Re: Optimal NFS mount options to safely allow interrupts and >> timeouts on newer kernels >> >>> From: "Jim Rees" <rees@xxxxxxxxx> >>> Andrew Martin wrote: >>> >>>> From: "Jim Rees" <rees@xxxxxxxxx> >>>> Given this is apache, I think if I were doing this I'd use >>>> ro,soft,intr,tcp >>>> and not try to write anything to nfs. >>> I was using tcp,bg,soft,intr when this problem occurred. I do not know if >>> apache was attempting to do a write or a read, but it seems that >>> tcp,soft,intr >>> was not sufficient to prevent the problem. >>> >>> I had the impression from your original message that you were not using >>> "soft" and were asking if it's safe to use it. Are you saying that even with >>> the "soft" option the apache gets stuck forever? >> Yes, even with soft, it gets stuck forever. I had been using tcp,bg,soft,intr >> when the problem occurred (on several ocassions), so my original question was >> if it would be safe to use a small timeo and retrans values to hopefully >> return I/O errors quickly to the application, rather than blocking forever >> (which causes the high load and inevitable reboot). It sounds like that isn't >> safe, but perhaps there is another way to resolve this problem? >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > _________________________________ > Trond Myklebust > Linux NFS client maintainer, PrimaryData > trond.myklebust@xxxxxxxxxxxxxxx > > N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±?û"žØ^n‡r¡ö¦zË?ëh™¨èÚ&¢ø ®G«?éh®(階ŠÝ¢j"?ú¶m§ÿï?êäz¹Þ–Šàþf£¢·hšˆ§~ˆmml== ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ¥Šwÿº{.nÇ+‰·¥Š{±þwìþ)í…æèw* jg¬±¨ ¶‰šŽŠÝ¢jÿ¾«þG«?éÿ¢¸ ¢·¦j:+v‰¨ŠwèjØm¶Ÿÿþø ¯ù ®w¥þŠàþf£¢·hš?â?úÿ†Ù¥
