I think you missed the context of my comment... Previous to this 4-year-old update, the writes were not sent with STABLE, this update forced that behavior. So, before then we sent an UNSTABLE write request. This would either give us back the UNSTABLE or FILE_SYNC response. My question is this: When the server sends back UNSTABLE, as a response to UNSTABLE, exactly what happens? By some chance is there a separate worker thread that occasionally sends COMMITs back to the server? The performance data we have would seem to bear that out. When we backed out the force of STABLE writes, the link times went back up and the reads stopped waiting on the cache flushes. If, as you say, this change had no impact on how the client actually performed these flushes, backing out the change would not have made links take 4x longer on Red Hat 5. All we did in our test was back out that change... I'm willing to discuss this issue in a conference call. I can send the bridge information to those who are interested, as well as the other people here in IBM I've been working with... At least one of them is a regular contributor -- Frank Filz... ================================================================= Brian Cowan Advisory Software Engineer ClearCase Customer Advocacy Group (CAG) Rational Software IBM Software Group 81 Hartwell Ave Lexington, MA Phone: 1.781.372.3580 Web: http://www.ibm.com/software/rational/support/ Please be sure to update your PMR using ESR at http://www-306.ibm.com/software/support/probsub.html or cc all correspondence to sw_support@xxxxxxxxxx to be sure your PMR is updated in case I am not available. From: Trond Myklebust <trond.myklebust@xxxxxxxxxx> To: Brian R Cowan/Cupertino/IBM@IBMUS Cc: Chuck Lever <chuck.lever@xxxxxxxxxx>, linux-nfs@xxxxxxxxxxxxxxx, linux-nfs-owner@xxxxxxxxxxxxxxx, Peter Staubach <staubach@xxxxxxxxxx> Date: 05/29/2009 02:31 PM Subject: Re: Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing On Fri, 2009-05-29 at 14:18 -0400, Brian R Cowan wrote: > There is a third option, that the COMMIT calls are not coming from the > same thread of execution that the write call is. The symptoms would seem > to bear that out. As would the fact that the performance degradation > occurs both when the server is Linux itself and when it is Solaris (any > NFSv3-supporting version). I'm not saying that Solaris is bug-free, but it > would be unusual if they are both broken the same way. The linux nfs FAQ > says: > > ----------------------- > * NFS Version 3 introduces the concept of "safe asynchronous writes." A > Version 3 client can specify that the server is allowed to reply before it > has saved the requested data to disk, permitting the server to gather > small NFS write operations into a single efficient disk write operation. A > Version 3 client can also specify that the data must be written to disk > before the server replies, just like a Version 2 write. The client > specifies the type of write by setting the stable_how field in the > arguments of each write operation to UNSTABLE to request a safe > asynchronous write, and FILE_SYNC for an NFS Version 2 style write. > > Servers indicate whether the requested data is permanently stored by > setting a corresponding field in the response to each NFS write operation. > A server can respond to an UNSTABLE write request with an UNSTABLE reply > or a FILE_SYNC reply, depending on whether or not the requested data > resides on permanent storage yet. An NFS protocol-compliant server must > respond to a FILE_SYNC request only with a FILE_SYNC reply. > > Clients ensure that data that was written using a safe asynchronous write > has been written onto permanent storage using a new operation available in > Version 3 called a COMMIT. Servers do not send a response to a COMMIT > operation until all data specified in the request has been written to > permanent storage. NFS Version 3 clients must protect buffered data that > has been written using a safe asynchronous write but not yet committed. If > a server reboots before a client has sent an appropriate COMMIT, the > server can reply to the eventual COMMIT request in a way that forces the > client to resend the original write operation. Version 3 clients use > COMMIT operations when flushing safe asynchronous writes to the server > during a close(2) or fsync(2) system call, or when encountering memory > pressure. > ----------------------- > > Now, what happens in the client when the server cones back with the > UNSTABLE reply? The server cannot reply with an UNSTABLE reply to a stable write request. See above. As for your assertion that the COMMIT comes from some other thread of execution. I don't see how that can change anything. Some thread, somewhere has to wait for that COMMIT to complete. If it isn't your application, then the same burden falls on another application or the pdflush thread. While that may feel more interactive to you, it still means that you are making the server + some local process do more work (extra RPC round trip) for no good reason. Trond > ================================================================= > Brian Cowan > Advisory Software Engineer > ClearCase Customer Advocacy Group (CAG) > Rational Software > IBM Software Group > 81 Hartwell Ave > Lexington, MA > > Phone: 1.781.372.3580 > Web: http://www.ibm.com/software/rational/support/ > > > Please be sure to update your PMR using ESR at > http://www-306.ibm.com/software/support/probsub.html or cc all > correspondence to sw_support@xxxxxxxxxx to be sure your PMR is updated in > case I am not available. > > > > From: > Trond Myklebust <trond.myklebust@xxxxxxxxxx> > To: > Brian R Cowan/Cupertino/IBM@IBMUS > Cc: > Chuck Lever <chuck.lever@xxxxxxxxxx>, linux-nfs@xxxxxxxxxxxxxxx, > linux-nfs-owner@xxxxxxxxxxxxxxx, Peter Staubach <staubach@xxxxxxxxxx> > Date: > 05/29/2009 02:07 PM > Subject: > Re: Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing > > > > On Fri, 2009-05-29 at 13:55 -0400, Brian R Cowan wrote: > > > Yes. If the page is dirty, but not up to date, then it needs to be > > > cleaned before you can overwrite the contents with the results of a > > > fresh read. > > > That means flushing the data to disk... Which again means doing either > a > > > stable write or an unstable write+commit. The former is more efficient > > > that the latter, 'cos it accomplishes the exact same work in a single > > > RPC call. > > > > I suspect that the COMMIT RPC's are done somewhere other than in the > flush > > itself. If the "write + commit" operation was happening in the that > exact > > matter, then the change in the git at the beginning of this thread > *would > > not have impacted client performance*. I can demonstrate -- at will -- > > that it does impact performance. So, there is something that keeps track > > > of the number of writes and issues the commits without slowing down the > > application. This git change bypasses that and degrades the linker > > performance. > > If the server gives slower performance for a single stable write, vs. > the same unstable write + commit, then you are demonstrating that the > server is seriously _broken_. > > The only other explanation, is if the client prior to that patch being > applied was somehow failing to send out the COMMIT. If so, then the > client was broken, and the patch is a fix that results in correct > behaviour. That would mean that the rest of the client flush code is > probably still broken, but at least the nfs_wb_page() is now correct. > > Those are the only 2 options. > > Trond > > > -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
