J. Bruce Fields wrote: > On Mon, Nov 09, 2009 at 08:30:44PM +0100, Jesper Krogh wrote: >> When a lot (~60 all on 1GbitE) of NFS clients are hitting an NFS server >> that has an 10GbitE NIC sitting on it I'm seeing high IO-wait load >> (>50%) and load number over 100 on the server. This is a change since >> 2.6.29 where the IO-wait load under similar workload was less than 10%. >> >> The system has 16 Opteron cores. >> >> All data the NFS-clients are reading are "memory recident" since they >> are all reading off the same 10GB of data and the server has 32GB of >> main memory dedicated to nothing else than serving NFS. >> >> A snapshot of top looks like this: >> http://krogh.cc/~jesper/top-hest-2.6.31.txt >> >> The load is generally alot higher than on 2.6.29 and it "explodes" to >> over 100 when a few processes begin utillizing the disk while serving >> files over NFS. "dstat" reports a read-out of 10-20MB/s from disk which >> is close to what I'd expect. and the system delivers around 600-800MB/s >> over the NIC in this workload. > > Is that the bandwidth you get with 2.6.31, with 2.6.29, or with both? Without being able to be fully accurate, I have a strong feeling that the comparative numbers on 2.6.29 were more around 800-1000MB/s. But this isn't based on any measurements so dont put too much into it. I'll try to make up something that I can use for testing over multiple kernel-versions. > Are you just noticing a change in the statistics, or are there concrete > changes in the performance of the server? Interactivity on the console is alot worse. Still usable, but top takes ~5s to start up on 2.6.31 where I didn't remember any lags on 2.6.29 (so less than 2s). >> Sorry that I cannot be more specific, I can answer questions on a >> running 2.6.31 kernel, but I cannot reboot the system back to 2.6.29 >> just to test since the system is "in production". I tried 2.6.30 and it >> has the same pattern as 2.6.31, so based on that fragile evidence the >> change should be found in between 2.6.29 and 2.6.30. I hope a "wague" >> report is better than none. > > Can you test whether this helps? I'll schedule testing.. -- Jesper -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
