The test was:
Iozone -I 0 -r 64k -s 2G -w
Actually, I started with a much larger test suite, and I'm actually most interested in small random write latency, rather than overall throughput, but the sequential writes test make the file system vs log white behaviour obvious via iostat. It also seems that in my environment, the disk write latency actually limits throughput slightly, presumably because the delay in syncing each write delays responses back to the client?
My initial test was with a with a pair of striped drives capable of about 130-160MB/sec on local writes (they're 5400rpm laptop drives).
Some slight tuning (removing the rsize/wsize parameters, and increasing the [r|w]mem_default/max parameters gets the NFS speed up to about 80mb/sec on writes.
Doing the exact same test on a filesystem backed onto some SSD's gets me up to about 100-105MB sec, although the write latency on these is much lower than the 5400 disks.
I'll get back to you on the ext3 mailing list responses, I'm going to redo my tests on ext4 first.
Regards,
Tristan
Tristan Ball - Hosted Services Manager VIC
Pronto Hosted Services
20 Lakeside Drive, Burwood East, VIC 3151, Australia
Phone: +61 3 9887 7770 | Email: tristanb@xxxxxxxxxxxxx
Mobile: +61 408 397 473
www.pronto.com.au
---Legal Notice---
The email message and any attachments are confidential and subject to copyright. If you are not the intended recipient, any use, interference with, disclosure or copying of this material is unauthorised and prohibited. No part may be reproduced, adapted or transmitted without the written permission of the copyright owner. If you have received this email in error, please immediately advise the sender by return email and delete the message from your system. Before opening or using attachments, check for viruses and defects. Our liability is limited to re-supplying any affected attachments.
-----Original Message-----
From: J. Bruce Fields [mailto:bfields@xxxxxxxxxxxx]
Sent: Wednesday, 8 June 2011 9:23 AM
To: Wendy Cheng
Cc: Tristan Ball; linux-nfs@xxxxxxxxxxxxxxx
Subject: Re: NFS Sync with External Journal
On Mon, Jun 06, 2011 at 01:12:51PM -0700, Wendy Cheng wrote:
> You'll probably get better answer(s) from ext3 user mailing list ....
> it is more about how journaling works for the specific file system.
Yes, though leave linux-nfs on the cc: as I'd be interested what you find out.
> In ext3 case, I believe "sync" forces data getting flushed to the file
> system *regardless* which journal mode is chosen. Using an external
> journal device, particularly on SSD, does help but the performance
> gain is limited by the amount of data that needs to be written into
> the file system itself.
> > /etc/exports:
> > /plain *(rw,async,no_subtree_check,no_root_squash)
> > /split *(rw,async,no_subtree_check,no_root_squash) # (FS
> > with external Journal)
> >
> > Client mounts were done simply with -o 'rw,rsize=32768,wsize=32768'
> >
> > Benchmark results:
> > Plain Ext3, data=ordered export=sync, write speed 56-62MB/sec Split
> > Ext3, data=journal export=sync, write speed = 46-50MB/sec
> >
> > For reference:
> > Plain Ext3, data=ordered export=async, write speed 111MB/sec Split
> > Ext3, data=journal export=async, write speed 110MB/sec
What exactly is your test?
For sufficiently large sequential writes, I wouldn't actually have expected sync vs. async to make much difference: eventually you're limited by the drive speed (I'm assuming your drive does ~60MB/s write througput?). And individual writes (for NFS v3 and higher) aren't necessarily required to be synchronous.
A better test would be creating or destroying a bunch of small files, as create and unlink are synchronous (the nfs server won't return, in the sync case, before each create and unlink actually hits the disk).
--b.
ÿô.nlj·Ÿ®‰†+%ŠË±é¥Šwÿº{.nlj·¥Š{±þwìíè jg¬±¨ ¶‰šŽŠÝjÿ¾«þG«é¸ ¢·¦j:+v‰¨Šwèm¶Ÿÿþø®w¥þŠà£¢·hšâÿ†Ù
