Hi Derick,
did you test some different config values for ggaoed? (man ggaoed.conf)
I would be interested in values for
* queue-length
* direct-io
* ring-buffer-size
Maybe raising the values from the default ones would be more suitable for a 10G Ethernet.
Thanks
Lars
Wed, 11 Sep 2013 15:59:06 +0200
Derick Swanepoel <dswanepoel@xxxxxxxxx> ==> Ed Cashin <ecashin@xxxxxxxxxx> :
> On 06 Sep 2013, at 4:10 PM, Ed Cashin <ecashin@xxxxxxxxxx> wrote:
>
> > I don't have a lot of experience with the other non-Coraid AoE targets that are out there, but you might check whether one of them that's oriented more toward performance could be useful to you.
> >
> > That said, while checking the vblade README for the design goals, I noticed that it advertises a capacity for 16 outstanding commands. If you want to try some tuning, you could adjust Bufcount in dat.h and then make sure your settings in /proc are sufficient to allow the kernel to buffer 16 writes. (Read commands are small.)
>
> I ran vblade with -b to increase the buffer count and it improved performance quite a bit, but it's now maxing out the CPU. I found that bufcount above 64 showed little or no improvement. There is however a big difference between using normal IO (dd with conv=fdatasync) and direct IO (dd with {o,i}flag=direct) on the initiator:
>
> Test MB/s CPU AvgPktSz Direct MB/s CPU AvgPktSz
> Disk Read 538 95% 2083 623 67% 4333
> Disk Write 443 97% 2095 582 75% 4345
> Ramdisk Read 655 97% 2083 778 69% 4333
> Ramdisk Write 424 100% 2095 624 81% 4345
>
> AvgPktSz shows the average packet size as measured by nettop. Wireshark confirms that "normal" IO generates 4132-byte packets while direct IO results in 8740-byte packets. I know Q 5.23 of the Coraid Linux FAQ says that AoE devices with an odd number of sectors result in 512-byte IO jobs, but mine have even sector counts. This is probably not the best way to benchmark but when I create a filesystem on top of my AoE device I get awful performance (50 MB/s) so there are obviously alignment issues.
>
> Either way, looking at the CPU usage it's clear that vblade isn't going reach 10 Gb/s.
>
> I also tried other Linux targets:
>
> kvblade: Doesn't compile against kernel 3.x.
>
> ggaoed: About 25% slower than vblade:
>
> Test MB/s CPU Direct MB/s CPU
> Disk Read 446 71% 446 51%
> Disk Write 355 63% 557 56%
> Ramdisk Read 531 91% 627 67%
> Ramdisk Write 399 85% 602 73%
>
> qaoed: 25 - 50% slower than vblade:
>
> Test MB/s CPU Direct MB/s CPU
> Disk Read 282 77% 473 73%
> Disk Write 259 85% 465 73%
> Ramisk Read 291 99% 521 69%
> Ramdisk Write 261 75% 467 75%
>
> Unless I'm missing any further tuning options, none of the open source Linux AoE targets seem to be suitable for a 10 Gb/s SAN.
>
> Regards,
> Derick
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> Aoetools-discuss mailing list
> Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx
> https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstrasse 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
_______________________________________________
Aoetools-discuss mailing list
Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
