Re: Software RAID5 Horrible Write Speed On 3ware Controller!!

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On Wed, 18 Jul 2007, Giuseppe Ghibò wrote:

Justin Piszcz ha scritto:

I recently got a chance to test SW RAID5 using 750GB disks (10) in a RAID5 on a 3ware card, model no: 9550SXU-12

The bottom line is the controller is doing some weird caching with writes on SW RAID5 which makes it not worth using.

Recall, with SW RAID5 using regular SATA cards with (mind you) 10 raptors:
write: 464MB/s
read: 627MB/s

Yes, these drives are different, 7200RPM 750GB drives, but write should not be 50-102MB/s as shown below.

First, lets test RAW performance of these 10 drives:

Create RAID 0 with 10 750GB Drives:
# mdadm /dev/md0 --create --level=0 -n 10 /dev/sd[bcdefghjik]1
mdadm: array /dev/md0 started.

--> XFS: (xfs default options, no optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 22.459 seconds, 478 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 28.7843 seconds, 373 MB/s

--> XFS: (xfs default options, enabled md-raid read optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 22.9623 seconds, 468 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 17.7328 seconds, 606 MB/s

Software RAID 5 on a real HW raid controller over 10 750GB disks JBOD:

UltraDense-AS-3ware-R5-9-disks,16G,50676,89,96019,34,46379,9,60267,99,501098,56,248.5,0,16:100000:16/64,240,3,21959,84,1109,10,286,4,22923,91,544,6 UltraDense-AS-3ware-R5-9-disks,16G,49983,88,96902,37,47951,10,59002,99,529121,60,210.3,0,16:100000:16/64,250,3,25506,98,1163,10,268,3,18003,71,772,8 UltraDense-AS-3ware-R5-9-disks,16G,49811,87,95759,35,48214,10,60153,99,538559,61,276.8,0,16:100000:16/64,233,3,25514,97,1100,9,279,3,21398,84,839,9 Write seems significantly impacted, where read is fine, the HW RAID controller cache must be doing something strange:

--> XFS SW RAID 5: (xfs noatime only, enabled md-raid read optimizations)
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 105.178 seconds, 102 MB/s
# dd if=10gb of=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 17.4893 seconds, 614 MB/s

-----

I am sure one of your questions is, well, why use SW RAID5 on the controller? Because SW RAID5 is usually much faster than HW RAID5, at least in my tests:

Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
------------------------------------------------------------------------
c0    9550SXU-12   12      12       3       0        1       4       -

Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC

------------------------------------------------------------------------------ u0 RAID-1 OK - - 698.481 ON ON OFF
u1    RAID-5    OK             -      64K     5587.85   ON     OFF      OFF
u2    SPARE     OK             -      -       698.629   -      OFF      -

--> XFS:
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 74.5648 seconds, 144 MB/s

--> JFS:
# dd if=/dev/zero of=10gb bs=1M count=10240
10737418240 bytes (11 GB) copied, 108.631 seconds, 98.8 MB/s

The controller is set to performance, and this is nothing close to performance.

How much is your RAM size? Is the size you tried (10G) at
least twice the size of the RAM seen by the OS? What
are the values returned by hdparm -t /dev/sda (it test only raw reading
speed)?

Total: 4GB of ram-- I am using the array for other things right now, did not get a chance to run that.


In RAID0, the controller is ok with the disks JBOD, but I cannot recommend buying a controller (12,16,24 port) for Linux SW RAID 5.

Its too bad that there are no regular > 4 port SATA PCI-e controllers out there.

Justin.


Indeed not exists for PCI-e but Oden has spotted this PCI-X card
(which is around 97$), based on marvell chipset:

http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

which can be used on motherboard with PCI-X slot (the ASUS M2N32 WS Professional AM2, or the ASUS P5W64-WS-PRO, both are for consumer desktop and have 2 PCI-X slots) though probably if you have either one of that mobo you already have at least 10 onboard SATA connectors.
Indeed, wish there was a PCI-e version!


Bye
Giuseppe.

[Index of Archives]     [Linux RAID Wiki]     [ATA RAID]     [Linux SCSI Target Infrastructure]     [Linux Block]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device Mapper]     [Device Mapper Cryptographics]     [Kernel]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Yosemite Forum]


  Powered by Linux