Here's benchmarks of RAID5x4 vs RAID10x4 on a Dell Perc5/I with 300 GB
10k RPM SAS drives. I know these are bonnie 1.9 instead of the older
version, but maybe it might still make for useful analysis of RAID5 vs.
RAID10.
Also, unfortunately I don't have the exact numbers, but RAID10x6
performed really poorly on the sequential IO (dd) tests- worse than the
4 disk RAID5, something around 120 MB/s. I'm currently running the
system as a RAID5x6, but would like to go back and do some further
testing if I get the chance to tear the box down again.
These tests were run on FreeBSD 6.1 amd64 RELEASE with UFS + soft
updates. For comparison, the dd for RAID5x6 was 255 MB/s so I think the
extra disks really help out with RAID5 write performance, as Scott
pointed out. (I'm using a 128k stripe size with a 256MB writeback
cache).
Personally, I'm not yet convinced that RAID10 offers dramatically better
performance than RAID5 for 6 disks (at least on the Dell PERC
controller), and available storgae is a significant factor for my
particular application. But I do feel the need to do more testing, so
any suggestions are appreciated. (and yes, I'll be using bonnie 1.03 in
the future, along with pgbench).
------ RAID5x4
# /usr/local/sbin/bonnie++ -d bonnie -s 1000:8k -u root
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
1000M 587 99 158889 30 127859 32 1005 99 824399 99
+++++ +++
Latency 14216us 181ms 48765us 56241us 1687us
47997us
Version 1.93c ------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
Latency 40365us 25us 35us 20030us 36us
52us
1.93c,1.93c,beast.corp.lumeta.com,1,1155204369,1000M,,587,99,158889,30,1
27859,32,1005,99,824399,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,++
+,+++++,+++,+++++,+++,+++++,+++,14216us,181ms,48765us,56241us,1687us,479
97us,40365us,25us,35us,20030us,36us,52us
# time bash -c "(dd if=/dev/zero of=bigfile count=125000 bs=8k && sync)"
125000+0 records in
125000+0 records out
1024000000 bytes transferred in 6.375067 secs (160625763 bytes/sec)
0.037u 1.669s 0:06.42 26.3% 29+211k 30+7861io 0pf+0w
------ RAID10 x 4
bash-2.05b$ bonnie++ -d bonnie -s 1000:8k
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
1000M 585 99 21705 4 28560 9 1004 99 812997 98 5436
454
Latency 14181us 81364us 50256us 57720us 1671us
1059ms
Version 1.93c ------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 4712 10 +++++ +++ +++++ +++ 4674 10 +++++ +++
+++++ +++
Latency 807ms 21us 36us 804ms 110us
36us
1.93c,1.93c,beast.corp.lumeta.com,1,1155207445,1000M,,585,99,21705,4,285
60,9,1004,99,812997,98,5436,454,16,,,,,4712,10,+++++,+++,+++++,+++,4674,
10,+++++,+++,+++++,+++,14181us,81364us,50256us,57720us,1671us,1059ms,807
ms,21us,36us,804ms,110us,36us
bash-2.05b$ time bash -c "(dd if=/dev/zero of=bigfile count=125000 bs=8k
&& sync)"
125000+0 records in
125000+0 records out
1024000000 bytes transferred in 45.565848 secs (22472971 bytes/sec)
- Bucky
-----Original Message-----
From: Scott Marlowe [mailto:smarlowe@xxxxxxxxxxxxxxxxx]
Sent: Thursday, August 24, 2006 3:38 PM
To: Merlin Moncure
Cc: Jeff Davis; Bucky Jordan; pgsql-performance@xxxxxxxxxxxxxx
Subject: Re: [PERFORM] PowerEdge 2950 questions
On Thu, 2006-08-24 at 13:57, Merlin Moncure wrote:
> On 8/24/06, Jeff Davis <pgsql@xxxxxxxxxxx> wrote:
> > On Thu, 2006-08-24 at 09:21 -0400, Merlin Moncure wrote:
> > > On 8/22/06, Jeff Davis <pgsql@xxxxxxxxxxx> wrote:
> > > > On Tue, 2006-08-22 at 17:56 -0400, Bucky Jordan wrote:
> > > it's not the parity, it's the seeking. Raid 5 gives you great
> > > sequential i/o but random is often not much better than a single
> > > drive. Actually it's the '1' in raid 10 that plays the biggest
role
> > > in optimizing seeks on an ideal raid controller. Calculating
parity
> > > was boring 20 years ago as it inolves one of the fastest
operations in
> > > computing, namely xor. :)
> >
> > Here's the explanation I got: If you do a write on RAID 5 to
something
> > that is not in the RAID controllers cache, it needs to do a read
first
> > in order to properly recalculate the parity for the write.
>
> it's worse than that. if you need to read something that is not in
> the o/s cache, all the disks except for one need to be sent to a
> physical location in order to get the data.
Ummmm. No. Not in my experience. If you need to read something that's
significantly larger than your stripe size, then yes, you'd need to do
that. With typical RAID 5 stripe sizes of 64k to 256k, you could read 8
to 32 PostgreSQL 8k blocks from a single disk before having to move the
heads on the next disk to get the next part of data. A RAID 5, being
read, acts much like a RAID 0 with n-1 disks.
It's the writes that kill performance, since you've got to read two
disks and write two disks for every write, at a minimum. This is why
small RAID 5 arrays bottleneck so quickly. a 4 disk RAID 4 with two
writing threads is likely already starting to thrash.
Or did you mean something else by that?
