Most of our servers currently run CentOS/RedHat, and we have little =
experience with Solaris, but we're not opposed to Solaris if there's a =
compelling reason to switch. For example, it sounds like zfs snapshots =
may have a lighter performance penalty than LVM snapshots. We've heard =
that just using LVM (even without active snapshots) imposes a maximum =
sequential I/O rate of around 600 MB/s (although we haven't yet reached =
this limit experimentally).
By the way, we've also heard that Solaris is "more stable" under heavy I/O =
load than Linux. Have any of you experienced this? It's hard to put much =
stock in such a blanket statement, but naturally we don't want to =
introduce instabilities.
Thanks in advance for your thoughts!
For reference:
Our database cluster will be 3-6 TB in size. The Postgres installation =
will be 8.1 (at least initially), compiled to use 32 KB blocks (rather =
than 8 KB). The workload will be predominantly OLAP. The Sunfire X4500 =
has 2 dual-core Opterons, 16 GB RAM, 48 SATA disks (500 GB/disk * 48 =3D =
24 TB raw -> 12 TB usable under RAID 10).
So far, we've seen the X4500 deliver impressive but suboptimal results =
using the out-of-the-box installation of Solaris + zfs. The Linux testing =
is in the early stages (no xfs, yet), but so far it yeilds comparatively =
modest write rates and very poor read and rewrite rates.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
Results under Solaris with zfs:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
Four concurrent writers:
% time dd if=3D/dev/zero of=3D/zpool1/test/50GB-zero1 bs=3D1024k count=3D51=
200 ; time sync
% time dd if=3D/dev/zero of=3D/zpool1/test/50GB-zero2 bs=3D1024k count=3D51=
200 ; time sync
% time dd if=3D/dev/zero of=3D/zpool1/test/50GB-zero3 bs=3D1024k count=3D51=
200 ; time sync
% time dd if=3D/dev/zero of=3D/zpool1/test/50GB-zero4 bs=3D1024k count=3D51=
200 ; time sync
Seq Write (bs =3D 1 MB): 128 + 122 + 131 + 124 =3D 505 MB/s
Four concurrent readers:
% time dd if=3D/zpool1/test/50GB-zero1 of=3D/dev/null bs=3D1024k
% time dd if=3D/zpool1/test/50GB-zero2 of=3D/dev/null bs=3D1024k
% time dd if=3D/zpool1/test/50GB-zero3 of=3D/dev/null bs=3D1024k
% time dd if=3D/zpool1/test/50GB-zero4 of=3D/dev/null bs=3D1024k
Seq Read (bs =3D 1 MB): 181 + 177 + 180 + 178 =3D 716 MB/s
One bonnie++ process:
% bonnie++ -r 16384 -s 32g:32k -f -n0 -d /zpool1/test/bonnie_scratch
Version 1.03 ------Sequential Output------ --Sequential Input- =
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- =
--Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP =
/sec %CP
thumper1 32G:32k 604173 98 268893 43 543389 59 =
519.2 3
thumper1,32G:32k,,,604173,98,268893,43,,,543389,59,519.2,3,,,,,,,,,,,,,
4 concurrent synchronized bonnie++ processes:
% bonnie++ -p4
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -r 16384 -s 32g:32k -y -f -n0 -d /zpool1/test/bonnie_scratch
% bonnie++ -p-1
Combined results of 4 sessions:
Seq Output: 124 + 124 + 124 + 140 =3D 512 MB/s
Rewrite: 93 + 94 + 93 + 96 =3D 376 MB/s
Seq Input: 192 + 194 + 193 + 197 =3D 776 MB/s
Random Seek: 327 + 327 + 335 + 332 =3D 1321 seeks/s
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Results under CentOS 4 with ext3 and LVM:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
% bonnie++ -s 32g:32k -f -n0 -d /large_lvm_stripe/test/bonnie_scratch
Version 1.03 ------Sequential Output------ --Sequential Input- =
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- =
--Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP =
/sec %CP
thumper1.rt 32G:32k 346595 94 59448 11 132471 12 =
479.4 2
thumper1.rtkinternal,32G:32k,,,346595,94,59448,11,,,132471,12,479.4,2,,,,,,=
,,,,,,,
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
Summary of bonnie++ results:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
sequential sequential sequential scattered
Test case write MB/s rewrite MB/s read MB/s seeks/s
------------------------- ---------- ------------ ---------- ---------
Sol10+zfs, 1 process 604 269 543 519
Sol10+zfs, 4 processes 512 376 776 1321
Cent4+ext3+LVM, 1 process 347 59 132 479
