Hi Kenji,
On 13-2-2007 20:46 Kenji Morishige wrote:
Here is the full specification of the new box I hope to build and run FreeBSD
6.X and PostgreSQL on:
- SuperMicro Dual Xeon X7DBE+ motherboard
+ 2 x Quad Core X5355 2.66Ghz
OR
+ 2 x Dual Core 5160 3.0Ghz
- 8 x 1GB PC2-4200 fully buffered DIMM
- LSI MegaRAID SAS 8408E w/BBU 256MB
- 16 x 73GB SAS disk
If this is in one of those 4U cases, make very, very sure it can
properly exhaust all the heat generated and has more than adequate power
supply. When going for a similar machine, we got a negative advice on
such a set-up from a server vendor who built such machines themselves.
Don't forget that the FB-dimms run pretty hot and they need sufficient
cooling. As you can see on these pictures Fujitsu thought it necessary
to add fan-ducts for the memory:
http://tweakers.net/reviews/646/7
Our own Dell systems have similar ducts. But a third-party server
builder we tested did not include those, and the machine ran very hot (I
couldn't touch the bottom for more than a short time) in a
not-too-good-ventilated, but mostly empty, server rack. Although that
was a 2U machine, but it didn't include any disks. Currently we have had
good experience with our new Dell 1950 (2x 5160, PC5300 FBD) combined
with a Dell MD1000 SAS disk unit (15x 15k 36G disks) described in the
second review linked below. HP offers similar options and there are
probably several other suppliers who can build something like that too.
Seperate SAS-JBOD disk units are available from other suppliers as well.
So, question #1, to go dual core or quad core? Quad core in theory seems to
scale the machine's processing potential by almost a factor of two.
I can partially answer that question, but than for linux + postgresql
8.2. In that case, postgresql can take advantage of the extra core. See
our review here:
http://tweakers.net/reviews/661
This includes comparisons between the X5355 and 5160 with postgresql on
the seventh page, here: http://tweakers.net/reviews/661/7
But be aware that there can be substantial and unexpected differences on
this relatively new platform due to simply changing the OS, like we saw
when going from linux 2.6.15 to 2.6.18, as you can see here:
http://tweakers.net/reviews/657/2
Our benchmark has relatively little writing and a smallish dataset (fits
in 4GB of memory), so I don't know how much use these benchmarks are for
you. But the conclusion was that the extra processor power isn't fully
available, possibly because there is less memory bandwidth per processor
core and more communication overhead. Then again, in our test the dual
quad core was faster than the dual dual core.
And lastly, up till now, I've only have experience configuring SCSI RAID
controllers. I believe this LSI MegaRAID unit has a dual channel setup, but
when it comes to SAS drives, I don't know what kind of flexibility this
provides. How should the disks be partitioned for maximum PostgreSQL
performance?
I'm thinking about keeping it simple assuming that the hot spare can only be
utilized one per channel leaving me only 14 disks to utilize.
In that Dell 1950 we use the Dell PERC5/e SAS-controller for the
database, which is based on that same LSI controller, although it has 2
external sas connections (for 4 channels each). Afaik it supports global
hot spares. But we use the full set of 15 disks as a 14+1 disk raid 5,
so I haven't looked at that too well. For the OS we have a seperate
PERC5/i "internal" raid controller with two internal disks. My
colleague also tested several raid set-ups with that equipment, and we
choose a raid5 for its slightly better read-performance. If you can make
something of these dutch pages, you can have a look at those results here:
http://tweakers.net/benchdb/test/122
Play around with the form at the bottom of the page to see some
comparisons between several raid set-ups. The sas configurations are of
course the ones with the "Fujitsu MAX3036RC 36GB" disks and "Dell PERC
5/E" controller.
If I am not able to utilize both channels to create a 12 disk RAID1+0 array,
then it might be better to create 2 seperate data partitions, one for
WAL/pg_xlog and the rest for the data store.
I'm not too sure how you can connect your disks and controller to a
SAS-expander (which you need to connect more than 8 disks to a
controller). I believe it is possible to use a 24-port expander,
allowing communication between the 16 disks and 8 ports of your
controller. A SAS-expander comes normally with the enclosure/disk unit,
but I have no idea about the details. Our own testing was done using
just a single 4-port connector, which can handle 1.2GB/sec (afaik this B
is for bytes) and we believe that's sufficient for our 15 disks.
Please comment on any issues you may see with this box and my assumptions.
Also any FreeBSD kernel issues or tweaks you could recommend.
Have a very good look at your heat production and exhaust and power
supply. It was one of the reasons we decided to use seperate enclosures,
seperating the processors/memory from the big disk array.
Best regards and good luck,
Arjen van der Meijden
