On 6/27/12 8:04 AM, Stefan Priebe - Profihost AG wrote:
Hello list,
i'm still thinking about optimal OSD hardware and while reading through
the mailinglist and wiki had some questions.
I want to use SSD so my idea was to use a fast single socket cpu with
8-10 SSD disks per OSD.
I got the following recommandations through the mailinglist:
"Dual socket servers will be overkill given the setup you're describing.
Our WAG rule of thumb is 1GHz of modern CPU per OSD daemon. You might
consider it if you decided you wanted to do an OSD per disk instead
(that's a more common configuration, but it requires more CPU and RAM
per disk and we don't know yet which is the better choice)."
but in my tests i see a CPU usage of 160% + 15% kworker per OSD Daemon
on a 3,6ghz intel xeon CPU. That's far away of 1GHz per OSD. That's
around 6,3Ghz per OSD. Is anything wrong here?
When i want to use 8-10 SSD Disks i need around 20 cores with 3,6Ghz.
But there is no single socket with 20 cores with 3,6Ghz.
Or should i consider using a Raid 5 or 6?
Anything wrong?
Stefan
Hi Stefan,
I'm not entirely clear how you are coming to the conclusion regarding
the CPU requirements. If we go by the "1Ghz per OSD" suggestion, does
that mean you plan to have 3.6GHz*20/1GHz = 72 OSDs per server?
Having said that, not all CPU cores are created equal. Intel CPUs tend
to be faster per clock than AMD CPUs, though AMD systems can potentially
have more cores per node (16 per socket). If you are really planning on
having 72 OSDs per node, other things are going to come into play
including CPU interconnnect, raid controller performance, PCI bus,
network throughput, etc. I'd strongly recommend sticking with smaller
nodes unless you have the time/budget to test such large systems. I
haven't gotten a chance to really dig into CPU utilization yet, but I'd
say if you are going to go for big nodes you might try putting a single
Xeon E5 into a dual socket motherboard and see how it works. If it's
not fast enough stick the second CPU (and associated memory) in.
For what it's worth, I've got a pair of Dell R515 setup with a single
2.8GHz 6-core 4184 Opteron, 16GB of RAM, and 10 SSDs that are capable of
about 200MB/s each. Currently I'm topping out at about 600MB/s with
rados bench using half of the drives for data and half for journals (at
2x replication). Putting journals on the same drive and doing 10 OSDs
on each node is slower. Still working on figuring out why.
In terms of raid, the big consideration between raid 5 and raid 6 is the
potential for drive failure during a rebuild. Raid6 gives you extra
protection at the cost of reduced capacity and performance. If you have
a small array with small fast drives raid 5 might be fine. If you have
a large array with many high capacity drives, 6 may be better.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
