Hello,
On Thu, 8 May 2014 15:14:59 +0000 Carlos M. Perez wrote:
> Hi,
>
> We're just getting started with ceph, and were going to pull the order
> to get the needed hardware ordered. Wondering if anyone would offer any
> insight/suggestions on the following setup of 3 nodes:
>
> Dual 6-core Xeon (L5639/L5640/X5650 depending on what we find)
> 96GB RAM
> LSI 2008 based controller in IT mode
> 4 x 1TB Nearline SAS 7.2k rpm or
^^
I suppose that should be an "and" up there.
So a total of 2TB usable space for the whole cluster is good enough for
you?
> 2 x SSD (1 80GB+ for OS, 1 240GB+ for Journal, probably Intel
> S3500/S3700 or Samsung 840 Pro, but maybe we skip with Firefly and use
> 10k disks)
What makes you think that Firefly can do better than previous versions
without SSD journals?
A more balanced approach would be to use 2 identical SSDs and put 2
journals on each. That way your journal won't be slower than your actual
storage disks when it comes to sequential IO.
OS could easily go on the same SSDs as a software (MD) RAID1 in a
different partition.
> Infiniband QDR for Cluster/Storage traffic
Dual port Infiniband HCAs and 2 switches? Otherwise you'll have a SPOF.
> Dual Gig for network traffic
>
> This is pretty close to the recommended setup, and a lot more RAM than
> the test setups we found. We're going to be running proxmox containers
> on this setup, and the machines are low disk util, so extreme disk
> performance is not needed, which is why we we're using 96GB.
>
Proxmox... With the kernel from ages long past.
If you're using KVM, that kernel lacks a LOT of improvements to KVM that
were implemented in newer ones.
If you're using OpenVZ (no Proxmox expert here, just toyed with it once
for 2 weeks) I suppose that would need RBD devices from kernelspace, where
again the old 2.6.32 kernel might be less than optimal.
I have no idea if Proxmox allows for CPU pinning, but if you share your
storage nodes with VMs you really want that. Pin the VMS to CPUs/cores
that are not busy handling IRQs, leave in your case about 4 cores at least
free for the system and OSD tasks.
> I saw the performance tests run, and the only thing that "scares" me is
> the btfrs using a huge chunk of CPU. Will probably use ext4 Wondering
> if this was usage on a single core, or per core per instance of the
> test...
>
BTRFS has some great features.
It also is very resource intensive and has abysmal performance in many use
cases, often getting worse quickly, especially with Ceph.
Ext4 or XFS will serve you better in your case, no doubt.
Regards,
Christian
> Thanks in advance
>
> Carlos M. Perez
> CMP Consulting Services
> 305-669-1515
>
--
Christian Balzer Network/Systems Engineer
chibi at gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
