Hi,
some thoughts about network and disks inline
On 10/04/2016 03:43 PM, Denny Fuchs wrote:
Hello,
*snipsnap*
* Storage NIC: 1 x Infiniband MCX314A-BCCT
** I red, that ConnectX-3 Pro is better supported, than the X-4 and a
bit cheaper
** Switch: 2 x Mellanox SX6012 (56Gb/s)
** Active FC cables
** Maybe VPI is nice to have, but unsure.
The Infiniband support is Ceph is experimental and not recommended for
production uses. You'll have to fall back to IPoIB for the moment. The
ConnectX-3 has configurable ports and also supports 40GbE, so ethernet
switches might be an alternative for your setup. (some of the Mellanox
switch support both infiniband and ethernet).
* Production NIC: 1 x Intel 520 dual port SFP+
** Connected each to one of a HP 2920 10Gb/s ports via 802.3ad
All nodes are connected over cross to every switch, so if one switch
goes down, a second path is available.
* Disk:
** Storage: 24 x Crucial MX300 250GB (maybe for production 12xSSD /
12x big Sata disks)
** OSD journal: 1 x Intel SSD DC P3700 PCIe
Not sure about the OSD SSDs, but keep in mind that consumer SSDs are not
intended for running under load 24/7. Have a closer look at their write
endurance and ensure that the SSDs are monitored properly. If you take
ll of them into production at the same time and got a good data
distribution over all SSDs, they might also be failing within a very
short time span...
The journal SSD is OK (we have the same model), but according to tests
it is only capable of writing about 1 GB/s as journal SSD (obligatory
blog link:
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/).
With 2x 10GbE public network links, the SSD might become the bottleneck
in large scale write operations.
For read operations with 24 SSDs (assuming 400MB/s per SSD -> 9,6 GB/s)
the network will definitely become the bottleneck. You also might want
to check whether the I/O subsystem is able to drive 24 SSDs (SAS-3 has
12 GBit/s, expander are usually connected with 4 channels -> 6 GB/s).
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
