Hi and good morning,
Am 04.10.2016 17:19, schrieb Burkhard Linke:
* Storage NIC: 1 x Infiniband MCX314A-BCCT
** I red, that ConnectX-3 Pro is better supported, than the X-4 and a
bit cheaper
** Switch: 2 x Mellanox SX6012 (56Gb/s)
** Active FC cables
** Maybe VPI is nice to have, but unsure.
The Infiniband support is Ceph is experimental and not recommended for
production uses. You'll have to fall back to IPoIB for the moment. The
Oha, that is a bit new for me but we expecting to use IPoIB anyway, so
the SX6012 supports it out of the box, as I red.
ConnectX-3 has configurable ports and also supports 40GbE, so ethernet
switches might be an alternative for your setup. (some of the Mellanox
switch support both infiniband and ethernet).
We calculated how much costs full 10Gb network and we where at a point,
that IB makes more sense for us (as we have already 10Gb switches, but
with only 4 ports each switch).
All other kind of high network equipment isn't standardize in a way, we
would trust ;-) Think on how many ways you have, to connect 10Gb ...
So IB is from our side a proven network (I thought). One other reason
was, that we can split 100% the network, for security/policy reason.
[...]
data distribution over all SSDs, they might also be failing within a
very short time span...
Worst case: many SSDs fails at a very short timeslot .... :-(
The journal SSD is OK (we have the same model), but according to tests
it is only capable of writing about 1 GB/s as journal SSD (obligatory
blog link:
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/).
I added from our side benchmarks to this side too.
With 2x 10GbE public network links, the SSD might become the
bottleneck in large scale write operations.
Here we are a bit unsure ... at this time, we thought of 2 x 1Gb/s
(LACP) or 10Gb/s with Intel 520 ... The thing is, that we already bought
4 HP 2920 switches as a replacement for older switches. But now we
have/want to use them for our new project. So we have 10 physical server
(6 OSD nodes, 4 hypervisor nodes) but only 16 10Gb (SFP+) ports.
Not the best choice ... but bought without asking the right persons. :-/
For read operations with 24 SSDs (assuming 400MB/s per SSD -> 9,6
GB/s) the network will definitely become the bottleneck. You also
might want to check whether the I/O subsystem is able to drive 24 SSDs
(SAS-3 has 12 GBit/s, expander are usually connected with 4 channels
-> 6 GB/s).
Our chassis has 12Gb without expander and all drives connected directly
via 24 port HBA :-)
Thanks a lot for suggestions.
cu denny
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
