Re: Switches and latency

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

On Thu, 16 Jun 2016 12:44:51 +0200 Gandalf Corvotempesta wrote:

> 2016-06-16 3:53 GMT+02:00 Christian Balzer <chibi@xxxxxxx>:
> > Gandalf, first read:
> > https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg29546.html
> >
> > And this thread by Nick:
> > https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg29708.html
> 
> Interesting reading. Thanks.
> 
> > Overly optimistic.
> > In an idle cluster with synthetic tests you might get sequential reads
> > that are around 150MB/s per HDD.
> > As for writes, think 80MB/s, again in an idle cluster.
> >
> > Any realistic, random I/O and you're looking at 50MB/s at most either
> > way.
> >
> > So your storage nodes can't really saturate even a single 10Gb/s link
> > in real life situations.
> 
> Ok.
> 
> > Journal SSDs can improve on things, but that's mostly for IOPS.
> > In fact they easily become the bottleneck bandwidth wise and are so on
> > most of my storage nodes.
> > Because you'd need at least 2 400GB DC S3710 SSDs to get around 1GB/s
> > writes, or one link worth.
> 
> I plan to use 1 or 2 SSD journal (probably, 1 SSD every 6 spinning disks)
> 
That's as large as I would make that failure domain, also make sure to
choose SSDs that work well with Ceph, endurance and sync write speed wise
(lots of threads about this).

> > Splitting things in cluster and public networks ONLY makes sense when
> > your storage node can saturate ALL the network bandwidth, which
> > usually is only the case when it comes to very expensive SSD/NVMe only
> > nodes.
> 
> This is not my case.
> 
> > Going back to your original post, with a split network the latency in
> > both networks counts the same, as a client write will NOT be
> > acknowledged until it has reach the journal of all replicas, so having
> > a higher latency cluster network is counterproductive.
> 
> Ok.
> 
> > Or if you can start with a clean slate (including the clients), look at
> > Infiniband.
> > All my production clusters are running entirely IB (IPoIB currently)
> > and I'm very happy with the performance, latency and cost.
> 
> Yes, i'll start with a brand new network.

Alas you're not really, as you say below your clients already have 10GigE
ports.
So I'll be terse from here.

> Acutally i'm testing with some old IB switches (DDR) and i'm not very
> happy, as IPoIB doesn't go over 8/9Gbit/s in a DDR. 
You should have gotten some insight out of your "RDMA/Infiniband status"
thread, but I never bothered with DDR.

> Additionally, CX4
> cables used by DDR are... HUGE and very "hard" to bend in the rack.
> I don't know if QDR cables are thinner.
> 
Nope, but then again some of the 10GigE cables are rather stiff or
fragile, too.

> Are you using QDR? 
Yes, because it's cheaper, we don't need the bandwidth and the latency is
supposedly lower than FDR.

> I've seen a couple of mellanox used switches on ebay
> that seems to be ok for me. 36 QDR ports would be awesome but I don't
> have any IB knowledge.
Largest switch we use is 18 ports, our clusters are small.

> Could I keep the IB fabric unconfigured and use only IPoIB ?
Pretty much, a very basic OpenSM config and you're good to go.

> I can create a bonded (failover) IPoIB device on each node and add 2 or
> more IB cables between both switches. In a normal Ethernet network,
> these 2 cables must be joined in a LAG to avoid loops. Is infiniband
> able to manage this on their
> own ? 
Yes. The basic/default OpenSM router needs to be told to use more than one
path if possible "lmc 2", other routers are also available and have more bells and
whistles.


>I've never find a way to aggragate multiple ports.
Re-read that OSPF thread...
IB has means to do this, alas IPoIB bonding only supports failover at this
time, yes.

> The real drawback with IB is that I have to add IB cards on each compute
> nodes, where my current compute nodes a 2 10GBaseT ports onboard.
> 
> This add some costs....
> 
Then look at the 10GigE options I listed.

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi@xxxxxxx   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux