On Wed, 17 Apr 2019 10:39:10 +0200 Lars Täuber wrote: > Wed, 17 Apr 2019 09:52:29 +0200 > Stefan Kooman <stefan@xxxxxx> ==> Lars Täuber <taeuber@xxxxxxx> : > > Quoting Lars Täuber (taeuber@xxxxxxx): > > > > I'd probably only use the 25G network for both networks instead of > > > > using both. Splitting the network usually doesn't help. > > > > > > This is something i was told to do, because a reconstruction of failed > > > OSDs/disks would have a heavy impact on the backend network. > > > > Opinions vary on running "public" only versus "public" / "backend". > > Having a separate "backend" network might lead to difficult to debug > > issues when the "public" network is working fine, but the "backend" is > > having issues and OSDs can't peer with each other, while the clients can > > talk to all OSDs. You will get slow requests and OSDs marking each other > > down while they are still running etc. > > This I was not aware of. > Split networks are usually more trouble than their worth and as stated only help when your OSD speeds exceed the network bandwidth _and_ you can't do a CLAG bonding over switches that support it, gaining both additional bandwidth and redundancy. > > > In your case with only 6 spinners max per server there is no way you > > will every fill the network capacity of a 25 Gb/s network: 6 * 250 MB/s > > (for large spinners) should be just enough to fill a 10 Gb/s link. A > > redundant 25 Gb/s link would provide 50 Gb/s of bandwith, enough for > > both OSD replication traffic and client IO. > > The reason for the choice for the 25GBit network was because a remark of someone, that the latency in this ethernet is way below that of 10GBit. I never double checked this. > Correct, 25Gb/s is a split of 100GB/s, inheriting the latency advantages from it. So if you do a lot of small IOPS, this will help. But only completely so if everything is on the same boat. So if you clients (or most of them at least) can be on 25GB/s as well, that would be the best situation, with a non-split network. Christian > > > > > My 2 cents, > > > > Gr. Stefan > > > > Cheers, > Lars > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |