I disabled the complete bond and just used a single 25 GBit/s link. The output drops still appear on the switchports. On 08.09.2017 14:12, Marc Roos wrote: > > > Afaik ceph is is not supporting/working with bonding. > > https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg35474.html > (thread: Maybe some tuning for bonded network adapters) > > > > > -----Original Message----- > From: Andreas Herrmann [mailto:andreas@xxxxxxxx] > Sent: vrijdag 8 september 2017 13:58 > To: ceph-users@xxxxxxxxxxxxxx > Subject: output discards (queue drops) on switchport > > Hello, > > I have a fresh Proxmox installation on 5 servers (Supermciro X10SRW-F, > Xeon E5-1660 v4, 128 GB RAM) with each 8 Samsung SSD SM863 960GB > connected to a LSI-9300-8i (SAS3008) controller used as OSDs for Ceph > (12.1.2) > > The servers are connected to two Arista DCS-7060CX-32S switches. I'm > using MLAG bond (bondmode LACP, xmit_hash_policy layer3+4, MTU 9000): > * backend network for Ceph: cluster network & public network > Mellanox ConnectX-4 Lx dual-port 25 GBit/s > * frontend network: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > dual-port > > Ceph is quite a default installation with size=3. > > My problem: > I'm issuing a dd (dd if=/dev/urandom of=urandom.0 bs=10M count=1024) in > a test virtual machine (the only one running in the cluster) with > arround 210 MB/s. I get output drops on all switchports. The drop rate > is between 0.1 - 0.9 %. The drop rate of 0.9 % is reached when writing > with about 1300MB/s into ceph. > > First I thought about a problem with the Mellanox cards and used the > Intel cards for ceph traffic. The problem also exists. > > I tried quite a lot and nothing help: > * changed the MTU from 9000 to 1500 > * changed bond_xmit_hash_policy from layer3+4 to layer2+3 > * deactivated the bond and just used a single link > * disabled offloading > * disabled power management in BIOS > * perf-bias 0 > > I analyzed the traffic via tcpdump and got some of those "errors": > * TCP Previous segment not captured > * TCP Out-of-Order > * TCP Retransmission > * TCP Fast Retransmission > * TCP Dup ACK > * TCP ACKed unseen segment > * TCP Window Update > > Is that behaviour normal for ceph or has anyone ideas how to solve that > problem with the output drops at switch-side > > With iperf I can reach full 50 GBit/s on the bond with zero output > drops. > > Andreas > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |