Hello, I have a fresh Proxmox installation on 5 servers (Supermciro X10SRW-F, Xeon E5-1660 v4, 128 GB RAM) with each 8 Samsung SSD SM863 960GB connected to a LSI-9300-8i (SAS3008) controller used as OSDs for Ceph (12.1.2) The servers are connected to two Arista DCS-7060CX-32S switches. I'm using MLAG bond (bondmode LACP, xmit_hash_policy layer3+4, MTU 9000): * backend network for Ceph: cluster network & public network Mellanox ConnectX-4 Lx dual-port 25 GBit/s * frontend network: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ dual-port Ceph is quite a default installation with size=3. My problem: I'm issuing a dd (dd if=/dev/urandom of=urandom.0 bs=10M count=1024) in a test virtual machine (the only one running in the cluster) with arround 210 MB/s. I get output drops on all switchports. The drop rate is between 0.1 - 0.9 %. The drop rate of 0.9 % is reached when writing with about 1300MB/s into ceph. First I thought about a problem with the Mellanox cards and used the Intel cards for ceph traffic. The problem also exists. I tried quite a lot and nothing help: * changed the MTU from 9000 to 1500 * changed bond_xmit_hash_policy from layer3+4 to layer2+3 * deactivated the bond and just used a single link * disabled offloading * disabled power management in BIOS * perf-bias 0 I analyzed the traffic via tcpdump and got some of those "errors": * TCP Previous segment not captured * TCP Out-of-Order * TCP Retransmission * TCP Fast Retransmission * TCP Dup ACK * TCP ACKed unseen segment * TCP Window Update Is that behaviour normal for ceph or has anyone ideas how to solve that problem with the output drops at switch-side With iperf I can reach full 50 GBit/s on the bond with zero output drops. Andreas _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |