Re: Deadly slow Ceph cluster revisited

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I suggest you look at the current offload settings on the NIC

There have been quite a few bugs when bonding, vlans, bridges etc. are involved - sometimes you have to set the offload settings on the logical interface (like bond0.vlan_id) and not on the NIC.
Even then, what I’ve seen is that if the interface goes down and up, those settings are _reset_ and you need to set them again, but YMMV as that depends on what the driver does (and in our case has been fixed in newer EL6 kernels).

The behaviour was almost exactly as what you see and we use Intel 82599ES (very close to X540, not sure if the same driver).

run:
ethtool -K $dev tso off gso off gro off lro off
on the f18 and see if it makes a difference.

Jan



On 18 Jul 2015, at 02:01, Alex Gorbachev <ag@xxxxxxxxxxxxxxxxxxx> wrote:

May I suggest checking also the error counters on your network switch?

Check speed and duplex. Is bonding in use?  Is flow control on?  Can you swap the network cable?  Can you swap a NIC with another node and does the problem follow?

Hth, Alex

On Friday, July 17, 2015, Steve Thompson <smt@xxxxxxxxxxxx> wrote:
On Fri, 17 Jul 2015, J David wrote:

f16 inbound: 6Gbps
f16 outbound: 6Gbps
f17 inbound: 6Gbps
f17 outbound: 6Gbps
f18 inbound: 6Gbps
f18 outbound: 1.2Mbps

Unless the network was very busy when you did this, I think that 6 Gb/s may not be very good either. Usually iperf will give you much more than that. For example, between two of my OSD's, I get 9.4 Gb/s, or up to 9.9 Gb/s when nothing else is happening.

Steve
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux