Re: tons of "failed lossy con, dropping message" => root cause for bad performance ?

[Gregory Farnum <greg@xxxxxxxxxxx>]

Looks like maybe your network is faulty. The crc error means the OSD received a message with a checksum that didn't match. The dropped message indicates that the connection (in this case to a client) has failed (probably because of the bad crc?) and so it's dropping the outgoing message. This is intentional; if the connection is re-established from the other end that state gets replayed and handled properly.

-Greg

On Monday, September 2, 2013, Matthieu Patou wrote:

Hello All,

I have a simple test setup with 2 osd servers each 3 NICs (1Gb each):

* One for management (ssh and such)
* One for the public network (connected to ceph clients)
* One for the cluster (osd inter-connection)

I keep seeing this messages:


Aug 26 18:43:31 ceph01 ceph-osd: 2013-08-26 18:43:31.040038 7f1afe5b6700  0 -- 192.168.113.115:6801/14629 submit_message osd_op_reply(88713 rb.0.1133.74b0dc51.0000000003cd [write 2297856~4096] _ondisk_ = 0) v4 remote, 192.168.113.1:0/607109564, failed lossy con, dropping message 0xaf83680
Aug 26 18:43:32 ceph01 ceph-osd: 2013-08-26 18:43:32.578875 7f1afe5b6700  0 -- 192.168.113.115:6801/14629 submit_message osd_op_reply(88870 rb.0.1133.74b0dc51.000000000345 [write 3145728~524288] _ondisk_ = 0) v4 remote, 192.168.113.1:0/607109564, failed lossy con, dropping message 0x5c2e1e0

And also:
Aug 26 18:27:08 ceph01 ceph-osd: 2013-08-26 18:27:08.211604 7f3cf738f700  0 bad crc in data 1545773059 != exp 878537506
 Aug 26 18:27:08 ceph01 ceph-osd: 2013-08-26 18:27:08.225121 7f3cf738f700  0 bad crc in data 1929463652 != exp 2083940607

Any idea on the problem ?

Matthieu.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Software Engineer #42 @ http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com