Hello, everyone.
I have a pretty basic ceph setup running on top of Azure Cloud, (4 mons and 10 osd's) for rbd images.
Everything seems to be working as expected until i put some load on it, sometimes it doesn't complete the process (mysql restore for ex.) and sometimes it does without any issues.
Client Kernel: 3.10.0-327.10.1.el7.x86_64
OSD Kernel: 3.10.0-229.7.2.el7.x86_64
Ceph: ceph-0.94.5-0.el7.x86_64
On the client side, i have 100%iowait, a lot of "INFO: task blocked for more than 120 seconds"
On the osd side, i have no evidences of faulty disk or read/write latency, but i found the following messages:
2016-03-28 17:04:03.425249 7f7329fc5700 0 bad crc in data 641367213 != exp 3107019767
2016-03-28 17:04:03.440599 7f7329fc5700 0 -- 10.0.3.9:6800/2272 >> 10.0.2.5:0/1998047321 pipe(0x13cc4800 sd=54 :6800 s=0 pgs=0 cs=0 l=0 c=0x13883f40).accept peer addr is really 10.0.2.5:0/1998047321 (socket is 10.0.2.5:34702/0)
2016-03-28 17:04:03.487497 7f7333e6a700 0 -- 10.0.3.9:6800/2272 submit_message osd_op_reply(20046 rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~524288] v1753'32512 uv32512 _ondisk_ = 0) v6 remote, 10.0.2.5:0/1998047321, failed lossy con, dropping message 0x12b539c0
2016-03-28 17:04:03.532302 7f733666f700 0 -- 10.0.3.9:6800/2272 submit_message osd_op_reply(20047 rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] v1753'32513 uv32513 _ondisk_ = 0) v6 remote, 10.0.2.5:0/1998047321, failed lossy con, dropping message 0x1667bc80
2016-03-28 17:04:03.535143 7f7333e6a700 0 -- 10.0.3.9:6800/2272 submit_message osd_op_reply(20048 rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~524288] v1753'32514 uv32514 _ondisk_ = 0) v6 remote, 10.0.2.5:0/1998047321, failed lossy con, dropping message 0x12b56e00
---
Diego Castro / The CloudFather
GetupCloud.com - Eliminamos a Gravidade
GetupCloud.com - Eliminamos a Gravidade
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
