Hi Greg,
Thank you for your concern.
It seems that problem was caused by ceph-mds. While the rest of Ceph
modules have been upgraded to 0.61.8, ceph-mds was 0.56.7.
I've updated ceph-mds and cluster stabilised within few hours.
Kind regards, Serge
On 08/30/2013 08:22 PM, Gregory Farnum wrote:
Can you start up your mds with "dedug mds = 20" and "debug ms = 20"?
The "failed to decode message" line is suspicious but there's not
enough context here for me to be sure, and my pattern-matching isn't
reminding me of any serious bugs.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Thu, Aug 29, 2013 at 3:10 AM, Serge Slipchenko
<serge.slipchenko@xxxxxxxxxxxx> wrote:
Hi,
I upgraded Ceph from Bobtail to Cuttlefish and everything seemed good.
Then I started to write to cephfs, but at some moment write stalled.
After that I'm not able to mount either with kernel driver, or with
custom utility.
ceph -s shows that everything is good.
health HEALTH_OK
monmap e2: 2 mons at {m01=5.9.118.83:6789/0,m02=5.9.122.115:6789/0},
election epoch 1320, quorum 0,1 m01,m02
osdmap e3967: 16 osds: 16 up, 16 in
pgmap v1315932: 256 pgs: 255 active+clean, 1 active+clean+scrubbing; 215
GB data, 448 GB used, 38441 GB / 40971 GB avail; 37585KB/s rd, 1op/s
mdsmap e774: 1/1/1 up {0=m02=up:active}, 1 up:standby
But in the mds.a log I see the following messages:
2013-08-29 10:06:34.371166 7f49e68aa700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/2272475298 pipe(0x8de3780 sd=74 :6807 s=0 pgs=0 cs=0
l=0).accept peer addr is really 91.193.166.194:0/2272475298 (socket is
91.193.166.194:56649/0)
2013-08-29 10:07:38.454659 7f49e68aa700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/2272475298 pipe(0x8de3780 sd=74 :6807 s=2 pgs=2 cs=1
l=0).fault, server, going to standby
2013-08-29 10:23:06.898089 7f49e60a2700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/3930317661 pipe(0x7442c000 sd=78 :6807 s=0 pgs=0 cs=0
l=0).accept peer addr is really 91.193.166.194:0/3930317661 (socket is
91.193.166.194:56272/0)
2013-08-29 10:24:07.384136 7f49e60a2700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/3930317661 pipe(0x7442c000 sd=78 :6807 s=2 pgs=2 cs=1
l=0).fault, server, going to standby
2013-08-29 10:30:21.177807 7f49e5c9e700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/1838286378 pipe(0x73bd8a00 sd=80 :6807 s=0 pgs=0 cs=0
l=0).accept peer addr is really 91.193.166.194:0/1838286378 (socket is
91.193.166.194:59069/0)
2013-08-29 10:31:21.300004 7f49e5c9e700 0 -- 5.9.122.115:6807/1077 >>
91.193.166.194:0/1838286378 pipe(0x73bd8a00 sd=80 :6807 s=2 pgs=2 cs=1
l=0).fault, server, going to standby
2013-08-29 11:17:17.331613 7f040de6b700 0 -- 5.9.122.115:6807/7622 >>
91.193.166.194:0/2689145238 pipe(0x13ea780 sd=34 :6807 s=2 pgs=2 cs=1
l=0).fault with nothing to send, going to standby
2013-08-29 11:22:08.137711 7f0411897700 0 log [INF] : closing stale
session client.76201 91.193.166.194:0/2689145238 after 304.270364
And mds.b outputs a lot of:
2013-08-29 12:04:58.743938 7fa75604d700 -1 failed to decode message of
type 23 v2: buffer::end_of_buffer
2013-08-29 12:04:58.743969 7fa75604d700 0 -- 5.9.122.115:6800/977 >>
144.76.13.103:0/925435369 pipe(0x524e780 sd=39 :6800 s=2 pgs=130763
cs=12829 l=0).fault with nothing to send, going to standby
2013-08-29 12:04:58.744236 7fa755f4c700 0 -- 5.9.122.115:6800/977 >>
144.76.13.102:0/2955281877 pipe(0x524e500 sd=37 :6800 s=0 pgs=0 cs=0
l=0).accept connect_seq 12834 vs existing 12833 state standby
2013-08-29 12:04:58.744607 7fa756754700 0 -- 5.9.122.115:6800/977 >>
144.76.13.105:0/347604456 pipe(0x52c5a00 sd=38 :6800 s=0 pgs=0 cs=0
l=0).accept connect_seq 12538 vs existing 12537 state standby
2013-08-29 12:04:58.744627 7fa755f4c700 -1 failed to decode message of
type 23 v2: buffer::end_of_buffer
2013-08-29 12:04:58.744671 7fa755f4c700 0 -- 5.9.122.115:6800/977 >>
144.76.13.102:0/2955281877 pipe(0x524e500 sd=37 :6800 s=2 pgs=292532
cs=12835 l=0).fault with nothing to send, going to standby
2013-08-29 12:04:58.745006 7fa75614e700 0 -- 5.9.122.115:6800/977 >>
144.76.13.103:0/925435369 pipe(0x52c5780 sd=31 :6800 s=0 pgs=0 cs=0
l=0).accept connect_seq 12830 vs existing 12829 state standby
2013-08-29 12:04:58.745102 7fa756754700 -1 failed to decode message of
type 23 v2: buffer::end_of_buffer
2013-08-29 12:04:58.745146 7fa756754700 0 -- 5.9.122.115:6800/977 >>
144.76.13.105:0/347604456 pipe(0x52c5a00 sd=38 :6800 s=2 pgs=131368
cs=12539 l=0).fault with nothing to send, going to standby
--
Kind regards, Serge Slipchenko
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Serge Slipchenko
E-mail: serge.slipchenko@xxxxxxxxxxxx
Skype: serge.slipchenko
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
