On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@xxxxxxxxxxxx> wrote: > Hi, > I am doing cluster failure test, where I shut down one OSD an wait for the > cluster to sync. But the sync never finshed, at around 4-5% it stops. I > stoped osd2. ... > 2011-11-17 16:42:45.520740 pg v1337: 600 pgs: 547 active+clean, 53 > active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB avail; > 4025/82404 degraded (4.884%) ... > The osd log, the ceph.conf, pg dump, osd dump could be found here. > > http://85.214.49.87/ceph/ This looks a bit worrying: 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >> 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 - wrong node! So osd.0 is basically refusing to talk to one of the other OSDs. I don't understand the messenger well enough to know why this would be, but it wouldn't surprise me if this problem kept the objects degraded -- it looks like a breakage in the osd<->osd communication. Now if this was the reason, I'd expect a restart of all the OSDs to get it back in shape; messenger state is ephemeral. Can you confirm that? -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
