On Wed, Jun 26, 2013 at 1:37 PM, Wido den Hollander <wido@xxxxxxxx> wrote: > On 06/26/2013 06:54 PM, Gregory Farnum wrote: >> >> On Wed, Jun 26, 2013 at 12:24 AM, Wido den Hollander <wido@xxxxxxxx> >> wrote: >>> >>> On 06/26/2013 01:18 AM, Gregory Farnum wrote: >>>> >>>> >>>> Some guesses are inline. >>>> >>>> On Tue, Jun 25, 2013 at 4:06 PM, Wido den Hollander <wido@xxxxxxxx> >>>> wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> I'm not sure what happened, but on a Ceph cluster I noticed that the >>>>> monitors (running 0.61) started filling up the disks, so they were >>>>> restarted >>>>> with: >>>>> >>>>> mon compact on start = true >>>>> >>>>> After a restart the osdmap was empty, it showed: >>>>> >>>>> osdmap e2: 0 osds: 0 up, 0 in >>>>> pgmap v624077: 15296 pgs: 15296 stale+active+clean; 78104 MB >>>>> data, >>>>> 243 >>>>> GB used, 66789 GB / 67032 GB avail >>>>> mdsmap e1: 0/0/1 up >>>>> >>>>> This cluster has 36 OSDs over 9 hosts, but suddenly that was all gone. >>>>> >>>>> I also checked the crushmap, all 36 OSDs were removed, no trace of >>>>> them. >>>> >>>> >>>> >>>> As you guess, this is probably because the disks filled up. It >>>> shouldn't be able to happen but we found an edge case where leveldb >>>> falls apart; there's a fix for it in the repository now (asserting >>>> that we get back what we just wrote) that Sage can talk more about. >>>> Probably both disappeared because the monitor got nothing back when >>>> reading in the newest OSD Map, and so it's all empty. >>>> >>> >>> Sounds reasonable and logical. >>> >>> >>>>> "ceph auth list" still showed their keys though. >>>>> >>>>> Restarting the OSDs didn't help, since create-or-move complained that >>>>> the >>>>> OSDs didn't exist and didn't do anything. I ran "ceph osd create" to >>>>> get >>>>> the >>>>> 36 OSDs created again, but when the OSDs boot they never start working. >>>>> >>>>> The only thing they log is: >>>>> >>>>> 2013-06-26 01:00:08.852410 7f17f3f16700 0 -- 0.0.0.0:6801/4767 >> >>>>> 10.23.24.53:6801/1758 pipe(0x1025fc80 sd=116 :40516 s=1 pgs=0 cs=0 >>>>> l=0).fault with nothing to send, going to standby >>>> >>>> >>>> >>>> Are they going up and just sitting idle? This is probably because none >>>> of their peers are telling them to be responsible for any placement >>>> groups on startup. >>>> >>> >>> No, they never come up. So checking the monitor logs I only see the >>> create-or-move command changing their crush position, but they never mark >>> themselves as "up", so all the OSDs stay down. >>> >>> netstat however shows a connection with the monitor between the OSD and >>> the >>> Mon, but nothing special in the logs at lower debugging. >> >> >> So the process is still running? Can you generate full logs with debug >> ms = 5, debug osd = 20, debug monc = 20? >> > > I've done so with 4 OSDs and I uploaded the logs of one OSD: > > root@data1:~# sftp cephdrop@xxxxxxxx > cephdrop@xxxxxxxx's password: > Connected to ceph.com. > sftp> put ceph-osd-0-widodh-empty-osdmap.log.gz > Uploading ceph-osd-0-widodh-empty-osdmap.log.gz to > /home/cephdrop/ceph-osd-0-widodh-empty-osdmap.log.gz > ceph-osd-0-widodh-empty-osdmap.log.gz > 100% 14MB 3.5MB/s 00:04 > sftp> > > My internet here is to slow to go through the logs and I haven't checked > them yet. Okay, looks like the OSD is sending a boot message and the monitor is never completing the boot process. This is probably because of the lost OSDMaps (it also seems to think that it has maps 0-53, and I admit I'm a little confused about how that could have happened). So you'll need to unstick the monitors, which I'm afraid will be a bit of an adventure. Yes, you should be able to extract the maps from the OSDs and put them in the monitor, but I'm not sure exactly how that will work — you probably want to come in to irc and get somebody to walk you through manipulating the OSD store; I think with the monstore-tool there's a put but you or somebody might need to add that. :) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |