Hi,
I haven't been able to do this. I was asked to wipe the Ceph cluster, so
I wiped all the OSDs and the Mons and restored the system from backups.
Would have been something interesting to investigate, but this was a
system ready to go production and had to be online before a deadline.
On 06/26/2013 10:51 PM, Gregory Farnum wrote:
On Wed, Jun 26, 2013 at 1:37 PM, Wido den Hollander <wido@xxxxxxxx> wrote:
On 06/26/2013 06:54 PM, Gregory Farnum wrote:
On Wed, Jun 26, 2013 at 12:24 AM, Wido den Hollander <wido@xxxxxxxx>
wrote:
On 06/26/2013 01:18 AM, Gregory Farnum wrote:
Some guesses are inline.
On Tue, Jun 25, 2013 at 4:06 PM, Wido den Hollander <wido@xxxxxxxx>
wrote:
Hi,
I'm not sure what happened, but on a Ceph cluster I noticed that the
monitors (running 0.61) started filling up the disks, so they were
restarted
with:
mon compact on start = true
After a restart the osdmap was empty, it showed:
osdmap e2: 0 osds: 0 up, 0 in
pgmap v624077: 15296 pgs: 15296 stale+active+clean; 78104 MB
data,
243
GB used, 66789 GB / 67032 GB avail
mdsmap e1: 0/0/1 up
This cluster has 36 OSDs over 9 hosts, but suddenly that was all gone.
I also checked the crushmap, all 36 OSDs were removed, no trace of
them.
As you guess, this is probably because the disks filled up. It
shouldn't be able to happen but we found an edge case where leveldb
falls apart; there's a fix for it in the repository now (asserting
that we get back what we just wrote) that Sage can talk more about.
Probably both disappeared because the monitor got nothing back when
reading in the newest OSD Map, and so it's all empty.
Sounds reasonable and logical.
"ceph auth list" still showed their keys though.
Restarting the OSDs didn't help, since create-or-move complained that
the
OSDs didn't exist and didn't do anything. I ran "ceph osd create" to
get
the
36 OSDs created again, but when the OSDs boot they never start working.
The only thing they log is:
2013-06-26 01:00:08.852410 7f17f3f16700 0 -- 0.0.0.0:6801/4767 >>
10.23.24.53:6801/1758 pipe(0x1025fc80 sd=116 :40516 s=1 pgs=0 cs=0
l=0).fault with nothing to send, going to standby
Are they going up and just sitting idle? This is probably because none
of their peers are telling them to be responsible for any placement
groups on startup.
No, they never come up. So checking the monitor logs I only see the
create-or-move command changing their crush position, but they never mark
themselves as "up", so all the OSDs stay down.
netstat however shows a connection with the monitor between the OSD and
the
Mon, but nothing special in the logs at lower debugging.
So the process is still running? Can you generate full logs with debug
ms = 5, debug osd = 20, debug monc = 20?
I've done so with 4 OSDs and I uploaded the logs of one OSD:
root@data1:~# sftp cephdrop@xxxxxxxx
cephdrop@xxxxxxxx's password:
Connected to ceph.com.
sftp> put ceph-osd-0-widodh-empty-osdmap.log.gz
Uploading ceph-osd-0-widodh-empty-osdmap.log.gz to
/home/cephdrop/ceph-osd-0-widodh-empty-osdmap.log.gz
ceph-osd-0-widodh-empty-osdmap.log.gz
100% 14MB 3.5MB/s 00:04
sftp>
My internet here is to slow to go through the logs and I haven't checked
them yet.
Okay, looks like the OSD is sending a boot message and the monitor is
never completing the boot process. This is probably because of the
lost OSDMaps (it also seems to think that it has maps 0-53, and I
admit I'm a little confused about how that could have happened). So
you'll need to unstick the monitors, which I'm afraid will be a bit of
an adventure.
Yes, you should be able to extract the maps from the OSDs and put them
in the monitor, but I'm not sure exactly how that will work — you
probably want to come in to irc and get somebody to walk you through
manipulating the OSD store; I think with the monstore-tool there's a
put but you or somebody might need to add that. :)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
