On 06/26/2013 06:54 PM, Gregory Farnum wrote:
On Wed, Jun 26, 2013 at 12:24 AM, Wido den Hollander <wido@xxxxxxxx> wrote:
On 06/26/2013 01:18 AM, Gregory Farnum wrote:
Some guesses are inline.
On Tue, Jun 25, 2013 at 4:06 PM, Wido den Hollander <wido@xxxxxxxx> wrote:
Hi,
I'm not sure what happened, but on a Ceph cluster I noticed that the
monitors (running 0.61) started filling up the disks, so they were
restarted
with:
mon compact on start = true
After a restart the osdmap was empty, it showed:
osdmap e2: 0 osds: 0 up, 0 in
pgmap v624077: 15296 pgs: 15296 stale+active+clean; 78104 MB data,
243
GB used, 66789 GB / 67032 GB avail
mdsmap e1: 0/0/1 up
This cluster has 36 OSDs over 9 hosts, but suddenly that was all gone.
I also checked the crushmap, all 36 OSDs were removed, no trace of them.
As you guess, this is probably because the disks filled up. It
shouldn't be able to happen but we found an edge case where leveldb
falls apart; there's a fix for it in the repository now (asserting
that we get back what we just wrote) that Sage can talk more about.
Probably both disappeared because the monitor got nothing back when
reading in the newest OSD Map, and so it's all empty.
Sounds reasonable and logical.
"ceph auth list" still showed their keys though.
Restarting the OSDs didn't help, since create-or-move complained that the
OSDs didn't exist and didn't do anything. I ran "ceph osd create" to get
the
36 OSDs created again, but when the OSDs boot they never start working.
The only thing they log is:
2013-06-26 01:00:08.852410 7f17f3f16700 0 -- 0.0.0.0:6801/4767 >>
10.23.24.53:6801/1758 pipe(0x1025fc80 sd=116 :40516 s=1 pgs=0 cs=0
l=0).fault with nothing to send, going to standby
Are they going up and just sitting idle? This is probably because none
of their peers are telling them to be responsible for any placement
groups on startup.
No, they never come up. So checking the monitor logs I only see the
create-or-move command changing their crush position, but they never mark
themselves as "up", so all the OSDs stay down.
netstat however shows a connection with the monitor between the OSD and the
Mon, but nothing special in the logs at lower debugging.
So the process is still running? Can you generate full logs with debug
ms = 5, debug osd = 20, debug monc = 20?
I've done so with 4 OSDs and I uploaded the logs of one OSD:
root@data1:~# sftp cephdrop@xxxxxxxx
cephdrop@xxxxxxxx's password:
Connected to ceph.com.
sftp> put ceph-osd-0-widodh-empty-osdmap.log.gz
Uploading ceph-osd-0-widodh-empty-osdmap.log.gz to
/home/cephdrop/ceph-osd-0-widodh-empty-osdmap.log.gz
ceph-osd-0-widodh-empty-osdmap.log.gz
100% 14MB 3.5MB/s 00:04
sftp>
My internet here is to slow to go through the logs and I haven't checked
them yet.
The internet connection I'm behind is a 3G connection, so I can't go
skimming through the logs with debugging at very high levels, but I'm
just
wondering what this could be?
It's obvious that the monitors filling up probably triggered the problem,
but I'm now looking at a way to get the OSDs back up again.
In the meantime I upgraded all the nodes to 0.61.4, but that didn't
change
anything.
Any ideas on what this might be and how to resolve it?
At a guess, you can go in and grab the last good version of the OSD
Map and inject that back into the cluster, then restart the OSDs? If
that doesn't work then we'll need to figure out the right way to kick
them into being responsible for their stuff.
(First, make sure that when you turn them on they are actually
connecting to the monitors.)
You mean grabbing the old OSDMap from an OSD or the Monitor store? Both are
using leveldb for their storage now, right? So I'd have to grab the OSD Map
using some leveldb tooling?
There's a ceph-monstore-tool or similar that provides this
functionality, although it's pretty new so you might need to grab an
autobuilt package somewhere instead of the cuttlefish one (not sure)
Ah, cool! I'll give that a try.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
