On Thu, Dec 4, 2014 at 7:03 PM, Christian Balzer <chibi@xxxxxxx> wrote:
>
> Hello,
>
> This morning I decided to reboot a storage node (Debian Jessie, thus 3.16
> kernel and Ceph 0.80.7, HDD OSDs with SSD journals) after applying some
> changes.
>
> It came back up one OSD short, the last log lines before the reboot are:
> ---
> 2014-12-05 09:35:27.700330 7f87e789c700 2 -- 10.0.8.21:6823/29520 >> 10.0.8.22:0/5161 pipe(0x7f881b772580 sd=247 :6823 s=2 pgs=21 cs=1 l=1 c=0x7f881f469020).fault (0) Success
> 2014-12-05 09:35:27.700350 7f87f011d700 10 osd.4 pg_epoch: 293 pg[3.316( v 289'1347 (0'0,289'1347] local-les=289 n=8 ec=5 les/c 289/289 288/288/288) [8,4,16] r=1 lpr=288 pi=276-287/1 luod=0'0 crt=289'1345 lcod 289'1346 active] cancel_copy_ops
> ---
>
> Quite obviously it didn't complete its shutdown, so unsurprisingly we get:
> ---
> 2014-12-05 09:37:40.278128 7f218a7037c0 1 journal _open /var/lib/ceph/osd/ceph-4/journal fd 24: 10000269312 bytes, block size 4096 bytes, directio = 1, aio = 1
> 2014-12-05 09:37:40.278427 7f218a7037c0 -1 journal read_header error decoding journal header
> 2014-12-05 09:37:40.278479 7f218a7037c0 -1 filestore(/var/lib/ceph/osd/ceph-4) mount failed to open journal /var/lib/ceph/osd/ceph-4/journal: (22) Invalid argument
> 2014-12-05 09:37:40.776203 7f218a7037c0 -1 osd.4 0 OSD:init: unable to mount object store
> 2014-12-05 09:37:40.776223 7f218a7037c0 -1 ESC[0;31m ** ERROR: osd init failed: (22) Invalid argument
> ESC[0m
> ---
>
> Thankfully this isn't production yet and I was eventually able to recover
> the OSD by re-creating the journal ("ceph-osd -i 4 --mkjournal"), but it
> leaves me with a rather bad taste in my mouth.
>
> So the pertinent questions would be:
>
> 1. What caused this?
> My bet is on the evil systemd just pulling the plug before the poor OSD
> had finished its shutdown job.
>
> 2. How to prevent it from happening again?
> Is there something the Ceph developers can do with regards to init scripts?
> Or is this something to be brought up with the Debian maintainer?
> Debian is transiting from sysv-init to systemd (booo!) with Jessie, but
> the OSDs still have a sysvinit magic file in their top directory. Could
> this have an affect on things?
>
> 3. Is it really that easy to trash your OSDs?
> In the case a storage node crashes, am I to expect most if not all OSDs or
> at least their journals to require manual loving?
So this "can't happen". Being force killed definitely can't kill the
OSD's disk state; that's the whole point of the journaling. The error
message indicates that the header written on disk is nonsense to the
OSD, which means that the local filesystem or disk lost something
somehow (assuming you haven't done something silly like downgrading
the software version it's running) and doesn't know it (if there had
been a read error the output would be different). I'd double-check
your disk settings etc just to be sure, and check for known issues
with xfs on Jessie.
-Greg
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
