Sam,
Thanks for taking a look. It does seem to fit my issue. Would just removing the 5.0_head directory be appropriate or would using ceph-objectstore-tool be better?
Thanks,
Berant
On Mon, May 18, 2015 at 1:47 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
You have most likely hit http://tracker.ceph.com/issues/11429. There are some workarounds in the bugs marked as duplicates of that bug, or you can wait for the next hammer point release.
-Sam
_______________________________________________
----- Original Message -----
From: "Berant Lemmenes" <berant@xxxxxxxxxxxx>
To: ceph-users@xxxxxxxxxxxxxx
Sent: Monday, May 18, 2015 10:24:38 AM
Subject: OSD unable to start (giant -> hammer)
Hello all,
I've encountered a problem when upgrading my single node home cluster from giant to hammer, and I would greatly appreciate any insight.
I upgraded the packages like normal, then proceeded to restart the mon and once that came back restarted the first OSD (osd.3). However it subsequently won't start and crashes with the following failed assertion:
osd/OSD.h: 716: FAILED assert(ret)
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xb1784f]
2: (OSD::load_pgs()+0x277b) [0x6850fb]
3: (OSD::init()+0x1448) [0x6930b8]
4: (main()+0x26b9) [0x62fd89]
5: (__libc_start_main()+0xed) [0x7f2345bc976d]
6: ceph-osd() [0x635679]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f2347f71780
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: ceph-osd() [0xa1fe55]
2: (()+0xfcb0) [0x7f2346fb1cb0]
3: (gsignal()+0x35) [0x7f2345bde0d5]
4: (abort()+0x17b) [0x7f2345be183b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f234652f69d]
6: (()+0xb5846) [0x7f234652d846]
7: (()+0xb5873) [0x7f234652d873]
8: (()+0xb596e) [0x7f234652d96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xb17a29]
10: (OSD::load_pgs()+0x277b) [0x6850fb]
11: (OSD::init()+0x1448) [0x6930b8]
12: (main()+0x26b9) [0x62fd89]
13: (__libc_start_main()+0xed) [0x7f2345bc976d]
14: ceph-osd() [0x635679]
2015-05-18 13:02:33.643064 7f2347f71780 -1 *** Caught signal (Aborted) **
in thread 7f2347f71780
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: ceph-osd() [0xa1fe55]
2: (()+0xfcb0) [0x7f2346fb1cb0]
3: (gsignal()+0x35) [0x7f2345bde0d5]
4: (abort()+0x17b) [0x7f2345be183b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f234652f69d]
6: (()+0xb5846) [0x7f234652d846]
7: (()+0xb5873) [0x7f234652d873]
8: (()+0xb596e) [0x7f234652d96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xb17a29]
10: (OSD::load_pgs()+0x277b) [0x6850fb]
11: (OSD::init()+0x1448) [0x6930b8]
12: (main()+0x26b9) [0x62fd89]
13: (__libc_start_main()+0xed) [0x7f2345bc976d]
14: ceph-osd() [0x635679]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2015-05-18 13:02:33.643064 7f2347f71780 -1 *** Caught signal (Aborted) **
in thread 7f2347f71780
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: ceph-osd() [0xa1fe55]
2: (()+0xfcb0) [0x7f2346fb1cb0]
3: (gsignal()+0x35) [0x7f2345bde0d5]
4: (abort()+0x17b) [0x7f2345be183b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f234652f69d]
6: (()+0xb5846) [0x7f234652d846]
7: (()+0xb5873) [0x7f234652d873]
8: (()+0xb596e) [0x7f234652d96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xb17a29]
10: (OSD::load_pgs()+0x277b) [0x6850fb]
11: (OSD::init()+0x1448) [0x6930b8]
12: (main()+0x26b9) [0x62fd89]
13: (__libc_start_main()+0xed) [0x7f2345bc976d]
14: ceph-osd() [0x635679]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
I've included a 'ceph osd dump' here:
http://pastebin.com/RKbaY7nv
ceph osd tree:
ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 24.14000 root default
-3 0 rack unknownrack
-2 0 host ceph-test
-4 24.14000 host ceph01
0 1.50000 osd.0 down 0 1.00000
2 1.50000 osd.2 down 0 1.00000
3 1.50000 osd.3 down 1.00000 1.00000
5 2.00000 osd.5 up 1.00000 1.00000
6 2.00000 osd.6 up 1.00000 1.00000
7 2.00000 osd.7 up 1.00000 1.00000
8 2.00000 osd.8 up 1.00000 1.00000
9 2.00000 osd.9 up 1.00000 1.00000
10 2.00000 osd.10 up 1.00000 1.00000
4 4.00000 osd.4 up 1.00000 1.00000
1 3.64000 osd.1 up 1.00000 1.00000
Note that osd.0 and osd.2 were down prior to the upgrade and the cluster was healthy (these are failed disks that have been out for some time just not removed from CRUSH.
I've also included a log with OSD debugging set to 20 here:
https://dl.dropboxusercontent.com/u/1043493/osd.3.log.gz
Looking through that file, it appears the last pg that it loads successfully is 2.3f6 then it moves to 5.0
-3> 2015-05-18 12:25:24.292091 7f6f407f9780 10 osd.3 39533 load_pgs loaded pg[2.3f6( v 39533'289849 (37945'286848,39533'289849] local-les=39532 n=99 ec=1 les/c 39532/39532 39531/39531/39523) [5,4,3] r=2 lpr=39533 pi=34961-39530/34 crt=39533'289846 lcod 0'0 inactive NOTIFY] log((37945'286848,39533'289849], crt=39533'289846)
-2> 2015-05-18 12:25:24.292100 7f6f407f9780 10 osd.3 39533 pgid 5.0 coll 5.0_head
-1> 2015-05-18 12:25:24.570188 7f6f407f9780 20 osd.3 0 get_map 34144 - loading and decoding 0x411fd80
0> 2015-05-18 12:26:02.758914 7f6f407f9780 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f6f407f9780 time 2015-05-18 12:25:24.620468
osd/OSD.h: 716: FAILED assert(ret)
[snip]
Which I don't see 5.0 in a pg dump.
Thanks in advance!
Berant
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
