Hi all,
I'm trying to upgrade a ceph cluster (prev hammer release) to the last release of infernalis. So far so good while upgrading the mon servers, all work fine. But then when trying to upgrade the OSD servers I got an error while trying to start the osd services again:
What I did is first to upgrade the packages, then stop the osd daemons, run the chown -R ceph:ceph /var/lib/ceph command, and then try to start again all the daemons. Well, they are not coming back and the error on one of the OSD is the following:
(...)
5 10:21:05.910850
os/FileStore.cc: 1698: FAILED assert(r == 0)
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fec7b74489b]
2: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
3: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
4: (OSD::init()+0x269) [0x7fec7b1bc2f9]
5: (main()+0x2817) [0x7fec7b142bb7]
6: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
7: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
20/20 filestore
1/ 3 keyvaluestore
20/20 journal
1/ 1 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2015-10-15 10:21:05.923314 7fec7bc4f980 -1 *** Caught signal (Aborted) **
in thread 7fec7bc4f980
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (()+0x7f031a) [0x7fec7b65831a]
2: (()+0x10340) [0x7fec79b02340]
3: (gsignal()+0x39) [0x7fec77d7dcc9]
4: (abort()+0x148) [0x7fec77d810d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fec78688535]
6: (()+0x5e6d6) [0x7fec786866d6]
7: (()+0x5e703) [0x7fec78686703]
8: (()+0x5e922) [0x7fec78686922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7fec7b744a88]
10: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
11: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
12: (OSD::init()+0x269) [0x7fec7b1bc2f9]
13: (main()+0x2817) [0x7fec7b142bb7]
14: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
15: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2015-10-15 10:21:05.923314 7fec7bc4f980 -1 *** Caught signal (Aborted) **
in thread 7fec7bc4f980
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (()+0x7f031a) [0x7fec7b65831a]
2: (()+0x10340) [0x7fec79b02340]
3: (gsignal()+0x39) [0x7fec77d7dcc9]
4: (abort()+0x148) [0x7fec77d810d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fec78688535]
6: (()+0x5e6d6) [0x7fec786866d6]
7: (()+0x5e703) [0x7fec78686703]
8: (()+0x5e922) [0x7fec78686922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7fec7b744a88]
10: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
11: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
12: (OSD::init()+0x269) [0x7fec7b1bc2f9]
13: (main()+0x2817) [0x7fec7b142bb7]
14: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
15: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
20/20 filestore
1/ 3 keyvaluestore
20/20 journal
1/ 1 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
(...)
5 10:21:05.910850
os/FileStore.cc: 1698: FAILED assert(r == 0)
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fec7b74489b]
2: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
3: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
4: (OSD::init()+0x269) [0x7fec7b1bc2f9]
5: (main()+0x2817) [0x7fec7b142bb7]
6: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
7: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
20/20 filestore
1/ 3 keyvaluestore
20/20 journal
1/ 1 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2015-10-15 10:21:05.923314 7fec7bc4f980 -1 *** Caught signal (Aborted) **
in thread 7fec7bc4f980
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (()+0x7f031a) [0x7fec7b65831a]
2: (()+0x10340) [0x7fec79b02340]
3: (gsignal()+0x39) [0x7fec77d7dcc9]
4: (abort()+0x148) [0x7fec77d810d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fec78688535]
6: (()+0x5e6d6) [0x7fec786866d6]
7: (()+0x5e703) [0x7fec78686703]
8: (()+0x5e922) [0x7fec78686922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7fec7b744a88]
10: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
11: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
12: (OSD::init()+0x269) [0x7fec7b1bc2f9]
13: (main()+0x2817) [0x7fec7b142bb7]
14: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
15: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2015-10-15 10:21:05.923314 7fec7bc4f980 -1 *** Caught signal (Aborted) **
in thread 7fec7bc4f980
ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
1: (()+0x7f031a) [0x7fec7b65831a]
2: (()+0x10340) [0x7fec79b02340]
3: (gsignal()+0x39) [0x7fec77d7dcc9]
4: (abort()+0x148) [0x7fec77d810d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fec78688535]
6: (()+0x5e6d6) [0x7fec786866d6]
7: (()+0x5e703) [0x7fec78686703]
8: (()+0x5e922) [0x7fec78686922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7fec7b744a88]
10: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
11: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
12: (OSD::init()+0x269) [0x7fec7b1bc2f9]
13: (main()+0x2817) [0x7fec7b142bb7]
14: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
15: (()+0x30a9e7) [0x7fec7b1729e7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
20/20 filestore
1/ 3 keyvaluestore
20/20 journal
1/ 1 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
any ideas?
Thanks in advance,
cheers,
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
