I have an OSD that’s failing to start. I can’t make heads or tails of the error (pasted below).
Thanks!
~Noah
2015-03-22 16:32:39.265116 7f4da7fa0780 0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-osd, pid 13483
2015-03-22 16:32:39.269499 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs
2015-03-22 16:32:39.269509 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2015-03-22 16:32:39.450031 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work
2015-03-22 16:32:39.450069 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-22 16:32:39.450743 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs
2015-03-22 16:32:39.499753 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-22 16:32:39.500078 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <>
2015-03-22 16:32:40.765736 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2015-03-22 16:32:40.777156 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-03-22 16:32:40.777278 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-03-22 16:32:40.778223 7f4da7fa0780 1 journal close /var/lib/ceph/osd/ceph-1/journal
2015-03-22 16:32:41.066655 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs
2015-03-22 16:32:41.150578 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work
2015-03-22 16:32:41.150624 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-22 16:32:41.151359 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs
2015-03-22 16:32:41.225302 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-22 16:32:41.225498 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <>
2015-03-22 16:32:42.375558 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2015-03-22 16:32:42.382958 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1429: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-03-22 16:32:42.383187 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1481: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-03-22 16:32:43.076434 7f4da7fa0780 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7f4da7fa0780 time 2015-03-22 16:32:43.075101
osd/PG.cc: 2270: FAILED assert(values.size() == 1)
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7]
2: (OSD::load_pgs()+0x14ce) [0x694efe]
3: (OSD::init()+0x11be) [0x69cffe]
4: (main()+0x1d09) [0x5c3509]
5: (__libc_start_main()+0xed) [0x7f4da5bde76d]
6: /usr/bin/ceph-osd() [0x5c6e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-75> 2015-03-22 16:32:39.259280 7f4da7fa0780 5 asok(0x1aec1c0) register_command perfcounters_dump hook 0x1ae4010
-74> 2015-03-22 16:32:39.259373 7f4da7fa0780 5 asok(0x1aec1c0) register_command 1 hook 0x1ae4010
-73> 2015-03-22 16:32:39.259393 7f4da7fa0780 5 asok(0x1aec1c0) register_command perf dump hook 0x1ae4010
-72> 2015-03-22 16:32:39.259429 7f4da7fa0780 5 asok(0x1aec1c0) register_command perfcounters_schema hook 0x1ae4010
-71> 2015-03-22 16:32:39.259445 7f4da7fa0780 5 asok(0x1aec1c0) register_command 2 hook 0x1ae4010
-70> 2015-03-22 16:32:39.259453 7f4da7fa0780 5 asok(0x1aec1c0) register_command perf schema hook 0x1ae4010
-69> 2015-03-22 16:32:39.259467 7f4da7fa0780 5 asok(0x1aec1c0) register_command config show hook 0x1ae4010
-68> 2015-03-22 16:32:39.259481 7f4da7fa0780 5 asok(0x1aec1c0) register_command config set hook 0x1ae4010
-67> 2015-03-22 16:32:39.259495 7f4da7fa0780 5 asok(0x1aec1c0) register_command config get hook 0x1ae4010
-66> 2015-03-22 16:32:39.259505 7f4da7fa0780 5 asok(0x1aec1c0) register_command log flush hook 0x1ae4010
-65> 2015-03-22 16:32:39.259519 7f4da7fa0780 5 asok(0x1aec1c0) register_command log dump hook 0x1ae4010
-64> 2015-03-22 16:32:39.259536 7f4da7fa0780 5 asok(0x1aec1c0) register_command log reopen hook 0x1ae4010
-63> 2015-03-22 16:32:39.265116 7f4da7fa0780 0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-osd, pid 13483
-62> 2015-03-22 16:32:39.266443 7f4da7fa0780 1 -- 192.168.41.42:0/0 learned my addr 192.168.41.42:0/0
-61> 2015-03-22 16:32:39.266462 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.41.42:6803/13483 need_addr=0
-60> 2015-03-22 16:32:39.266500 7f4da7fa0780 1 -- 192.168.42.42:0/0 learned my addr 192.168.42.42:0/0
-59> 2015-03-22 16:32:39.266537 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.42.42:6802/13483 need_addr=0
-58> 2015-03-22 16:32:39.266551 7f4da7fa0780 1 -- 192.168.42.42:0/0 learned my addr 192.168.42.42:0/0
-57> 2015-03-22 16:32:39.266560 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.42.42:6803/13483 need_addr=0
-56> 2015-03-22 16:32:39.266580 7f4da7fa0780 1 -- 192.168.41.42:0/0 learned my addr 192.168.41.42:0/0
-55> 2015-03-22 16:32:39.266602 7f4da7fa0780 1 accepter.accepter.bind my_inst.addr is 192.168.41.42:6808/13483 need_addr=0
-54> 2015-03-22 16:32:39.269108 7f4da7fa0780 5 asok(0x1aec1c0) init /var/run/ceph/ceph-osd.1.asok
-53> 2015-03-22 16:32:39.269138 7f4da7fa0780 5 asok(0x1aec1c0) bind_and_listen /var/run/ceph/ceph-osd.1.asok
-52> 2015-03-22 16:32:39.269185 7f4da7fa0780 5 asok(0x1aec1c0) register_command 0 hook 0x1ae30b0
-51> 2015-03-22 16:32:39.269203 7f4da7fa0780 5 asok(0x1aec1c0) register_command version hook 0x1ae30b0
-50> 2015-03-22 16:32:39.269206 7f4da7fa0780 5 asok(0x1aec1c0) register_command git_version hook 0x1ae30b0
-49> 2015-03-22 16:32:39.269210 7f4da7fa0780 5 asok(0x1aec1c0) register_command help hook 0x1ae40d0
-48> 2015-03-22 16:32:39.269231 7f4da7fa0780 5 asok(0x1aec1c0) register_command get_command_descriptions hook 0x1ae40c0
-47> 2015-03-22 16:32:39.269273 7f4da3c28700 5 asok(0x1aec1c0) entry start
-46> 2015-03-22 16:32:39.269499 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs
-45> 2015-03-22 16:32:39.269509 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
-44> 2015-03-22 16:32:39.450031 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work
-43> 2015-03-22 16:32:39.450069 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
-42> 2015-03-22 16:32:39.450743 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs
-41> 2015-03-22 16:32:39.499753 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel)
-40> 2015-03-22 16:32:39.500078 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <>
-39> 2015-03-22 16:32:40.765736 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected
-38> 2015-03-22 16:32:40.777088 7f4da7fa0780 2 journal open /var/lib/ceph/osd/ceph-1/journal fsid e2ad61ec-c581-4159-8671-bab77c7d4e97 fs_op_seq 81852815
-37> 2015-03-22 16:32:40.777156 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-36> 2015-03-22 16:32:40.777242 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid
-35> 2015-03-22 16:32:40.777252 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid
-34> 2015-03-22 16:32:40.777255 7f4da7fa0780 3 journal journal_replay: end of journal, done.
-33> 2015-03-22 16:32:40.777278 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 2551: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-32> 2015-03-22 16:32:40.777874 7f4d9fc20700 1 FileStore::op_tp worker finish
-31> 2015-03-22 16:32:40.777930 7f4da0421700 1 FileStore::op_tp worker finish
-30> 2015-03-22 16:32:40.778223 7f4da7fa0780 1 journal close /var/lib/ceph/osd/ceph-1/journal
-29> 2015-03-22 16:32:41.066043 7f4da7fa0780 10 monclient(hunting): build_initial_monmap
-28> 2015-03-22 16:32:41.066137 7f4da7fa0780 5 adding auth protocol: cephx
-27> 2015-03-22 16:32:41.066147 7f4da7fa0780 5 adding auth protocol: cephx
-26> 2015-03-22 16:32:41.066384 7f4da7fa0780 1 -- 192.168.41.42:6803/13483 messenger.start
-25> 2015-03-22 16:32:41.066418 7f4da7fa0780 1 -- :/0 messenger.start
-24> 2015-03-22 16:32:41.066444 7f4da7fa0780 1 -- 192.168.41.42:6808/13483 messenger.start
-23> 2015-03-22 16:32:41.066469 7f4da7fa0780 1 -- 192.168.42.42:6803/13483 messenger.start
-22> 2015-03-22 16:32:41.066512 7f4da7fa0780 1 -- 192.168.42.42:6802/13483 messenger.start
-21> 2015-03-22 16:32:41.066610 7f4da7fa0780 2 osd.1 0 mounting /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
-20> 2015-03-22 16:32:41.066655 7f4da7fa0780 1 filestore(/var/lib/ceph/osd/ceph-1) mount detected xfs
-19> 2015-03-22 16:32:41.150578 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is supported and appears to work
-18> 2015-03-22 16:32:41.150624 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
-17> 2015-03-22 16:32:41.151359 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount did NOT detect btrfs
-16> 2015-03-22 16:32:41.225302 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount syncfs(2) syscall fully supported (by glibc and kernel)
-15> 2015-03-22 16:32:41.225498 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount found snaps <>
-14> 2015-03-22 16:32:42.375558 7f4da7fa0780 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: btrfs not detected
-13> 2015-03-22 16:32:42.382825 7f4da7fa0780 2 journal open /var/lib/ceph/osd/ceph-1/journal fsid e2ad61ec-c581-4159-8671-bab77c7d4e97 fs_op_seq 81852815
-12> 2015-03-22 16:32:42.382958 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1429: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-11> 2015-03-22 16:32:42.383091 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid
-10> 2015-03-22 16:32:42.383108 7f4da7fa0780 2 journal No further valid entries found, journal is most likely valid
-9> 2015-03-22 16:32:42.383111 7f4da7fa0780 3 journal journal_replay: end of journal, done.
-8> 2015-03-22 16:32:42.383187 7f4da7fa0780 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 1481: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-7> 2015-03-22 16:32:42.383761 7f4da7fa0780 2 osd.1 0 boot
-6> 2015-03-22 16:32:42.388322 7f4da7fa0780 1 <cls> cls/rgw/cls_rgw.cc:1596: Loaded rgw class!
-5> 2015-03-22 16:32:42.389272 7f4da7fa0780 1 <cls> cls/log/cls_log.cc:313: Loaded log class!
-4> 2015-03-22 16:32:42.392742 7f4da7fa0780 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
-3> 2015-03-22 16:32:42.393520 7f4da7fa0780 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-2> 2015-03-22 16:32:42.394181 7f4da7fa0780 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-1> 2015-03-22 16:32:42.394476 7f4da7fa0780 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
0> 2015-03-22 16:32:43.076434 7f4da7fa0780 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7f4da7fa0780 time 2015-03-22 16:32:43.075101
osd/PG.cc: 2270: FAILED assert(values.size() == 1)
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
1: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7]
2: (OSD::load_pgs()+0x14ce) [0x694efe]
3: (OSD::init()+0x11be) [0x69cffe]
4: (main()+0x1d09) [0x5c3509]
5: (__libc_start_main()+0xed) [0x7f4da5bde76d]
6: /usr/bin/ceph-osd() [0x5c6e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.1.log
--- end dump of recent events ---
2015-03-22 16:32:43.079587 7f4da7fa0780 -1 *** Caught signal (Aborted) **
in thread 7f4da7fa0780
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
1: /usr/bin/ceph-osd() [0x8001ea]
2: (()+0xfcb0) [0x7f4da743acb0]
3: (gsignal()+0x35) [0x7f4da5bf3425]
4: (abort()+0x17b) [0x7f4da5bf6b8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4da654569d]
6: (()+0xb5846) [0x7f4da6543846]
7: (()+0xb5873) [0x7f4da6543873]
8: (()+0xb596e) [0x7f4da654396e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x8c5e7f]
10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7]
11: (OSD::load_pgs()+0x14ce) [0x694efe]
12: (OSD::init()+0x11be) [0x69cffe]
13: (main()+0x1d09) [0x5c3509]
14: (__libc_start_main()+0xed) [0x7f4da5bde76d]
15: /usr/bin/ceph-osd() [0x5c6e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2015-03-22 16:32:43.079587 7f4da7fa0780 -1 *** Caught signal (Aborted) **
in thread 7f4da7fa0780
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
1: /usr/bin/ceph-osd() [0x8001ea]
2: (()+0xfcb0) [0x7f4da743acb0]
3: (gsignal()+0x35) [0x7f4da5bf3425]
4: (abort()+0x17b) [0x7f4da5bf6b8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4da654569d]
6: (()+0xb5846) [0x7f4da6543846]
7: (()+0xb5873) [0x7f4da6543873]
8: (()+0xb596e) [0x7f4da654396e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x8c5e7f]
10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x4d7) [0x70ebf7]
11: (OSD::load_pgs()+0x14ce) [0x694efe]
12: (OSD::init()+0x11be) [0x69cffe]
13: (main()+0x1d09) [0x5c3509]
14: (__libc_start_main()+0xed) [0x7f4da5bde76d]
15: /usr/bin/ceph-osd() [0x5c6e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.1.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
