--- begin dump of recent events ---
-72> 2016-06-06 11:49:37.720315 7f09d0152800 5 asok(0x7f09dbb78280) register_command perfcounters_dump hook 0x7f09dbb58050
-71> 2016-06-06 11:49:37.720328 7f09d0152800 5 asok(0x7f09dbb78280) register_command 1 hook 0x7f09dbb58050
-70> 2016-06-06 11:49:37.720330 7f09d0152800 5 asok(0x7f09dbb78280) register_command perf dump hook 0x7f09dbb58050
-69> 2016-06-06 11:49:37.720332 7f09d0152800 5 asok(0x7f09dbb78280) register_command perfcounters_schema hook 0x7f09dbb58050
-68> 2016-06-06 11:49:37.720333 7f09d0152800 5 asok(0x7f09dbb78280) register_command 2 hook 0x7f09dbb58050
-67> 2016-06-06 11:49:37.720334 7f09d0152800 5 asok(0x7f09dbb78280) register_command perf schema hook 0x7f09dbb58050
-66> 2016-06-06 11:49:37.720335 7f09d0152800 5 asok(0x7f09dbb78280) register_command perf reset hook 0x7f09dbb58050
-65> 2016-06-06 11:49:37.720337 7f09d0152800 5 asok(0x7f09dbb78280) register_command config show hook 0x7f09dbb58050
-64> 2016-06-06 11:49:37.720338 7f09d0152800 5 asok(0x7f09dbb78280) register_command config set hook 0x7f09dbb58050
-63> 2016-06-06 11:49:37.720339 7f09d0152800 5 asok(0x7f09dbb78280) register_command config get hook 0x7f09dbb58050
-62> 2016-06-06 11:49:37.720340 7f09d0152800 5 asok(0x7f09dbb78280) register_command config diff hook 0x7f09dbb58050
-61> 2016-06-06 11:49:37.720342 7f09d0152800 5 asok(0x7f09dbb78280) register_command log flush hook 0x7f09dbb58050
-60> 2016-06-06 11:49:37.720343 7f09d0152800 5 asok(0x7f09dbb78280) register_command log dump hook 0x7f09dbb58050
-59> 2016-06-06 11:49:37.720344 7f09d0152800 5 asok(0x7f09dbb78280) register_command log reopen hook 0x7f09dbb58050
-58> 2016-06-06 11:49:37.723459 7f09d0152800 0 set uid:gid to 1000:1000 (ceph:ceph)
-57> 2016-06-06 11:49:37.723476 7f09d0152800 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 9943
-56> 2016-06-06 11:49:37.727080 7f09d0152800 1 -- 10.253.50.213:0/0 learned my addr 10.253.50.213:0/0
-55> 2016-06-06 11:49:37.727092 7f09d0152800 1 accepter.accepter.bind my_inst.addr is 10.253.50.213:6806/9943 need_addr=0
-54> 2016-06-06 11:49:37.727104 7f09d0152800 1 -- 172.16.1.3:0/0 learned my addr 172.16.1.3:0/0
-53> 2016-06-06 11:49:37.727109 7f09d0152800 1 accepter.accepter.bind my_inst.addr is 172.16.1.3:6806/9943 need_addr=0
-52> 2016-06-06 11:49:37.727119 7f09d0152800 1 -- 172.16.1.3:0/0 learned my addr 172.16.1.3:0/0
-51> 2016-06-06 11:49:37.727129 7f09d0152800 1 accepter.accepter.bind my_inst.addr is 172.16.1.3:6807/9943 need_addr=0
-50> 2016-06-06 11:49:37.727139 7f09d0152800 1 -- 10.253.50.213:0/0 learned my addr 10.253.50.213:0/0
-49> 2016-06-06 11:49:37.727143 7f09d0152800 1 accepter.accepter.bind my_inst.addr is 10.253.50.213:6807/9943 need_addr=0
-48> 2016-06-06 11:49:37.727148 7f09d0152800 0 pidfile_write: ignore empty --pid-file
-47> 2016-06-06 11:49:37.728364 7f09d0152800 5 asok(0x7f09dbb78280) init /var/run/ceph/ceph-osd.8.asok
-46> 2016-06-06 11:49:37.728417 7f09d0152800 5 asok(0x7f09dbb78280) bind_and_listen /var/run/ceph/ceph-osd.8.asok
-45> 2016-06-06 11:49:37.728472 7f09d0152800 5 asok(0x7f09dbb78280) register_command 0 hook 0x7f09dbb54110
-44> 2016-06-06 11:49:37.728488 7f09d0152800 5 asok(0x7f09dbb78280) register_command version hook 0x7f09dbb54110
-43> 2016-06-06 11:49:37.728493 7f09d0152800 5 asok(0x7f09dbb78280) register_command git_version hook 0x7f09dbb54110
-42> 2016-06-06 11:49:37.728498 7f09d0152800 5 asok(0x7f09dbb78280) register_command help hook 0x7f09dbb58230
-41> 2016-06-06 11:49:37.728502 7f09d0152800 5 asok(0x7f09dbb78280) register_command get_command_descriptions hook 0x7f09dbb58220
-40> 2016-06-06 11:49:37.728544 7f09d0152800 10 monclient(hunting): build_initial_monmap
-39> 2016-06-06 11:49:37.734765 7f09c9df5700 5 asok(0x7f09dbb78280) entry start
-38> 2016-06-06 11:49:37.736541 7f09d0152800 5 adding auth protocol: cephx
-37> 2016-06-06 11:49:37.736552 7f09d0152800 5 adding auth protocol: cephx
-36> 2016-06-06 11:49:37.736672 7f09d0152800 5 asok(0x7f09dbb78280) register_command objecter_requests hook 0x7f09dbb58270
-35> 2016-06-06 11:49:37.736710 7f09d0152800 1 -- 10.253.50.213:6806/9943 messenger.start
-34> 2016-06-06 11:49:37.736732 7f09d0152800 1 -- :/0 messenger.start
-33> 2016-06-06 11:49:37.736748 7f09d0152800 1 -- 10.253.50.213:6807/9943 messenger.start
-32> 2016-06-06 11:49:37.736763 7f09d0152800 1 -- 172.16.1.3:6807/9943 messenger.start
-31> 2016-06-06 11:49:37.736774 7f09d0152800 1 -- 172.16.1.3:6806/9943 messenger.start
-30> 2016-06-06 11:49:37.736786 7f09d0152800 1 -- :/0 messenger.start
-29> 2016-06-06 11:49:37.736821 7f09d0152800 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
-28> 2016-06-06 11:49:37.736864 7f09d0152800 0 filestore(/var/lib/ceph/osd/ceph-8) backend xfs (magic 0x58465342)
-27> 2016-06-06 11:49:37.737113 7f09d0152800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-26> 2016-06-06 11:49:37.737127 7f09d0152800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-25> 2016-06-06 11:49:37.737144 7f09d0152800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: splice is supported
-24> 2016-06-06 11:49:37.748618 7f09d0152800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-23> 2016-06-06 11:49:37.748704 7f09d0152800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_feature: extsize is disabled by conf
-22> 2016-06-06 11:49:37.749225 7f09d0152800 1 leveldb: Recovering log #62532
-21> 2016-06-06 11:49:37.837041 7f09d0152800 1 leveldb: Delete type=3 #62531
-20> 2016-06-06 11:49:37.837098 7f09d0152800 1 leveldb: Delete type=0 #62532
-19> 2016-06-06 11:49:37.837539 7f09d0152800 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-18> 2016-06-06 11:49:37.839069 7f09d0152800 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 12375982-ca61-4170-93a0-03f1930ced83 fs_op_seq 58991306
-17> 2016-06-06 11:49:37.839106 7f09d0152800 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 10737418240 bytes, block size 4096 bytes, directio = 1, aio = 1
-16> 2016-06-06 11:49:37.873359 7f09d0152800 2 journal No further valid entries found, journal is most likely valid
-15> 2016-06-06 11:49:37.873378 7f09d0152800 2 journal No further valid entries found, journal is most likely valid
-14> 2016-06-06 11:49:37.873382 7f09d0152800 3 journal journal_replay: end of journal, done.
-13> 2016-06-06 11:49:37.873430 7f09d0152800 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 10737418240 bytes, block size 4096 bytes, directio = 1, aio = 1
-12> 2016-06-06 11:49:37.896293 7f09d0152800 1 filestore(/var/lib/ceph/osd/ceph-8) upgrade
-11> 2016-06-06 11:49:37.896346 7f09d0152800 2 osd.8 0 boot
-10> 2016-06-06 11:49:37.896597 7f09d0152800 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-9> 2016-06-06 11:49:37.898335 7f09d0152800 1 <cls> cls/refcount/cls_refcount.cc:232: Loaded refcount class!
-8> 2016-06-06 11:49:37.898423 7f09d0152800 1 <cls> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class!
-7> 2016-06-06 11:49:37.898507 7f09d0152800 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-6> 2016-06-06 11:49:37.898587 7f09d0152800 1 <cls> cls/log/cls_log.cc:317: Loaded log class!
-5> 2016-06-06 11:49:37.908832 7f09d0152800 1 <cls> cls/user/cls_user.cc:375: Loaded user class!
-4> 2016-06-06 11:49:37.910755 7f09d0152800 1 <cls> cls/rgw/cls_rgw.cc:3207: Loaded rgw class!
-3> 2016-06-06 11:49:37.920661 7f09d0152800 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
-2> 2016-06-06 11:49:37.920774 7f09d0152800 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
-1> 2016-06-06 11:49:37.920926 7f09d0152800 1 <cls> cls/version/cls_version.cc:228: Loaded version class!
0> 2016-06-06 11:49:37.940701 7f09d0152800 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f09d0152800 time 2016-06-06 11:49:37.921083
osd/OSD.h: 885: FAILED assert(ret)
ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f09d0b50e2b]
2: (OSDService::get_map(unsigned int)+0x3d) [0x7f09d055c85d]
3: (OSD::init()+0x1ed2) [0x7f09d0512052]
4: (main()+0x29d1) [0x7f09d0479cb1]
5: (__libc_start_main()+0xf5) [0x7f09cd075ec5]
6: (()+0x353987) [0x7f09d04c2987]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
--- end dump of recent events ---
Now, if I do a zap on the disk and add everything back manually, that works, however I would really not like to do that for 12 x 10 nodes.
Does anyone have any ideas?
Thanks.
-Tu Holmes
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
