Bluestore OSDs stay down

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi,

I have been very impressed with the BlueStore test environment I made, which is build on the Ubuntu 16.04 using the Ceph development master repository.

But now I have run into some self inflicted problems.

Yesterday I accidentally updated the OSD while they were being heavily used. Then the OSD started to go down one by one, and when they all had did I ended up with pgs in practically every possible state.

:~# ceph health
2016-09-30 09:50:39.044987 7f27e4ee2700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 09:50:39.052592 7f27e4ee2700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
HEALTH_ERR 243 pgs are stuck inactive for more than 300 seconds; 130 pgs backfill_wait; 488 pgs degraded; 55 pgs down; 49 pgs incomplete; 63 pgs peering; 2 pgs recovering; 281 pgs recovery_wait; 600 pgs stale; 243 pgs stuck inactive; 357 pgs stuck unclean; 488 pgs undersized; recovery 1240205/1848822 objects degraded (67.081%); recovery 397635/1848822 objects misplaced (21.507%); recovery 57149/616274 unfound (9.273%); mds cluster is degraded; 8/8 in osds are down

As mentioned all OSD's are now down and refuse to come back up.

From the osd log file I see this error message:

/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

Of course the data was not important in this test environment and the easiest would properly be to start over, but I am considering building a production environment build on Bluestore as soon as it becomes stable, so for the sport of it I would like to see if I can actually recover the OSD's. Just to get some deeper insight into Ceph recovery.

I have been though:   http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/  without any luck.

What would be the next steps to try?

Thanks!

//Jesper


  


2016-09-30 08:51:23.464389 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:23.464409 7f17985528c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:23.464430 7f17985528c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16590
2016-09-30 08:51:23.464479 7f17985528c0  5 object store type is bluestore
2016-09-30 08:51:23.464504 7f17985528c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

2016-09-30 08:51:23.466344 7f17985528c0  0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:23.468345 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:23.473859 7f17985528c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
2016-09-30 08:51:23.475249 7f17985528c0 10 load: jerasure load: lrc load: isa 
2016-09-30 08:51:23.475657 7f17985528c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:23.475669 7f17985528c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:23.475707 7f17985528c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:23.476182 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:23.476545 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:23.476814 7f17985528c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:23.477256 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:23.477501 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:23.477509 7f17985528c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:23.477533 7f17985528c0  1 bluefs mount
2016-09-30 08:51:23.553415 7f17985528c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f17985528c0 time 2016-09-30 08:51:23.550071
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d4c31a6240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
 3: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
 5: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
 6: (OSD::init()+0x273) [0x55d4c2b17ad3]
 7: (main()+0x2ea9) [0x55d4c2a65cd9]
 8: (__libc_start_main()+0xf0) [0x7f1794f93830]
 9: (_start()+0x29) [0x55d4c2ac3899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -69> 2016-09-30 08:51:23.459785 7f17985528c0  5 asok(0x55d4cc850f00) register_command perfcounters_dump hook 0x55d4cc814030
   -68> 2016-09-30 08:51:23.459810 7f17985528c0  5 asok(0x55d4cc850f00) register_command 1 hook 0x55d4cc814030
   -67> 2016-09-30 08:51:23.459815 7f17985528c0  5 asok(0x55d4cc850f00) register_command perf dump hook 0x55d4cc814030
   -66> 2016-09-30 08:51:23.459822 7f17985528c0  5 asok(0x55d4cc850f00) register_command perfcounters_schema hook 0x55d4cc814030
   -65> 2016-09-30 08:51:23.459829 7f17985528c0  5 asok(0x55d4cc850f00) register_command 2 hook 0x55d4cc814030
   -64> 2016-09-30 08:51:23.459834 7f17985528c0  5 asok(0x55d4cc850f00) register_command perf schema hook 0x55d4cc814030
   -63> 2016-09-30 08:51:23.459841 7f17985528c0  5 asok(0x55d4cc850f00) register_command perf reset hook 0x55d4cc814030
   -62> 2016-09-30 08:51:23.459845 7f17985528c0  5 asok(0x55d4cc850f00) register_command config show hook 0x55d4cc814030
   -61> 2016-09-30 08:51:23.459852 7f17985528c0  5 asok(0x55d4cc850f00) register_command config set hook 0x55d4cc814030
   -60> 2016-09-30 08:51:23.459855 7f17985528c0  5 asok(0x55d4cc850f00) register_command config get hook 0x55d4cc814030
   -59> 2016-09-30 08:51:23.459862 7f17985528c0  5 asok(0x55d4cc850f00) register_command config diff hook 0x55d4cc814030
   -58> 2016-09-30 08:51:23.459867 7f17985528c0  5 asok(0x55d4cc850f00) register_command log flush hook 0x55d4cc814030
   -57> 2016-09-30 08:51:23.459872 7f17985528c0  5 asok(0x55d4cc850f00) register_command log dump hook 0x55d4cc814030
   -56> 2016-09-30 08:51:23.459876 7f17985528c0  5 asok(0x55d4cc850f00) register_command log reopen hook 0x55d4cc814030
   -55> 2016-09-30 08:51:23.464140 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -54> 2016-09-30 08:51:23.464389 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -53> 2016-09-30 08:51:23.464409 7f17985528c0  0 set uid:gid to 64045:64045 (ceph:ceph)
   -52> 2016-09-30 08:51:23.464430 7f17985528c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16590
   -51> 2016-09-30 08:51:23.464479 7f17985528c0  5 object store type is bluestore
   -50> 2016-09-30 08:51:23.464504 7f17985528c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

   -49> 2016-09-30 08:51:23.465448 7f179309f700  2 Event(0x55d4cc851160 nevent=5000 time_id=1).set_owner idx=0 owner=139739227879168
   -48> 2016-09-30 08:51:23.465482 7f179209d700  2 Event(0x55d4cc851520 nevent=5000 time_id=1).set_owner idx=2 owner=139739211093760
   -47> 2016-09-30 08:51:23.465523 7f179289e700  2 Event(0x55d4cc851340 nevent=5000 time_id=1).set_owner idx=1 owner=139739219486464
   -46> 2016-09-30 08:51:23.465740 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -45> 2016-09-30 08:51:23.465854 7f17985528c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16590
   -44> 2016-09-30 08:51:23.465873 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -43> 2016-09-30 08:51:23.465949 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -42> 2016-09-30 08:51:23.465996 7f17985528c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16590
   -41> 2016-09-30 08:51:23.466009 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -40> 2016-09-30 08:51:23.466053 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -39> 2016-09-30 08:51:23.466104 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -38> 2016-09-30 08:51:23.466166 7f17985528c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16590
   -37> 2016-09-30 08:51:23.466181 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -36> 2016-09-30 08:51:23.466220 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -35> 2016-09-30 08:51:23.466257 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -34> 2016-09-30 08:51:23.466295 7f17985528c0  2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
   -33> 2016-09-30 08:51:23.466330 7f17985528c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16590
   -32> 2016-09-30 08:51:23.466344 7f17985528c0  0 pidfile_write: ignore empty --pid-file
   -31> 2016-09-30 08:51:23.468345 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -30> 2016-09-30 08:51:23.468362 7f17985528c0  5 asok(0x55d4cc850f00) init /var/run/ceph/ceph-osd.0.asok
   -29> 2016-09-30 08:51:23.468374 7f17985528c0  5 asok(0x55d4cc850f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -28> 2016-09-30 08:51:23.468536 7f17985528c0  5 asok(0x55d4cc850f00) register_command 0 hook 0x55d4cc810108
   -27> 2016-09-30 08:51:23.468551 7f17985528c0  5 asok(0x55d4cc850f00) register_command version hook 0x55d4cc810108
   -26> 2016-09-30 08:51:23.468561 7f17985528c0  5 asok(0x55d4cc850f00) register_command git_version hook 0x55d4cc810108
   -25> 2016-09-30 08:51:23.468571 7f17985528c0  5 asok(0x55d4cc850f00) register_command help hook 0x55d4cc8143c0
   -24> 2016-09-30 08:51:23.468581 7f17985528c0  5 asok(0x55d4cc850f00) register_command get_command_descriptions hook 0x55d4cc8143d0
   -23> 2016-09-30 08:51:23.468643 7f1790900700  5 asok(0x55d4cc850f00) entry start
   -22> 2016-09-30 08:51:23.468684 7f17985528c0 10 monclient(hunting): build_initial_monmap
   -21> 2016-09-30 08:51:23.473859 7f17985528c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
   -20> 2016-09-30 08:51:23.475249 7f17985528c0 10 load: jerasure load: lrc load: isa 
   -19> 2016-09-30 08:51:23.475320 7f17985528c0  5 adding auth protocol: cephx
   -18> 2016-09-30 08:51:23.475325 7f17985528c0  5 adding auth protocol: cephx
   -17> 2016-09-30 08:51:23.475508 7f17985528c0  5 asok(0x55d4cc850f00) register_command objecter_requests hook 0x55d4cc814470
   -16> 2016-09-30 08:51:23.475571 7f17985528c0  1 -- 0.0.0.0:6800/16590 start start
   -15> 2016-09-30 08:51:23.475581 7f17985528c0  1 -- :/0 start start
   -14> 2016-09-30 08:51:23.475583 7f17985528c0  1 -- 0.0.0.0:6803/16590 start start
   -13> 2016-09-30 08:51:23.475586 7f17985528c0  1 -- 0.0.0.0:6802/16590 start start
   -12> 2016-09-30 08:51:23.475589 7f17985528c0  1 -- 0.0.0.0:6801/16590 start start
   -11> 2016-09-30 08:51:23.475591 7f17985528c0  1 -- :/0 start start
   -10> 2016-09-30 08:51:23.475657 7f17985528c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
    -9> 2016-09-30 08:51:23.475669 7f17985528c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
    -8> 2016-09-30 08:51:23.475707 7f17985528c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -7> 2016-09-30 08:51:23.476182 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -6> 2016-09-30 08:51:23.476545 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -5> 2016-09-30 08:51:23.476814 7f17985528c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -4> 2016-09-30 08:51:23.477256 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -3> 2016-09-30 08:51:23.477501 7f17985528c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -2> 2016-09-30 08:51:23.477509 7f17985528c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
    -1> 2016-09-30 08:51:23.477533 7f17985528c0  1 bluefs mount
     0> 2016-09-30 08:51:23.553415 7f17985528c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f17985528c0 time 2016-09-30 08:51:23.550071
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d4c31a6240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
 3: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
 5: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
 6: (OSD::init()+0x273) [0x55d4c2b17ad3]
 7: (main()+0x2ea9) [0x55d4c2a65cd9]
 8: (__libc_start_main()+0xf0) [0x7f1794f93830]
 9: (_start()+0x29) [0x55d4c2ac3899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:23.556963 7f17985528c0 -1 *** Caught signal (Aborted) **
 in thread 7f17985528c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x55d4c3005c1e]
 2: (()+0x113d0) [0x7f179679a3d0]
 3: (gsignal()+0x38) [0x7f1794fa8418]
 4: (abort()+0x16a) [0x7f1794faa01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d4c31a642b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
 7: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
 9: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
 10: (OSD::init()+0x273) [0x55d4c2b17ad3]
 11: (main()+0x2ea9) [0x55d4c2a65cd9]
 12: (__libc_start_main()+0xf0) [0x7f1794f93830]
 13: (_start()+0x29) [0x55d4c2ac3899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-09-30 08:51:23.556963 7f17985528c0 -1 *** Caught signal (Aborted) **
 in thread 7f17985528c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x55d4c3005c1e]
 2: (()+0x113d0) [0x7f179679a3d0]
 3: (gsignal()+0x38) [0x7f1794fa8418]
 4: (abort()+0x16a) [0x7f1794faa01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d4c31a642b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
 7: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
 9: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
 10: (OSD::init()+0x273) [0x55d4c2b17ad3]
 11: (main()+0x2ea9) [0x55d4c2a65cd9]
 12: (__libc_start_main()+0xf0) [0x7f1794f93830]
 13: (_start()+0x29) [0x55d4c2ac3899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.040032 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.040044 7fe98b6148c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:24.040055 7fe98b6148c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16661
2016-09-30 08:51:24.040085 7fe98b6148c0  5 object store type is bluestore
2016-09-30 08:51:24.040100 7fe98b6148c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

2016-09-30 08:51:24.041250 7fe98b6148c0  0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:24.042414 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.046019 7fe98b6148c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
2016-09-30 08:51:24.047218 7fe98b6148c0 10 load: jerasure load: lrc load: isa 
2016-09-30 08:51:24.047598 7fe98b6148c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:24.047608 7fe98b6148c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:24.047651 7fe98b6148c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.048033 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.048354 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.048637 7fe98b6148c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.048927 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.049169 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.049177 7fe98b6148c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:24.049200 7fe98b6148c0  1 bluefs mount
2016-09-30 08:51:24.116575 7fe98b6148c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe98b6148c0 time 2016-09-30 08:51:24.114653
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558f57423240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
 3: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
 5: (BlueStore::mount()+0x38d) [0x558f57195aad]
 6: (OSD::init()+0x273) [0x558f56d94ad3]
 7: (main()+0x2ea9) [0x558f56ce2cd9]
 8: (__libc_start_main()+0xf0) [0x7fe988055830]
 9: (_start()+0x29) [0x558f56d40899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -69> 2016-09-30 08:51:24.036666 7fe98b6148c0  5 asok(0x558f62502f00) register_command perfcounters_dump hook 0x558f624c6030
   -68> 2016-09-30 08:51:24.036692 7fe98b6148c0  5 asok(0x558f62502f00) register_command 1 hook 0x558f624c6030
   -67> 2016-09-30 08:51:24.036699 7fe98b6148c0  5 asok(0x558f62502f00) register_command perf dump hook 0x558f624c6030
   -66> 2016-09-30 08:51:24.036706 7fe98b6148c0  5 asok(0x558f62502f00) register_command perfcounters_schema hook 0x558f624c6030
   -65> 2016-09-30 08:51:24.036712 7fe98b6148c0  5 asok(0x558f62502f00) register_command 2 hook 0x558f624c6030
   -64> 2016-09-30 08:51:24.036714 7fe98b6148c0  5 asok(0x558f62502f00) register_command perf schema hook 0x558f624c6030
   -63> 2016-09-30 08:51:24.036718 7fe98b6148c0  5 asok(0x558f62502f00) register_command perf reset hook 0x558f624c6030
   -62> 2016-09-30 08:51:24.036723 7fe98b6148c0  5 asok(0x558f62502f00) register_command config show hook 0x558f624c6030
   -61> 2016-09-30 08:51:24.036727 7fe98b6148c0  5 asok(0x558f62502f00) register_command config set hook 0x558f624c6030
   -60> 2016-09-30 08:51:24.036732 7fe98b6148c0  5 asok(0x558f62502f00) register_command config get hook 0x558f624c6030
   -59> 2016-09-30 08:51:24.036734 7fe98b6148c0  5 asok(0x558f62502f00) register_command config diff hook 0x558f624c6030
   -58> 2016-09-30 08:51:24.036737 7fe98b6148c0  5 asok(0x558f62502f00) register_command log flush hook 0x558f624c6030
   -57> 2016-09-30 08:51:24.036741 7fe98b6148c0  5 asok(0x558f62502f00) register_command log dump hook 0x558f624c6030
   -56> 2016-09-30 08:51:24.036744 7fe98b6148c0  5 asok(0x558f62502f00) register_command log reopen hook 0x558f624c6030
   -55> 2016-09-30 08:51:24.039856 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -54> 2016-09-30 08:51:24.040032 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -53> 2016-09-30 08:51:24.040044 7fe98b6148c0  0 set uid:gid to 64045:64045 (ceph:ceph)
   -52> 2016-09-30 08:51:24.040055 7fe98b6148c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16661
   -51> 2016-09-30 08:51:24.040085 7fe98b6148c0  5 object store type is bluestore
   -50> 2016-09-30 08:51:24.040100 7fe98b6148c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

   -49> 2016-09-30 08:51:24.040658 7fe986161700  2 Event(0x558f62503160 nevent=5000 time_id=1).set_owner idx=0 owner=140640953702144
   -48> 2016-09-30 08:51:24.040707 7fe985960700  2 Event(0x558f62503340 nevent=5000 time_id=1).set_owner idx=1 owner=140640945309440
   -47> 2016-09-30 08:51:24.040735 7fe98515f700  2 Event(0x558f62503520 nevent=5000 time_id=1).set_owner idx=2 owner=140640936916736
   -46> 2016-09-30 08:51:24.040869 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -45> 2016-09-30 08:51:24.040953 7fe98b6148c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16661
   -44> 2016-09-30 08:51:24.040975 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -43> 2016-09-30 08:51:24.041009 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -42> 2016-09-30 08:51:24.041050 7fe98b6148c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16661
   -41> 2016-09-30 08:51:24.041057 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -40> 2016-09-30 08:51:24.041093 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -39> 2016-09-30 08:51:24.041118 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -38> 2016-09-30 08:51:24.041143 7fe98b6148c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16661
   -37> 2016-09-30 08:51:24.041149 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -36> 2016-09-30 08:51:24.041171 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -35> 2016-09-30 08:51:24.041196 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -34> 2016-09-30 08:51:24.041221 7fe98b6148c0  2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
   -33> 2016-09-30 08:51:24.041244 7fe98b6148c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16661
   -32> 2016-09-30 08:51:24.041250 7fe98b6148c0  0 pidfile_write: ignore empty --pid-file
   -31> 2016-09-30 08:51:24.042414 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -30> 2016-09-30 08:51:24.042426 7fe98b6148c0  5 asok(0x558f62502f00) init /var/run/ceph/ceph-osd.0.asok
   -29> 2016-09-30 08:51:24.042433 7fe98b6148c0  5 asok(0x558f62502f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -28> 2016-09-30 08:51:24.042514 7fe98b6148c0  5 asok(0x558f62502f00) register_command 0 hook 0x558f624c2108
   -27> 2016-09-30 08:51:24.042520 7fe98b6148c0  5 asok(0x558f62502f00) register_command version hook 0x558f624c2108
   -26> 2016-09-30 08:51:24.042523 7fe98b6148c0  5 asok(0x558f62502f00) register_command git_version hook 0x558f624c2108
   -25> 2016-09-30 08:51:24.042527 7fe98b6148c0  5 asok(0x558f62502f00) register_command help hook 0x558f624c63c0
   -24> 2016-09-30 08:51:24.042530 7fe98b6148c0  5 asok(0x558f62502f00) register_command get_command_descriptions hook 0x558f624c63d0
   -23> 2016-09-30 08:51:24.042565 7fe9839c2700  5 asok(0x558f62502f00) entry start
   -22> 2016-09-30 08:51:24.042574 7fe98b6148c0 10 monclient(hunting): build_initial_monmap
   -21> 2016-09-30 08:51:24.046019 7fe98b6148c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
   -20> 2016-09-30 08:51:24.047218 7fe98b6148c0 10 load: jerasure load: lrc load: isa 
   -19> 2016-09-30 08:51:24.047248 7fe98b6148c0  5 adding auth protocol: cephx
   -18> 2016-09-30 08:51:24.047251 7fe98b6148c0  5 adding auth protocol: cephx
   -17> 2016-09-30 08:51:24.047462 7fe98b6148c0  5 asok(0x558f62502f00) register_command objecter_requests hook 0x558f624c6470
   -16> 2016-09-30 08:51:24.047527 7fe98b6148c0  1 -- 0.0.0.0:6800/16661 start start
   -15> 2016-09-30 08:51:24.047537 7fe98b6148c0  1 -- :/0 start start
   -14> 2016-09-30 08:51:24.047539 7fe98b6148c0  1 -- 0.0.0.0:6803/16661 start start
   -13> 2016-09-30 08:51:24.047541 7fe98b6148c0  1 -- 0.0.0.0:6802/16661 start start
   -12> 2016-09-30 08:51:24.047543 7fe98b6148c0  1 -- 0.0.0.0:6801/16661 start start
   -11> 2016-09-30 08:51:24.047545 7fe98b6148c0  1 -- :/0 start start
   -10> 2016-09-30 08:51:24.047598 7fe98b6148c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
    -9> 2016-09-30 08:51:24.047608 7fe98b6148c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
    -8> 2016-09-30 08:51:24.047651 7fe98b6148c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -7> 2016-09-30 08:51:24.048033 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -6> 2016-09-30 08:51:24.048354 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -5> 2016-09-30 08:51:24.048637 7fe98b6148c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -4> 2016-09-30 08:51:24.048927 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -3> 2016-09-30 08:51:24.049169 7fe98b6148c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -2> 2016-09-30 08:51:24.049177 7fe98b6148c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
    -1> 2016-09-30 08:51:24.049200 7fe98b6148c0  1 bluefs mount
     0> 2016-09-30 08:51:24.116575 7fe98b6148c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe98b6148c0 time 2016-09-30 08:51:24.114653
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558f57423240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
 3: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
 5: (BlueStore::mount()+0x38d) [0x558f57195aad]
 6: (OSD::init()+0x273) [0x558f56d94ad3]
 7: (main()+0x2ea9) [0x558f56ce2cd9]
 8: (__libc_start_main()+0xf0) [0x7fe988055830]
 9: (_start()+0x29) [0x558f56d40899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.118766 7fe98b6148c0 -1 *** Caught signal (Aborted) **
 in thread 7fe98b6148c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x558f57282c1e]
 2: (()+0x113d0) [0x7fe98985c3d0]
 3: (gsignal()+0x38) [0x7fe98806a418]
 4: (abort()+0x16a) [0x7fe98806c01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x558f5742342b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
 7: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
 9: (BlueStore::mount()+0x38d) [0x558f57195aad]
 10: (OSD::init()+0x273) [0x558f56d94ad3]
 11: (main()+0x2ea9) [0x558f56ce2cd9]
 12: (__libc_start_main()+0xf0) [0x7fe988055830]
 13: (_start()+0x29) [0x558f56d40899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-09-30 08:51:24.118766 7fe98b6148c0 -1 *** Caught signal (Aborted) **
 in thread 7fe98b6148c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x558f57282c1e]
 2: (()+0x113d0) [0x7fe98985c3d0]
 3: (gsignal()+0x38) [0x7fe98806a418]
 4: (abort()+0x16a) [0x7fe98806c01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x558f5742342b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
 7: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
 9: (BlueStore::mount()+0x38d) [0x558f57195aad]
 10: (OSD::init()+0x273) [0x558f56d94ad3]
 11: (main()+0x2ea9) [0x558f56ce2cd9]
 12: (__libc_start_main()+0xf0) [0x7fe988055830]
 13: (_start()+0x29) [0x558f56d40899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.541635 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.541644 7fe252eb28c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:24.541653 7fe252eb28c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16736
2016-09-30 08:51:24.541678 7fe252eb28c0  5 object store type is bluestore
2016-09-30 08:51:24.541690 7fe252eb28c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

2016-09-30 08:51:24.542708 7fe252eb28c0  0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:24.543843 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.546781 7fe252eb28c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
2016-09-30 08:51:24.547655 7fe252eb28c0 10 load: jerasure load: lrc load: isa 
2016-09-30 08:51:24.547889 7fe252eb28c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:24.547897 7fe252eb28c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:24.547923 7fe252eb28c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.548198 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.548437 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.548636 7fe252eb28c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.548890 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.549025 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.549030 7fe252eb28c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:24.549047 7fe252eb28c0  1 bluefs mount
2016-09-30 08:51:24.614917 7fe252eb28c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe252eb28c0 time 2016-09-30 08:51:24.612943
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d2d96d1240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
 3: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
 5: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
 6: (OSD::init()+0x273) [0x55d2d9042ad3]
 7: (main()+0x2ea9) [0x55d2d8f90cd9]
 8: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
 9: (_start()+0x29) [0x55d2d8fee899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -69> 2016-09-30 08:51:24.539155 7fe252eb28c0  5 asok(0x55d2e287af00) register_command perfcounters_dump hook 0x55d2e283e030
   -68> 2016-09-30 08:51:24.539166 7fe252eb28c0  5 asok(0x55d2e287af00) register_command 1 hook 0x55d2e283e030
   -67> 2016-09-30 08:51:24.539169 7fe252eb28c0  5 asok(0x55d2e287af00) register_command perf dump hook 0x55d2e283e030
   -66> 2016-09-30 08:51:24.539171 7fe252eb28c0  5 asok(0x55d2e287af00) register_command perfcounters_schema hook 0x55d2e283e030
   -65> 2016-09-30 08:51:24.539174 7fe252eb28c0  5 asok(0x55d2e287af00) register_command 2 hook 0x55d2e283e030
   -64> 2016-09-30 08:51:24.539181 7fe252eb28c0  5 asok(0x55d2e287af00) register_command perf schema hook 0x55d2e283e030
   -63> 2016-09-30 08:51:24.539184 7fe252eb28c0  5 asok(0x55d2e287af00) register_command perf reset hook 0x55d2e283e030
   -62> 2016-09-30 08:51:24.539186 7fe252eb28c0  5 asok(0x55d2e287af00) register_command config show hook 0x55d2e283e030
   -61> 2016-09-30 08:51:24.539189 7fe252eb28c0  5 asok(0x55d2e287af00) register_command config set hook 0x55d2e283e030
   -60> 2016-09-30 08:51:24.539192 7fe252eb28c0  5 asok(0x55d2e287af00) register_command config get hook 0x55d2e283e030
   -59> 2016-09-30 08:51:24.539194 7fe252eb28c0  5 asok(0x55d2e287af00) register_command config diff hook 0x55d2e283e030
   -58> 2016-09-30 08:51:24.539197 7fe252eb28c0  5 asok(0x55d2e287af00) register_command log flush hook 0x55d2e283e030
   -57> 2016-09-30 08:51:24.539199 7fe252eb28c0  5 asok(0x55d2e287af00) register_command log dump hook 0x55d2e283e030
   -56> 2016-09-30 08:51:24.539202 7fe252eb28c0  5 asok(0x55d2e287af00) register_command log reopen hook 0x55d2e283e030
   -55> 2016-09-30 08:51:24.541487 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -54> 2016-09-30 08:51:24.541635 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -53> 2016-09-30 08:51:24.541644 7fe252eb28c0  0 set uid:gid to 64045:64045 (ceph:ceph)
   -52> 2016-09-30 08:51:24.541653 7fe252eb28c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16736
   -51> 2016-09-30 08:51:24.541678 7fe252eb28c0  5 object store type is bluestore
   -50> 2016-09-30 08:51:24.541690 7fe252eb28c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

   -49> 2016-09-30 08:51:24.542285 7fe24d9ff700  2 Event(0x55d2e287b160 nevent=5000 time_id=1).set_owner idx=0 owner=140609941665536
   -48> 2016-09-30 08:51:24.542323 7fe24d1fe700  2 Event(0x55d2e287b340 nevent=5000 time_id=1).set_owner idx=1 owner=140609933272832
   -47> 2016-09-30 08:51:24.542337 7fe24c9fd700  2 Event(0x55d2e287b520 nevent=5000 time_id=1).set_owner idx=2 owner=140609924880128
   -46> 2016-09-30 08:51:24.542439 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -45> 2016-09-30 08:51:24.542487 7fe252eb28c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16736
   -44> 2016-09-30 08:51:24.542496 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -43> 2016-09-30 08:51:24.542525 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -42> 2016-09-30 08:51:24.542547 7fe252eb28c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16736
   -41> 2016-09-30 08:51:24.542554 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -40> 2016-09-30 08:51:24.542575 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -39> 2016-09-30 08:51:24.542597 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -38> 2016-09-30 08:51:24.542617 7fe252eb28c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16736
   -37> 2016-09-30 08:51:24.542623 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -36> 2016-09-30 08:51:24.542642 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -35> 2016-09-30 08:51:24.542662 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -34> 2016-09-30 08:51:24.542682 7fe252eb28c0  2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
   -33> 2016-09-30 08:51:24.542702 7fe252eb28c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16736
   -32> 2016-09-30 08:51:24.542708 7fe252eb28c0  0 pidfile_write: ignore empty --pid-file
   -31> 2016-09-30 08:51:24.543843 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -30> 2016-09-30 08:51:24.543854 7fe252eb28c0  5 asok(0x55d2e287af00) init /var/run/ceph/ceph-osd.0.asok
   -29> 2016-09-30 08:51:24.543861 7fe252eb28c0  5 asok(0x55d2e287af00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -28> 2016-09-30 08:51:24.543937 7fe252eb28c0  5 asok(0x55d2e287af00) register_command 0 hook 0x55d2e283a108
   -27> 2016-09-30 08:51:24.543944 7fe252eb28c0  5 asok(0x55d2e287af00) register_command version hook 0x55d2e283a108
   -26> 2016-09-30 08:51:24.543947 7fe252eb28c0  5 asok(0x55d2e287af00) register_command git_version hook 0x55d2e283a108
   -25> 2016-09-30 08:51:24.543950 7fe252eb28c0  5 asok(0x55d2e287af00) register_command help hook 0x55d2e283e3c0
   -24> 2016-09-30 08:51:24.543953 7fe252eb28c0  5 asok(0x55d2e287af00) register_command get_command_descriptions hook 0x55d2e283e3d0
   -23> 2016-09-30 08:51:24.543986 7fe24b260700  5 asok(0x55d2e287af00) entry start
   -22> 2016-09-30 08:51:24.543996 7fe252eb28c0 10 monclient(hunting): build_initial_monmap
   -21> 2016-09-30 08:51:24.546781 7fe252eb28c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
   -20> 2016-09-30 08:51:24.547655 7fe252eb28c0 10 load: jerasure load: lrc load: isa 
   -19> 2016-09-30 08:51:24.547669 7fe252eb28c0  5 adding auth protocol: cephx
   -18> 2016-09-30 08:51:24.547671 7fe252eb28c0  5 adding auth protocol: cephx
   -17> 2016-09-30 08:51:24.547786 7fe252eb28c0  5 asok(0x55d2e287af00) register_command objecter_requests hook 0x55d2e283e470
   -16> 2016-09-30 08:51:24.547832 7fe252eb28c0  1 -- 0.0.0.0:6800/16736 start start
   -15> 2016-09-30 08:51:24.547839 7fe252eb28c0  1 -- :/0 start start
   -14> 2016-09-30 08:51:24.547840 7fe252eb28c0  1 -- 0.0.0.0:6803/16736 start start
   -13> 2016-09-30 08:51:24.547842 7fe252eb28c0  1 -- 0.0.0.0:6802/16736 start start
   -12> 2016-09-30 08:51:24.547844 7fe252eb28c0  1 -- 0.0.0.0:6801/16736 start start
   -11> 2016-09-30 08:51:24.547846 7fe252eb28c0  1 -- :/0 start start
   -10> 2016-09-30 08:51:24.547889 7fe252eb28c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
    -9> 2016-09-30 08:51:24.547897 7fe252eb28c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
    -8> 2016-09-30 08:51:24.547923 7fe252eb28c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -7> 2016-09-30 08:51:24.548198 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -6> 2016-09-30 08:51:24.548437 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -5> 2016-09-30 08:51:24.548636 7fe252eb28c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -4> 2016-09-30 08:51:24.548890 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -3> 2016-09-30 08:51:24.549025 7fe252eb28c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -2> 2016-09-30 08:51:24.549030 7fe252eb28c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
    -1> 2016-09-30 08:51:24.549047 7fe252eb28c0  1 bluefs mount
     0> 2016-09-30 08:51:24.614917 7fe252eb28c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe252eb28c0 time 2016-09-30 08:51:24.612943
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d2d96d1240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
 3: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
 5: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
 6: (OSD::init()+0x273) [0x55d2d9042ad3]
 7: (main()+0x2ea9) [0x55d2d8f90cd9]
 8: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
 9: (_start()+0x29) [0x55d2d8fee899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.617118 7fe252eb28c0 -1 *** Caught signal (Aborted) **
 in thread 7fe252eb28c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x55d2d9530c1e]
 2: (()+0x113d0) [0x7fe2510fa3d0]
 3: (gsignal()+0x38) [0x7fe24f908418]
 4: (abort()+0x16a) [0x7fe24f90a01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d2d96d142b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
 7: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
 9: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
 10: (OSD::init()+0x273) [0x55d2d9042ad3]
 11: (main()+0x2ea9) [0x55d2d8f90cd9]
 12: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
 13: (_start()+0x29) [0x55d2d8fee899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-09-30 08:51:24.617118 7fe252eb28c0 -1 *** Caught signal (Aborted) **
 in thread 7fe252eb28c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x55d2d9530c1e]
 2: (()+0x113d0) [0x7fe2510fa3d0]
 3: (gsignal()+0x38) [0x7fe24f908418]
 4: (abort()+0x16a) [0x7fe24f90a01a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d2d96d142b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
 7: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
 9: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
 10: (OSD::init()+0x273) [0x55d2d9042ad3]
 11: (main()+0x2ea9) [0x55d2d8f90cd9]
 12: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
 13: (_start()+0x29) [0x55d2d8fee899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:25.033591 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:25.033599 7f0f52f8c8c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:25.033607 7f0f52f8c8c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16762
2016-09-30 08:51:25.033630 7f0f52f8c8c0  5 object store type is bluestore
2016-09-30 08:51:25.033643 7f0f52f8c8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

2016-09-30 08:51:25.034470 7f0f52f8c8c0  0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:25.035546 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:25.038375 7f0f52f8c8c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
2016-09-30 08:51:25.039178 7f0f52f8c8c0 10 load: jerasure load: lrc load: isa 
2016-09-30 08:51:25.039432 7f0f52f8c8c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:25.039441 7f0f52f8c8c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:25.039465 7f0f52f8c8c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:25.039757 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:25.039987 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:25.040181 7f0f52f8c8c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:25.040429 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:25.040563 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:25.040568 7f0f52f8c8c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:25.040584 7f0f52f8c8c0  1 bluefs mount
2016-09-30 08:51:25.106090 7f0f52f8c8c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f0f52f8c8c0 time 2016-09-30 08:51:25.104252
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555e8167e240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
 3: (BlueFS::mount()+0x40a) [0x555e814a72ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
 5: (BlueStore::mount()+0x38d) [0x555e813f0aad]
 6: (OSD::init()+0x273) [0x555e80fefad3]
 7: (main()+0x2ea9) [0x555e80f3dcd9]
 8: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
 9: (_start()+0x29) [0x555e80f9b899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -69> 2016-09-30 08:51:25.031050 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command perfcounters_dump hook 0x555e8c21a030
   -68> 2016-09-30 08:51:25.031061 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command 1 hook 0x555e8c21a030
   -67> 2016-09-30 08:51:25.031064 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command perf dump hook 0x555e8c21a030
   -66> 2016-09-30 08:51:25.031068 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command perfcounters_schema hook 0x555e8c21a030
   -65> 2016-09-30 08:51:25.031074 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command 2 hook 0x555e8c21a030
   -64> 2016-09-30 08:51:25.031077 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command perf schema hook 0x555e8c21a030
   -63> 2016-09-30 08:51:25.031080 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command perf reset hook 0x555e8c21a030
   -62> 2016-09-30 08:51:25.031084 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command config show hook 0x555e8c21a030
   -61> 2016-09-30 08:51:25.031086 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command config set hook 0x555e8c21a030
   -60> 2016-09-30 08:51:25.031089 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command config get hook 0x555e8c21a030
   -59> 2016-09-30 08:51:25.031091 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command config diff hook 0x555e8c21a030
   -58> 2016-09-30 08:51:25.031094 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command log flush hook 0x555e8c21a030
   -57> 2016-09-30 08:51:25.031098 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command log dump hook 0x555e8c21a030
   -56> 2016-09-30 08:51:25.031100 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command log reopen hook 0x555e8c21a030
   -55> 2016-09-30 08:51:25.033408 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -54> 2016-09-30 08:51:25.033591 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -53> 2016-09-30 08:51:25.033599 7f0f52f8c8c0  0 set uid:gid to 64045:64045 (ceph:ceph)
   -52> 2016-09-30 08:51:25.033607 7f0f52f8c8c0  0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16762
   -51> 2016-09-30 08:51:25.033630 7f0f52f8c8c0  5 object store type is bluestore
   -50> 2016-09-30 08:51:25.033643 7f0f52f8c8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

   -49> 2016-09-30 08:51:25.034030 7f0f4dad9700  2 Event(0x555e8c257160 nevent=5000 time_id=1).set_owner idx=0 owner=139703704459008
   -48> 2016-09-30 08:51:25.034045 7f0f4d2d8700  2 Event(0x555e8c257340 nevent=5000 time_id=1).set_owner idx=1 owner=139703696066304
   -47> 2016-09-30 08:51:25.034087 7f0f4cad7700  2 Event(0x555e8c257520 nevent=5000 time_id=1).set_owner idx=2 owner=139703687673600
   -46> 2016-09-30 08:51:25.034187 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -45> 2016-09-30 08:51:25.034237 7f0f52f8c8c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16762
   -44> 2016-09-30 08:51:25.034245 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -43> 2016-09-30 08:51:25.034273 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -42> 2016-09-30 08:51:25.034297 7f0f52f8c8c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16762
   -41> 2016-09-30 08:51:25.034303 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -40> 2016-09-30 08:51:25.034324 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -39> 2016-09-30 08:51:25.034347 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -38> 2016-09-30 08:51:25.034374 7f0f52f8c8c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16762
   -37> 2016-09-30 08:51:25.034381 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -36> 2016-09-30 08:51:25.034402 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -35> 2016-09-30 08:51:25.034423 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -34> 2016-09-30 08:51:25.034445 7f0f52f8c8c0  2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
   -33> 2016-09-30 08:51:25.034465 7f0f52f8c8c0  1  Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16762
   -32> 2016-09-30 08:51:25.034470 7f0f52f8c8c0  0 pidfile_write: ignore empty --pid-file
   -31> 2016-09-30 08:51:25.035546 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -30> 2016-09-30 08:51:25.035556 7f0f52f8c8c0  5 asok(0x555e8c256f00) init /var/run/ceph/ceph-osd.0.asok
   -29> 2016-09-30 08:51:25.035562 7f0f52f8c8c0  5 asok(0x555e8c256f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -28> 2016-09-30 08:51:25.035632 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command 0 hook 0x555e8c216108
   -27> 2016-09-30 08:51:25.035638 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command version hook 0x555e8c216108
   -26> 2016-09-30 08:51:25.035642 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command git_version hook 0x555e8c216108
   -25> 2016-09-30 08:51:25.035646 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command help hook 0x555e8c21a3c0
   -24> 2016-09-30 08:51:25.035648 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command get_command_descriptions hook 0x555e8c21a3d0
   -23> 2016-09-30 08:51:25.035675 7f0f4b33a700  5 asok(0x555e8c256f00) entry start
   -22> 2016-09-30 08:51:25.035690 7f0f52f8c8c0 10 monclient(hunting): build_initial_monmap
   -21> 2016-09-30 08:51:25.038375 7f0f52f8c8c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4 
   -20> 2016-09-30 08:51:25.039178 7f0f52f8c8c0 10 load: jerasure load: lrc load: isa 
   -19> 2016-09-30 08:51:25.039190 7f0f52f8c8c0  5 adding auth protocol: cephx
   -18> 2016-09-30 08:51:25.039194 7f0f52f8c8c0  5 adding auth protocol: cephx
   -17> 2016-09-30 08:51:25.039333 7f0f52f8c8c0  5 asok(0x555e8c256f00) register_command objecter_requests hook 0x555e8c21a470
   -16> 2016-09-30 08:51:25.039376 7f0f52f8c8c0  1 -- 0.0.0.0:6800/16762 start start
   -15> 2016-09-30 08:51:25.039383 7f0f52f8c8c0  1 -- :/0 start start
   -14> 2016-09-30 08:51:25.039385 7f0f52f8c8c0  1 -- 0.0.0.0:6803/16762 start start
   -13> 2016-09-30 08:51:25.039387 7f0f52f8c8c0  1 -- 0.0.0.0:6802/16762 start start
   -12> 2016-09-30 08:51:25.039388 7f0f52f8c8c0  1 -- 0.0.0.0:6801/16762 start start
   -11> 2016-09-30 08:51:25.039390 7f0f52f8c8c0  1 -- :/0 start start
   -10> 2016-09-30 08:51:25.039432 7f0f52f8c8c0  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
    -9> 2016-09-30 08:51:25.039441 7f0f52f8c8c0  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
    -8> 2016-09-30 08:51:25.039465 7f0f52f8c8c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -7> 2016-09-30 08:51:25.039757 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -6> 2016-09-30 08:51:25.039987 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -5> 2016-09-30 08:51:25.040181 7f0f52f8c8c0  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -4> 2016-09-30 08:51:25.040429 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -3> 2016-09-30 08:51:25.040563 7f0f52f8c8c0  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
    -2> 2016-09-30 08:51:25.040568 7f0f52f8c8c0  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
    -1> 2016-09-30 08:51:25.040584 7f0f52f8c8c0  1 bluefs mount
     0> 2016-09-30 08:51:25.106090 7f0f52f8c8c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f0f52f8c8c0 time 2016-09-30 08:51:25.104252
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555e8167e240]
 2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
 3: (BlueFS::mount()+0x40a) [0x555e814a72ba]
 4: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
 5: (BlueStore::mount()+0x38d) [0x555e813f0aad]
 6: (OSD::init()+0x273) [0x555e80fefad3]
 7: (main()+0x2ea9) [0x555e80f3dcd9]
 8: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
 9: (_start()+0x29) [0x555e80f9b899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:25.108243 7f0f52f8c8c0 -1 *** Caught signal (Aborted) **
 in thread 7f0f52f8c8c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x555e814ddc1e]
 2: (()+0x113d0) [0x7f0f511d43d0]
 3: (gsignal()+0x38) [0x7f0f4f9e2418]
 4: (abort()+0x16a) [0x7f0f4f9e401a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555e8167e42b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
 7: (BlueFS::mount()+0x40a) [0x555e814a72ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
 9: (BlueStore::mount()+0x38d) [0x555e813f0aad]
 10: (OSD::init()+0x273) [0x555e80fefad3]
 11: (main()+0x2ea9) [0x555e80f3dcd9]
 12: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
 13: (_start()+0x29) [0x555e80f9b899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-09-30 08:51:25.108243 7f0f52f8c8c0 -1 *** Caught signal (Aborted) **
 in thread 7f0f52f8c8c0 thread_name:ceph-osd

 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
 1: (()+0x8d8c1e) [0x555e814ddc1e]
 2: (()+0x113d0) [0x7f0f511d43d0]
 3: (gsignal()+0x38) [0x7f0f4f9e2418]
 4: (abort()+0x16a) [0x7f0f4f9e401a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555e8167e42b]
 6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
 7: (BlueFS::mount()+0x40a) [0x555e814a72ba]
 8: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
 9: (BlueStore::mount()+0x38d) [0x555e813f0aad]
 10: (OSD::init()+0x273) [0x555e80fefad3]
 11: (main()+0x2ea9) [0x555e80f3dcd9]
 12: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
 13: (_start()+0x29) [0x555e80f9b899]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---

[global]
fsid = 34ba1a46-0936-4da5-9b8f-299c5e973ca3
mon_initial_members = dementor, banshee
mon_host = 10.14.65.176,10.14.65.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
enable_experimental_unrecoverable_data_corrupting_features = bluestore rocksdb
bluestore_default_buffered_read = true

mds cache size = 4000000
mon_clock_drift_allowed = .10
osd crush chooseleaf type = 3
osd op threads = 4

[mon]
        debug mon = 20

[osd]
	debug osd = 20

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




















[Index of Archives]
 
 
[Information on CEPH]
 
 
[Linux Filesystem Development]
 
 
[Ceph Development]
 
 
[Ceph Large]
 
 
[Ceph Dev]
 
 
[Linux USB Development]
 
 
[Video for Linux]
 
 
[Linux Audio Users]
 
 
[Yosemite News]
 
 
[Linux Kernel]
 
 
[Linux SCSI]
 
 
[xfs]

















 
Powered by Linux