can you find the full osdmap.16024 in other osd? seems like the osd::init doesnt read the incremental osdmap but the full osdmap, if you find it, then copy to osd.3. 2016-04-16 13:27 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: > I found below file missing on osd.3 so I copied over. Still fails with the > similar message. What can I try next? > > -1> 2016-04-16 00:22:32.579622 7f8d5c340800 20 osd.3 0 get_map 16024 - > loading and decoding 0x7f8d65d04900 > 0> 2016-04-16 00:22:32.584406 7f8d5c340800 -1 osd/OSD.h: In function > 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f8d5c340800 time 2016-04-16 > 00:22:32.579890 > osd/OSD.h: 885: FAILED assert(ret) > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x82) [0x7f8d5bdc64f2] > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f8d5b74d83d] > 3: (OSD::init()+0x1862) [0x7f8d5b6fba52] > 4: (main()+0x2b05) [0x7f8d5b661735] > 5: (__libc_start_main()+0xf5) [0x7f8d581f7b45] > 6: (()+0x337197) [0x7f8d5b6ac197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > Regards, > Hong > > > On Saturday, April 16, 2016 12:11 AM, hjcho616 <hjcho616@xxxxxxxxx> wrote: > > > Is this it? > > root@OSD2:/var/lib/ceph/osd/ceph-3/current/meta# find ./ | grep osdmap | > grep 16024 > ./DIR_E/DIR_3/inc\uosdmap.16024__0_46887E3E__none > > Regards, > Hong > > > On Friday, April 15, 2016 11:53 PM, huang jun <hjwsm1989@xxxxxxxxx> wrote: > > > First, you should check whether file osdmap.16024 exists in your > osd.3/current/meta dir, > if not, you can copy it from other OSD who has it. > > > 2016-04-16 12:36 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: >> Here is what I get wtih debug_osd = 20. >> >> 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 >> (ceph:ceph) >> 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 >> 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore empty >> --pid-file >> 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 >> ErasureCodePluginSelectJerasure: >> load: jerasure_sse3 >> 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc load: >> isa >> 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting >> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >> 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log #20901 >> 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 #20901 >> >> 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 #20900 >> >> 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot >> 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock >> sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 >> 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] >> lci=[16010,16024]) >> 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes >> 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found journal >> 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new class >> name >> journal 0x7f9caa628808 >> 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from >> /usr/lib/rados-classes/libcls_journal.so >> 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal status 3 >> 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method >> journal.create flags 3 0x7f9c8dadac00 >> 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method >> journal.get_order flags 1 0x7f9c8dada3c0 >> 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method >> journal.get_splay_width flags 1 0x7f9c8dada360 >> 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method >> journal.get_pool_id flags 1 0x7f9c8dadaa30 >> 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method >> journal.get_minimum_set flags 1 0x7f9c8dada9c0 >> 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method >> journal.set_minimum_set flags 3 0x7f9c8dada830 >> 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method >> journal.get_active_set flags 1 0x7f9c8dada7c0 >> 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method >> journal.set_active_set flags 3 0x7f9c8dada630 >> 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method >> journal.get_client flags 1 0x7f9c8dadafb0 >> 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method >> journal.client_register flags 3 0x7f9c8dadc140 >> 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_data flags 3 0x7f9c8dadb730 >> 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_state flags 3 0x7f9c8dadb300 >> 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method >> journal.client_unregister flags 3 0x7f9c8dadf060 >> 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method >> journal.client_commit flags 3 0x7f9c8dadbc40 >> 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method >> journal.client_list flags 1 0x7f9c8dadc9c0 >> 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method >> journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 >> 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method >> journal.get_tag flags 1 0x7f9c8dadf9b0 >> 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method >> journal.tag_create flags 3 0x7f9c8dadfd10 >> 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method >> journal.tag_list flags 1 0x7f9c8dadd290 >> 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method >> journal.guard_append flags 3 0x7f9c8dad97f0 >> 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal success >> 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found statelog >> 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new class >> name >> statelog 0x7f9caa628928 >> 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog from >> /usr/lib/rados-classes/libcls_statelog.so >> 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog status >> 3 >> 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method >> statelog.add >> flags 3 0x7f9c8d8cdb60 >> 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method >> statelog.list >> flags 1 0x7f9c8d8cdef0 >> 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method >> statelog.remove flags 3 0x7f9c8d8cd130 >> 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method >> statelog.check_state flags 1 0x7f9c8d8cc680 >> 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog success >> 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found cephfs >> 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new class >> name >> cephfs 0x7f9caa628a48 >> 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from >> /usr/lib/rados-classes/libcls_cephfs.so >> 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs status 3 >> 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method >> cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 >> 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs success >> 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found rbd >> 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new class >> name >> rbd 0x7f9caa628b68 >> 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from >> /usr/lib/rados-classes/libcls_rbd.so >> 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd status 3 >> 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method rbd.create >> flags 3 0x7f9c8d4a6be0 >> 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method >> rbd.get_features flags 1 0x7f9c8d4af550 >> 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method >> rbd.set_features flags 3 0x7f9c8d4a3390 >> 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method >> rbd.get_size >> flags 1 0x7f9c8d4af940 >> 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method >> rbd.set_size >> flags 3 0x7f9c8d4a2bf0 >> 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapcontext flags 1 0x7f9c8d4aa510 >> 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method >> rbd.get_object_prefix flags 1 0x7f9c8d4a0060 >> 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 >> 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_add flags 3 0x7f9c8d4b2600 >> 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_remove flags 3 0x7f9c8d4b2300 >> 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 >> 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method >> rbd.get_all_features flags 1 0x7f9c8d49d240 >> 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method rbd.copyup >> flags 3 0x7f9c8d49d380 >> 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method >> rbd.get_parent flags 1 0x7f9c8d4b1410 >> 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method >> rbd.set_parent flags 3 0x7f9c8d4a7560 >> 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_parent flags 3 0x7f9c8d4a8cb0 >> 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method >> rbd.set_protection_status flags 3 0x7f9c8d4b00b0 >> 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method >> rbd.get_protection_status flags 1 0x7f9c8d4afdd0 >> 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method >> rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 >> 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method >> rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 >> 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method >> rbd.get_flags >> flags 1 0x7f9c8d4b0880 >> 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method >> rbd.set_flags >> flags 3 0x7f9c8d4b0c30 >> 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_list flags 1 0x7f9c8d4ac290 >> 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_set flags 3 0x7f9c8d4ac9e0 >> 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_remove flags 3 0x7f9c8d49e330 >> 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_get flags 1 0x7f9c8d4a28f0 >> 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method >> rbd.add_child >> flags 3 0x7f9c8d4a9f90 >> 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_child flags 3 0x7f9c8d4abba0 >> 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method >> rbd.get_children flags 1 0x7f9c8d4a9c00 >> 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method rbd.get_id >> flags 1 0x7f9c8d4a2640 >> 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method rbd.set_id >> flags 3 0x7f9c8d4a5f30 >> 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_id flags 1 0x7f9c8d4a0920 >> 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_name flags 1 0x7f9c8d4a0c00 >> 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_list >> flags 1 0x7f9c8d4adc20 >> 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 >> 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_remove_image flags 3 0x7f9c8d4a0770 >> 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_rename_image flags 3 0x7f9c8d4ad770 >> 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_load flags 1 0x7f9c8d4a8090 >> 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_save flags 3 0x7f9c8d4a8710 >> 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_resize flags 3 0x7f9c8d4a81c0 >> 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_update flags 3 0x7f9c8d4a61b0 >> 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 >> 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 >> 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_list >> flags 1 0x7f9c8d4a7d90 >> 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_add >> flags 3 0x7f9c8d4a4230 >> 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_remove flags 3 0x7f9c8d4a52e0 >> 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_rename flags 3 0x7f9c8d4a4b00 >> 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 >> 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 >> 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 >> 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 >> 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 >> 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 >> 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 >> 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 >> 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 >> 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 >> 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 >> 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get flags 1 0x7f9c8d4a1400 >> 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_set flags 3 0x7f9c8d4a4050 >> 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 >> 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success >> 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found version >> 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new class >> name >> version 0x7f9caa628c88 >> 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from >> /usr/lib/rados-classes/libcls_version.so >> 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version status 3 >> 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method version.set >> flags 3 0x7f9c8d28eb60 >> 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method version.inc >> flags 3 0x7f9c8d28fdd0 >> 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method >> version.inc_conds flags 3 0x7f9c8d28fdd0 >> 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method >> version.read >> flags 1 0x7f9c8d28e880 >> 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method >> version.check_conds flags 1 0x7f9c8d28f410 >> 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version success >> 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found kvs >> 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new class >> name >> kvs 0x7f9caa628da8 >> 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from >> /usr/lib/rados-classes/libcls_kvs.so >> 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs status 3 >> 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method >> kvs.get_idata_from_key flags 1 0x7f9c8d07e990 >> 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method >> kvs.get_next_idata flags 1 0x7f9c8d080500 >> 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method >> kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 >> 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method >> kvs.read_many >> flags 1 0x7f9c8d078a40 >> 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method >> kvs.check_writable flags 3 0x7f9c8d0783a0 >> 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method >> kvs.assert_size_in_bound flags 2 0x7f9c8d078640 >> 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_insert flags 2 0x7f9c8d07a870 >> 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method >> kvs.create_with_omap flags 2 0x7f9c8d07c370 >> 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_remove flags 2 0x7f9c8d0799e0 >> 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method >> kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 >> 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success >> 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found hello >> 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new class >> name >> hello 0x7f9caa628fe8 >> 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from >> /usr/lib/rados-classes/libcls_hello.so >> 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> >> cls/hello/cls_hello.cc:305: >> loading cls_hello >> 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello status 3 >> 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method >> hello.say_hello flags 1 0x7f9c8ce6f8f0 >> 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method >> hello.record_hello flags 10 0x7f9c8ce70a50 >> 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method >> hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 >> 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method >> hello.replay >> flags 1 0x7f9c8ce6f880 >> 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method >> hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 >> 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method >> hello.bad_reader flags 2 0x7f9c8ce6f870 >> 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method >> hello.bad_writer flags 1 0x7f9c8ce6f860 >> 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello success >> 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found refcount >> 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new class >> name >> refcount 0x7f9caa629108 >> 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount from >> /usr/lib/rados-classes/libcls_refcount.so >> 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount status >> 3 >> 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method >> refcount.get >> flags 3 0x7f9c8cbc5e80 >> 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method >> refcount.put >> flags 3 0x7f9c8cbc50f0 >> 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method >> refcount.set >> flags 3 0x7f9c8cbc4060 >> 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method >> refcount.read >> flags 1 0x7f9c8cbc57b0 >> 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount success >> 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found user >> 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new class >> name >> user 0x7f9caa629228 >> 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from >> /usr/lib/rados-classes/libcls_user.so >> 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user status 3 >> 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method >> user.set_buckets_info flags 3 0x7f9c8c91edc0 >> 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method >> user.complete_stats_sync flags 3 0x7f9c8c91e5f0 >> 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method >> user.remove_bucket flags 3 0x7f9c8c91d410 >> 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method >> user.list_buckets flags 1 0x7f9c8c91ffb0 >> 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method >> user.get_header flags 1 0x7f9c8c91bee0 >> 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user success >> 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found numops >> 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new class >> name >> numops 0x7f9caa629348 >> 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from >> /usr/lib/rados-classes/libcls_numops.so >> 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops status 3 >> 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method numops.add >> flags 3 0x7f9c8c716860 >> 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method numops.mul >> flags 3 0x7f9c8c715cc0 >> 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops success >> 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found >> replica_log >> 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new class >> name >> replica_log 0x7f9caa629468 >> 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log from >> /usr/lib/rados-classes/libcls_replica_log.so >> 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class replica_log >> status >> 3 >> 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method >> replica_log.set flags 3 0x7f9c8c50eed0 >> 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method >> replica_log.get flags 1 0x7f9c8c50fed0 >> 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method >> replica_log.delete flags 3 0x7f9c8c50f670 >> 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log success >> 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found >> timeindex >> 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new class >> name >> timeindex 0x7f9caa629588 >> 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex from >> /usr/lib/rados-classes/libcls_timeindex.so >> 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class timeindex status >> 3 >> 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method >> timeindex.add >> flags 3 0x7f9c8c306690 >> 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method >> timeindex.list flags 1 0x7f9c8c307200 >> 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method >> timeindex.trim flags 3 0x7f9c8c3068d0 >> 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex success >> 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found lock >> 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new class >> name >> lock 0x7f9caa6296a8 >> 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from >> /usr/lib/rados-classes/libcls_lock.so >> 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock status 3 >> 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method lock.lock >> flags 11 0x7f9c8c0fdb80 >> 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method lock.unlock >> flags 11 0x7f9c8c0fd560 >> 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method >> lock.break_lock flags 3 0x7f9c8c0fcf50 >> 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method >> lock.get_info >> flags 1 0x7f9c8c0fbc20 >> 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method >> lock.list_locks flags 1 0x7f9c8c0fa770 >> 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method >> lock.assert_locked flags 9 0x7f9c8c0fb180 >> 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock success >> 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found log >> 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new class >> name >> log 0x7f9caa6298e8 >> 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from >> /usr/lib/rados-classes/libcls_log.so >> 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log status 3 >> 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method log.add >> flags >> 3 0x7f9c8bef0320 >> 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method log.list >> flags 1 0x7f9c8bef1660 >> 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method log.trim >> flags 3 0x7f9c8bef0d30 >> 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method log.info >> flags 1 0x7f9c8beefc70 >> 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success >> 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found rgw >> 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new class >> name >> rgw 0x7f9caa629a08 >> 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from >> /usr/lib/rados-classes/libcls_rgw.so >> 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw status 3 >> 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_init_index flags 3 0x7f9c8bc0c620 >> 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 >> 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_list flags 1 0x7f9c8bc145e0 >> 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_check_index flags 1 0x7f9c8bc164a0 >> 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 >> 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 >> 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 >> 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 >> 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 >> 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 >> 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 >> 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 >> 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_remove flags 3 0x7f9c8bc138b0 >> 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 >> 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 >> 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 >> 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method rgw.bi_get >> flags 1 0x7f9c8bc0aae0 >> 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method rgw.bi_put >> flags 3 0x7f9c8bc08fc0 >> 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method rgw.bi_list >> flags 1 0x7f9c8bc11ca0 >> 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_list flags 1 0x7f9c8bc0e790 >> 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 >> 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method >> rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 >> 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 >> 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 >> 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 >> 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 >> 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 >> 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method rgw.gc_list >> flags 1 0x7f9c8bc0cff0 >> 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_remove >> flags 3 0x7f9c8bc0c320 >> 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success >> 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - loading >> and decoding 0x7f9caa5f2900 >> 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function >> 'OSDMapRef >> OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time 2016-04-15 >> 23:28:24.870922 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f9ca04e14f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 4: (main()+0x2b05) [0x7f9c9fd7c735] >> 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 6: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> -322> 2016-04-15 23:28:24.401110 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perfcounters_dump hook 0x7f9caa53e030 >> -321> 2016-04-15 23:28:24.401210 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 1 hook 0x7f9caa53e030 >> -320> 2016-04-15 23:28:24.401241 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf dump hook 0x7f9caa53e030 >> -319> 2016-04-15 23:28:24.401270 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perfcounters_schema hook 0x7f9caa53e030 >> -318> 2016-04-15 23:28:24.401296 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 2 hook 0x7f9caa53e030 >> -317> 2016-04-15 23:28:24.401323 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf schema hook 0x7f9caa53e030 >> -316> 2016-04-15 23:28:24.401352 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf reset hook 0x7f9caa53e030 >> -315> 2016-04-15 23:28:24.401380 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config show hook 0x7f9caa53e030 >> -314> 2016-04-15 23:28:24.401411 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config set hook 0x7f9caa53e030 >> -313> 2016-04-15 23:28:24.401438 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config get hook 0x7f9caa53e030 >> -312> 2016-04-15 23:28:24.401468 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config diff hook 0x7f9caa53e030 >> -311> 2016-04-15 23:28:24.401495 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log flush hook 0x7f9caa53e030 >> -310> 2016-04-15 23:28:24.401522 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log dump hook 0x7f9caa53e030 >> -309> 2016-04-15 23:28:24.401549 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log reopen hook 0x7f9caa53e030 >> -308> 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 >> (ceph:ceph) >> -307> 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 >> -306> 2016-04-15 23:28:24.431394 7f9ca0a5b800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -305> 2016-04-15 23:28:24.431450 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6800/2092 need_addr=0 >> -304> 2016-04-15 23:28:24.431517 7f9ca0a5b800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -303> 2016-04-15 23:28:24.431554 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6800/2092 need_addr=0 >> -302> 2016-04-15 23:28:24.431619 7f9ca0a5b800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -301> 2016-04-15 23:28:24.431877 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6801/2092 need_addr=0 >> -300> 2016-04-15 23:28:24.431974 7f9ca0a5b800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -299> 2016-04-15 23:28:24.432008 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6801/2092 need_addr=0 >> -298> 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore >> empty --pid-file >> -297> 2016-04-15 23:28:24.440611 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> init >> /var/run/ceph/ceph-osd.3.asok >> -296> 2016-04-15 23:28:24.440650 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> bind_and_listen /var/run/ceph/ceph-osd.3.asok >> -295> 2016-04-15 23:28:24.440983 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 0 hook 0x7f9caa53a0d8 >> -294> 2016-04-15 23:28:24.441018 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command version hook 0x7f9caa53a0d8 >> -293> 2016-04-15 23:28:24.441038 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command git_version hook 0x7f9caa53a0d8 >> -292> 2016-04-15 23:28:24.441062 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command help hook 0x7f9caa53e1e0 >> -291> 2016-04-15 23:28:24.441084 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command get_command_descriptions hook 0x7f9caa53e1f0 >> -290> 2016-04-15 23:28:24.441316 7f9c99274700 5 asok(0x7f9caa5f6000) >> entry start >> -289> 2016-04-15 23:28:24.441424 7f9ca0a5b800 10 monclient(hunting): >> build_initial_monmap >> -288> 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 >> ErasureCodePluginSelectJerasure: load: jerasure_sse3 >> -287> 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc >> load: isa >> -286> 2016-04-15 23:28:24.470118 7f9ca0a5b800 5 adding auth protocol: >> cephx >> -285> 2016-04-15 23:28:24.470133 7f9ca0a5b800 5 adding auth protocol: >> cephx >> -284> 2016-04-15 23:28:24.470946 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command objecter_requests hook 0x7f9caa53e2b0 >> -283> 2016-04-15 23:28:24.471128 7f9ca0a5b800 1 -- >> 192.168.1.31:6800/2092 >> messenger.start >> -282> 2016-04-15 23:28:24.471264 7f9ca0a5b800 1 -- :/0 messenger.start >> -281> 2016-04-15 23:28:24.471373 7f9ca0a5b800 1 -- >> 192.168.1.31:6801/2092 >> messenger.start >> -280> 2016-04-15 23:28:24.471482 7f9ca0a5b800 1 -- >> 192.168.2.31:6801/2092 >> messenger.start >> -279> 2016-04-15 23:28:24.471583 7f9ca0a5b800 1 -- >> 192.168.2.31:6800/2092 >> messenger.start >> -278> 2016-04-15 23:28:24.471680 7f9ca0a5b800 1 -- :/0 messenger.start >> -277> 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting >> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >> -276> 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> -275> 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> -274> 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> -273> 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> -272> 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> -271> 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> -270> 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log >> #20901 >> -269> 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 >> #20901 >> >> -268> 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 >> #20900 >> >> -267> 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> -266> 2016-04-15 23:28:24.792679 7f9ca0a5b800 2 journal open >> /var/lib/ceph/osd/ceph-3/journal fsid 4f86a418-6c67-4cb4-83a1-6c123c890036 >> fs_op_seq 9829589 >> -265> 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -264> 2016-04-15 23:28:24.794982 7f9ca0a5b800 2 journal open advancing >> committed_seq 9829584 to fs op_seq 9829589 >> -263> 2016-04-15 23:28:24.796869 7f9ca0a5b800 2 journal read_entry >> 2537717760 : seq 9829585 29509 bytes >> -262> 2016-04-15 23:28:24.797943 7f9ca0a5b800 2 journal read_entry >> 2537750528 : seq 9829586 8134 bytes >> -261> 2016-04-15 23:28:24.798617 7f9ca0a5b800 2 journal read_entry >> 2537762816 : seq 9829587 3064 bytes >> -260> 2016-04-15 23:28:24.798938 7f9ca0a5b800 2 journal read_entry >> 2537766912 : seq 9829588 7647 bytes >> -259> 2016-04-15 23:28:24.799208 7f9ca0a5b800 2 journal read_entry >> 2537775104 : seq 9829589 4737 bytes >> -258> 2016-04-15 23:28:24.799373 7f9ca0a5b800 2 journal No further valid >> entries found, journal is most likely valid >> -257> 2016-04-15 23:28:24.799478 7f9ca0a5b800 2 journal No further valid >> entries found, journal is most likely valid >> -256> 2016-04-15 23:28:24.799542 7f9ca0a5b800 3 journal journal_replay: >> end of journal, done. >> -255> 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -254> 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> -253> 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot >> -252> 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock >> sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 >> 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] >> lci=[16010,16024]) >> -251> 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes >> -250> 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found >> journal >> -249> 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new >> class name journal 0x7f9caa628808 >> -248> 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from >> /usr/lib/rados-classes/libcls_journal.so >> -247> 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal >> status 3 >> -246> 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method >> journal.create flags 3 0x7f9c8dadac00 >> -245> 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method >> journal.get_order flags 1 0x7f9c8dada3c0 >> -244> 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method >> journal.get_splay_width flags 1 0x7f9c8dada360 >> -243> 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method >> journal.get_pool_id flags 1 0x7f9c8dadaa30 >> -242> 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method >> journal.get_minimum_set flags 1 0x7f9c8dada9c0 >> -241> 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method >> journal.set_minimum_set flags 3 0x7f9c8dada830 >> -240> 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method >> journal.get_active_set flags 1 0x7f9c8dada7c0 >> -239> 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method >> journal.set_active_set flags 3 0x7f9c8dada630 >> -238> 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method >> journal.get_client flags 1 0x7f9c8dadafb0 >> -237> 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method >> journal.client_register flags 3 0x7f9c8dadc140 >> -236> 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_data flags 3 0x7f9c8dadb730 >> -235> 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_state flags 3 0x7f9c8dadb300 >> -234> 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method >> journal.client_unregister flags 3 0x7f9c8dadf060 >> -233> 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method >> journal.client_commit flags 3 0x7f9c8dadbc40 >> -232> 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method >> journal.client_list flags 1 0x7f9c8dadc9c0 >> -231> 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method >> journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 >> -230> 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method >> journal.get_tag flags 1 0x7f9c8dadf9b0 >> -229> 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method >> journal.tag_create flags 3 0x7f9c8dadfd10 >> -228> 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method >> journal.tag_list flags 1 0x7f9c8dadd290 >> -227> 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method >> journal.guard_append flags 3 0x7f9c8dad97f0 >> -226> 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal >> success >> -225> 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found >> statelog >> -224> 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new >> class name statelog 0x7f9caa628928 >> -223> 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog >> from >> /usr/lib/rados-classes/libcls_statelog.so >> -222> 2016-04-15 23:28:24.818836 7f9ca0a5b800 1 <cls> >> cls/statelog/cls_statelog.cc:306: Loaded log class! >> -221> 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog >> status 3 >> -220> 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method >> statelog.add flags 3 0x7f9c8d8cdb60 >> -219> 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method >> statelog.list flags 1 0x7f9c8d8cdef0 >> -218> 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method >> statelog.remove flags 3 0x7f9c8d8cd130 >> -217> 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method >> statelog.check_state flags 1 0x7f9c8d8cc680 >> -216> 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog >> success >> -215> 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found >> cephfs >> -214> 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new >> class name cephfs 0x7f9caa628a48 >> -213> 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from >> /usr/lib/rados-classes/libcls_cephfs.so >> -212> 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> -211> 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs >> status 3 >> -210> 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method >> cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 >> -209> 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs >> success >> -208> 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found >> rbd >> -207> 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new >> class name rbd 0x7f9caa628b68 >> -206> 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from >> /usr/lib/rados-classes/libcls_rbd.so >> -205> 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd >> status >> 3 >> -204> 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method >> rbd.create flags 3 0x7f9c8d4a6be0 >> -203> 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method >> rbd.get_features flags 1 0x7f9c8d4af550 >> -202> 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method >> rbd.set_features flags 3 0x7f9c8d4a3390 >> -201> 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method >> rbd.get_size flags 1 0x7f9c8d4af940 >> -200> 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method >> rbd.set_size flags 3 0x7f9c8d4a2bf0 >> -199> 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapcontext flags 1 0x7f9c8d4aa510 >> -198> 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method >> rbd.get_object_prefix flags 1 0x7f9c8d4a0060 >> -197> 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 >> -196> 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_add flags 3 0x7f9c8d4b2600 >> -195> 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_remove flags 3 0x7f9c8d4b2300 >> -194> 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 >> -193> 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method >> rbd.get_all_features flags 1 0x7f9c8d49d240 >> -192> 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method >> rbd.copyup flags 3 0x7f9c8d49d380 >> -191> 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method >> rbd.get_parent flags 1 0x7f9c8d4b1410 >> -190> 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method >> rbd.set_parent flags 3 0x7f9c8d4a7560 >> -189> 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_parent flags 3 0x7f9c8d4a8cb0 >> -188> 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method >> rbd.set_protection_status flags 3 0x7f9c8d4b00b0 >> -187> 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method >> rbd.get_protection_status flags 1 0x7f9c8d4afdd0 >> -186> 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method >> rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 >> -185> 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method >> rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 >> -184> 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method >> rbd.get_flags flags 1 0x7f9c8d4b0880 >> -183> 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method >> rbd.set_flags flags 3 0x7f9c8d4b0c30 >> -182> 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_list flags 1 0x7f9c8d4ac290 >> -181> 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_set flags 3 0x7f9c8d4ac9e0 >> -180> 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_remove flags 3 0x7f9c8d49e330 >> -179> 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_get flags 1 0x7f9c8d4a28f0 >> -178> 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method >> rbd.add_child flags 3 0x7f9c8d4a9f90 >> -177> 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_child flags 3 0x7f9c8d4abba0 >> -176> 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method >> rbd.get_children flags 1 0x7f9c8d4a9c00 >> -175> 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method >> rbd.get_id flags 1 0x7f9c8d4a2640 >> -174> 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method >> rbd.set_id flags 3 0x7f9c8d4a5f30 >> -173> 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_id flags 1 0x7f9c8d4a0920 >> -172> 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_name flags 1 0x7f9c8d4a0c00 >> -171> 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_list flags 1 0x7f9c8d4adc20 >> -170> 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 >> -169> 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_remove_image flags 3 0x7f9c8d4a0770 >> -168> 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_rename_image flags 3 0x7f9c8d4ad770 >> -167> 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_load flags 1 0x7f9c8d4a8090 >> -166> 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_save flags 3 0x7f9c8d4a8710 >> -165> 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_resize flags 3 0x7f9c8d4a81c0 >> -164> 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_update flags 3 0x7f9c8d4a61b0 >> -163> 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 >> -162> 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 >> -161> 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_list flags 1 0x7f9c8d4a7d90 >> -160> 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_add flags 3 0x7f9c8d4a4230 >> -159> 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_remove flags 3 0x7f9c8d4a52e0 >> -158> 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_rename flags 3 0x7f9c8d4a4b00 >> -157> 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 >> -156> 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 >> -155> 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 >> -154> 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 >> -153> 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 >> -152> 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 >> -151> 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 >> -150> 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 >> -149> 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 >> -148> 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 >> -147> 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 >> -146> 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get flags 1 0x7f9c8d4a1400 >> -145> 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_set flags 3 0x7f9c8d4a4050 >> -144> 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 >> -143> 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success >> -142> 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found >> version >> -141> 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new >> class name version 0x7f9caa628c88 >> -140> 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from >> /usr/lib/rados-classes/libcls_version.so >> -139> 2016-04-15 23:28:24.833724 7f9ca0a5b800 1 <cls> >> cls/version/cls_version.cc:228: Loaded version class! >> -138> 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version >> status 3 >> -137> 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method >> version.set flags 3 0x7f9c8d28eb60 >> -136> 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method >> version.inc flags 3 0x7f9c8d28fdd0 >> -135> 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method >> version.inc_conds flags 3 0x7f9c8d28fdd0 >> -134> 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method >> version.read flags 1 0x7f9c8d28e880 >> -133> 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method >> version.check_conds flags 1 0x7f9c8d28f410 >> -132> 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version >> success >> -131> 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found >> kvs >> -130> 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new >> class name kvs 0x7f9caa628da8 >> -129> 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from >> /usr/lib/rados-classes/libcls_kvs.so >> -128> 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs >> status >> 3 >> -127> 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method >> kvs.get_idata_from_key flags 1 0x7f9c8d07e990 >> -126> 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method >> kvs.get_next_idata flags 1 0x7f9c8d080500 >> -125> 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method >> kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 >> -124> 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method >> kvs.read_many flags 1 0x7f9c8d078a40 >> -123> 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method >> kvs.check_writable flags 3 0x7f9c8d0783a0 >> -122> 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method >> kvs.assert_size_in_bound flags 2 0x7f9c8d078640 >> -121> 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_insert flags 2 0x7f9c8d07a870 >> -120> 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method >> kvs.create_with_omap flags 2 0x7f9c8d07c370 >> -119> 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_remove flags 2 0x7f9c8d0799e0 >> -118> 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method >> kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 >> -117> 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success >> -116> 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found >> hello >> -115> 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new >> class name hello 0x7f9caa628fe8 >> -114> 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from >> /usr/lib/rados-classes/libcls_hello.so >> -113> 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> >> cls/hello/cls_hello.cc:305: loading cls_hello >> -112> 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello >> status 3 >> -111> 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method >> hello.say_hello flags 1 0x7f9c8ce6f8f0 >> -110> 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method >> hello.record_hello flags 10 0x7f9c8ce70a50 >> -109> 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method >> hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 >> -108> 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method >> hello.replay flags 1 0x7f9c8ce6f880 >> -107> 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method >> hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 >> -106> 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method >> hello.bad_reader flags 2 0x7f9c8ce6f870 >> -105> 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method >> hello.bad_writer flags 1 0x7f9c8ce6f860 >> -104> 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello >> success >> -103> 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found >> refcount >> -102> 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new >> class name refcount 0x7f9caa629108 >> -101> 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount >> from >> /usr/lib/rados-classes/libcls_refcount.so >> -100> 2016-04-15 23:28:24.849726 7f9ca0a5b800 1 <cls> >> cls/refcount/cls_refcount.cc:232: Loaded refcount class! >> -99> 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount >> status 3 >> -98> 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method >> refcount.get flags 3 0x7f9c8cbc5e80 >> -97> 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method >> refcount.put flags 3 0x7f9c8cbc50f0 >> -96> 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method >> refcount.set flags 3 0x7f9c8cbc4060 >> -95> 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method >> refcount.read flags 1 0x7f9c8cbc57b0 >> -94> 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount >> success >> -93> 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found >> user >> -92> 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new >> class name user 0x7f9caa629228 >> -91> 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from >> /usr/lib/rados-classes/libcls_user.so >> -90> 2016-04-15 23:28:24.851663 7f9ca0a5b800 1 <cls> >> cls/user/cls_user.cc:375: Loaded user class! >> -89> 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user >> status 3 >> -88> 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method >> user.set_buckets_info flags 3 0x7f9c8c91edc0 >> -87> 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method >> user.complete_stats_sync flags 3 0x7f9c8c91e5f0 >> -86> 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method >> user.remove_bucket flags 3 0x7f9c8c91d410 >> -85> 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method >> user.list_buckets flags 1 0x7f9c8c91ffb0 >> -84> 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method >> user.get_header flags 1 0x7f9c8c91bee0 >> -83> 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user >> success >> -82> 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found >> numops >> -81> 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new >> class name numops 0x7f9caa629348 >> -80> 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from >> /usr/lib/rados-classes/libcls_numops.so >> -79> 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops >> status 3 >> -78> 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method >> numops.add flags 3 0x7f9c8c716860 >> -77> 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method >> numops.mul flags 3 0x7f9c8c715cc0 >> -76> 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops >> success >> -75> 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found >> replica_log >> -74> 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new >> class name replica_log 0x7f9caa629468 >> -73> 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log >> from /usr/lib/rados-classes/libcls_replica_log.so >> -72> 2016-04-15 23:28:24.854977 7f9ca0a5b800 1 <cls> >> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! >> -71> 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class >> replica_log status 3 >> -70> 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method >> replica_log.set flags 3 0x7f9c8c50eed0 >> -69> 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method >> replica_log.get flags 1 0x7f9c8c50fed0 >> -68> 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method >> replica_log.delete flags 3 0x7f9c8c50f670 >> -67> 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log >> success >> -66> 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found >> timeindex >> -65> 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new >> class name timeindex 0x7f9caa629588 >> -64> 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex >> from /usr/lib/rados-classes/libcls_timeindex.so >> -63> 2016-04-15 23:28:24.856621 7f9ca0a5b800 1 <cls> >> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! >> -62> 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class >> timeindex >> status 3 >> -61> 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method >> timeindex.add flags 3 0x7f9c8c306690 >> -60> 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method >> timeindex.list flags 1 0x7f9c8c307200 >> -59> 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method >> timeindex.trim flags 3 0x7f9c8c3068d0 >> -58> 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex >> success >> -57> 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found >> lock >> -56> 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new >> class name lock 0x7f9caa6296a8 >> -55> 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from >> /usr/lib/rados-classes/libcls_lock.so >> -54> 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock >> status 3 >> -53> 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method >> lock.lock flags 11 0x7f9c8c0fdb80 >> -52> 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method >> lock.unlock flags 11 0x7f9c8c0fd560 >> -51> 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method >> lock.break_lock flags 3 0x7f9c8c0fcf50 >> -50> 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method >> lock.get_info flags 1 0x7f9c8c0fbc20 >> -49> 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method >> lock.list_locks flags 1 0x7f9c8c0fa770 >> -48> 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method >> lock.assert_locked flags 9 0x7f9c8c0fb180 >> -47> 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock >> success >> -46> 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found >> log >> -45> 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new >> class name log 0x7f9caa6298e8 >> -44> 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from >> /usr/lib/rados-classes/libcls_log.so >> -43> 2016-04-15 23:28:24.860294 7f9ca0a5b800 1 <cls> >> cls/log/cls_log.cc:317: Loaded log class! >> -42> 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log >> status >> 3 >> -41> 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method >> log.add flags 3 0x7f9c8bef0320 >> -40> 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method >> log.list flags 1 0x7f9c8bef1660 >> -39> 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method >> log.trim flags 3 0x7f9c8bef0d30 >> -38> 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method >> log.info flags 1 0x7f9c8beefc70 >> -37> 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success >> -36> 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found >> rgw >> -35> 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new >> class name rgw 0x7f9caa629a08 >> -34> 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from >> /usr/lib/rados-classes/libcls_rgw.so >> -33> 2016-04-15 23:28:24.870017 7f9ca0a5b800 1 <cls> >> cls/rgw/cls_rgw.cc:3206: Loaded rgw class! >> -32> 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw >> status >> 3 >> -31> 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_init_index flags 3 0x7f9c8bc0c620 >> -30> 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 >> -29> 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_list flags 1 0x7f9c8bc145e0 >> -28> 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_check_index flags 1 0x7f9c8bc164a0 >> -27> 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 >> -26> 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 >> -25> 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 >> -24> 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 >> -23> 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 >> -22> 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 >> -21> 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 >> -20> 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 >> -19> 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_remove flags 3 0x7f9c8bc138b0 >> -18> 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 >> -17> 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 >> -16> 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 >> -15> 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_get flags 1 0x7f9c8bc0aae0 >> -14> 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_put flags 3 0x7f9c8bc08fc0 >> -13> 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_list flags 1 0x7f9c8bc11ca0 >> -12> 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_list flags 1 0x7f9c8bc0e790 >> -11> 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 >> -10> 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method >> rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 >> -9> 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 >> -8> 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 >> -7> 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 >> -6> 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 >> -5> 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 >> -4> 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_list flags 1 0x7f9c8bc0cff0 >> -3> 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_remove flags 3 0x7f9c8bc0c320 >> -2> 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success >> -1> 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - >> loading and decoding 0x7f9caa5f2900 >> 0> 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function >> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time >> 2016-04-15 >> 23:28:24.870922 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f9ca04e14f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 4: (main()+0x2b05) [0x7f9c9fd7c735] >> 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 6: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 20/20 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal (Aborted) ** >> in thread 7f9ca0a5b800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f9ca03d9117] >> 2: (()+0xf8d0) [0x7f9c9e8dc8d0] >> 3: (gsignal()+0x37) [0x7f9c9c926067] >> 4: (abort()+0x148) [0x7f9c9c927448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f9ca04e16c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 8: (main()+0x2b05) [0x7f9c9fd7c735] >> 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 10: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> 0> 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal >> (Aborted) ** >> in thread 7f9ca0a5b800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f9ca03d9117] >> 2: (()+0xf8d0) [0x7f9c9e8dc8d0] >> 3: (gsignal()+0x37) [0x7f9c9c926067] >> 4: (abort()+0x148) [0x7f9c9c927448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f9ca04e16c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 8: (main()+0x2b05) [0x7f9c9fd7c735] >> 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 10: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 20/20 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> >> >> >> On Friday, April 15, 2016 11:24 PM, huang jun <hjwsm1989@xxxxxxxxx> wrote: >> >> >> Can you set 'debug_osd = 20' in ceph.conf and restart the osd again, >> and post the corrupt log. >> I doubt it's problem related to "0 byte osdmap" decode problem? >> >> 2016-04-16 12:14 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: >>> I've been successfully running cephfs on my Debian Jessies for a while >>> and >>> one day after power outage, MDS wasn't happy. MDS crashing after it was >>> done loading, increasing the memory utilization quite a bit. I was >>> running >>> infernalis 9.2.0 and did successful upgrade from Hammer before... so I >>> thought I may have hit a bug and decided to try 9.2.1. >>> >>> In 9.2.1, it was not happy that my journal didn't have permission for >>> user >>> ceph. So corrected it. Then all of my OSDs are no longer starting. >>> Failing with similar messages as below. I upgraded to Jewel, as I didn't >>> see too much more complexitiy to upgrade from Infernalis and am still >>> seeing >>> these errors. >>> >>> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to 1001:1001 >>> (ceph:ceph) >>> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >>> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >>> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore empty >>> --pid-file >>> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >>> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >>> ioctl is disabled via 'filestore fiemap' config option >>> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >>> option >>> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >>> is >>> supported >>> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> syncfs(2) >>> syscall fully supported (by glibc and kernel) >>> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >>> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >>> disabled by conf >>> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log #20899 >>> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 #20899 >>> >>> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 #20898 >>> >>> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >>> mode: >>> checkpoint is not enabled >>> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >>> filestore(/var/lib/ceph/osd/ceph-3) upgrade >>> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >>> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >>> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >>> cls/hello/cls_hello.cc:305: >>> loading cls_hello >>> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >>> 'OSDMapRef >>> OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time 2016-04-15 >>> 22:47:06.494680 >>> osd/OSD.h: 885: FAILED assert(ret) >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x82) [0x7f65fb6364f2] >>> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 4: (main()+0x2b05) [0x7f65faed1735] >>> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 6: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- begin dump of recent events --- >>> -78> 2016-04-15 22:47:04.873688 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_dump hook 0x7f66067e2030 >>> -77> 2016-04-15 22:47:04.873771 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 1 hook 0x7f66067e2030 >>> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf dump hook 0x7f66067e2030 >>> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_schema hook 0x7f66067e2030 >>> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf dump hook 0x7f66067e2030 >>> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_schema hook 0x7f66067e2030 >>> -74> 2016-04-15 22:47:04.873860 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 2 hook 0x7f66067e2030 >>> -73> 2016-04-15 22:47:04.873886 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf schema hook 0x7f66067e2030 >>> -72> 2016-04-15 22:47:04.873916 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf reset hook 0x7f66067e2030 >>> -71> 2016-04-15 22:47:04.873943 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config show hook 0x7f66067e2030 >>> -70> 2016-04-15 22:47:04.873974 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config set hook 0x7f66067e2030 >>> -69> 2016-04-15 22:47:04.874000 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config get hook 0x7f66067e2030 >>> -68> 2016-04-15 22:47:04.874029 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config diff hook 0x7f66067e2030 >>> -67> 2016-04-15 22:47:04.874055 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log flush hook 0x7f66067e2030 >>> -66> 2016-04-15 22:47:04.874082 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log dump hook 0x7f66067e2030 >>> -65> 2016-04-15 22:47:04.874109 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log reopen hook 0x7f66067e2030 >>> -64> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to >>> 1001:1001 >>> (ceph:ceph) >>> -63> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >>> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >>> -62> 2016-04-15 22:47:04.900224 7f65fbbb0800 1 -- 192.168.1.31:0/0 >>> learned my addr 192.168.1.31:0/0 >>> -61> 2016-04-15 22:47:04.900286 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.1.31:6802/1284 need_addr=0 >>> -60> 2016-04-15 22:47:04.900350 7f65fbbb0800 1 -- 192.168.2.31:0/0 >>> learned my addr 192.168.2.31:0/0 >>> -59> 2016-04-15 22:47:04.900375 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.2.31:6802/1284 need_addr=0 >>> -58> 2016-04-15 22:47:04.900443 7f65fbbb0800 1 -- 192.168.2.31:0/0 >>> learned my addr 192.168.2.31:0/0 >>> -57> 2016-04-15 22:47:04.900475 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.2.31:6803/1284 need_addr=0 >>> -56> 2016-04-15 22:47:04.900538 7f65fbbb0800 1 -- 192.168.1.31:0/0 >>> learned my addr 192.168.1.31:0/0 >>> -55> 2016-04-15 22:47:04.900562 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.1.31:6803/1284 need_addr=0 >>> -54> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore >>> empty --pid-file >>> -53> 2016-04-15 22:47:04.909743 7f65fbbb0800 5 asok(0x7f660689a000) >>> init >>> /var/run/ceph/ceph-osd.3.asok >>> -52> 2016-04-15 22:47:04.909792 7f65fbbb0800 5 asok(0x7f660689a000) >>> bind_and_listen /var/run/ceph/ceph-osd.3.asok >>> -51> 2016-04-15 22:47:04.909891 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 0 hook 0x7f66067de0d8 >>> -50> 2016-04-15 22:47:04.909928 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command version hook 0x7f66067de0d8 >>> -49> 2016-04-15 22:47:04.909955 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command git_version hook 0x7f66067de0d8 >>> -48> 2016-04-15 22:47:04.909988 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command help hook 0x7f66067e21e0 >>> -47> 2016-04-15 22:47:04.910015 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command get_command_descriptions hook 0x7f66067e21f0 >>> -46> 2016-04-15 22:47:04.910205 7f65f43c9700 5 asok(0x7f660689a000) >>> entry start >>> -45> 2016-04-15 22:47:04.910330 7f65fbbb0800 10 monclient(hunting): >>> build_initial_monmap >>> -44> 2016-04-15 22:47:04.939070 7f65fbbb0800 5 adding auth protocol: >>> cephx >>> -43> 2016-04-15 22:47:04.939118 7f65fbbb0800 5 adding auth protocol: >>> cephx >>> -42> 2016-04-15 22:47:04.939986 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command objecter_requests hook 0x7f66067e22b0 >>> -41> 2016-04-15 22:47:04.940256 7f65fbbb0800 1 -- >>> 192.168.1.31:6802/1284 >>> messenger.start >>> -40> 2016-04-15 22:47:04.940413 7f65fbbb0800 1 -- :/0 messenger.start >>> -39> 2016-04-15 22:47:04.940557 7f65fbbb0800 1 -- >>> 192.168.1.31:6803/1284 >>> messenger.start >>> -38> 2016-04-15 22:47:04.940686 7f65fbbb0800 1 -- >>> 192.168.2.31:6803/1284 >>> messenger.start >>> -37> 2016-04-15 22:47:04.940798 7f65fbbb0800 1 -- >>> 192.168.2.31:6802/1284 >>> messenger.start >>> -36> 2016-04-15 22:47:04.940899 7f65fbbb0800 1 -- :/0 messenger.start >>> -35> 2016-04-15 22:47:04.941223 7f65fbbb0800 2 osd.3 0 mounting >>> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >>> -34> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >>> -33> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >>> ioctl is disabled via 'filestore fiemap' config option >>> -32> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >>> option >>> -31> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >>> is >>> supported >>> -30> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> syncfs(2) >>> syscall fully supported (by glibc and kernel) >>> -29> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >>> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >>> disabled by conf >>> -28> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering >>> log >>> #20899 >>> -27> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 >>> #20899 >>> >>> -26> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 >>> #20898 >>> >>> -25> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >>> mode: >>> checkpoint is not enabled >>> -24> 2016-04-15 22:47:06.377715 7f65fbbb0800 2 journal open >>> /var/lib/ceph/osd/ceph-3/journal fsid >>> 4f86a418-6c67-4cb4-83a1-6c123c890036 >>> fs_op_seq 9829589 >>> -23> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> -22> 2016-04-15 22:47:06.379811 7f65fbbb0800 2 journal open advancing >>> committed_seq 9829584 to fs op_seq 9829589 >>> -21> 2016-04-15 22:47:06.380757 7f65fbbb0800 2 journal read_entry >>> 2537717760 : seq 9829585 29509 bytes >>> -20> 2016-04-15 22:47:06.380996 7f65fbbb0800 2 journal read_entry >>> 2537750528 : seq 9829586 8134 bytes >>> -19> 2016-04-15 22:47:06.381091 7f65fbbb0800 2 journal read_entry >>> 2537762816 : seq 9829587 3064 bytes >>> -18> 2016-04-15 22:47:06.381155 7f65fbbb0800 2 journal read_entry >>> 2537766912 : seq 9829588 7647 bytes >>> -17> 2016-04-15 22:47:06.381219 7f65fbbb0800 2 journal read_entry >>> 2537775104 : seq 9829589 4737 bytes >>> -16> 2016-04-15 22:47:06.381257 7f65fbbb0800 2 journal No further >>> valid >>> entries found, journal is most likely valid >>> -15> 2016-04-15 22:47:06.381287 7f65fbbb0800 2 journal No further >>> valid >>> entries found, journal is most likely valid >>> -14> 2016-04-15 22:47:06.381302 7f65fbbb0800 3 journal >>> journal_replay: >>> end of journal, done. >>> -13> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> -12> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >>> filestore(/var/lib/ceph/osd/ceph-3) upgrade >>> -11> 2016-04-15 22:47:06.385071 7f65fbbb0800 2 osd.3 0 boot >>> -10> 2016-04-15 22:47:06.415253 7f65fbbb0800 1 <cls> >>> cls/statelog/cls_statelog.cc:306: Loaded log class! >>> -9> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >>> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >>> -8> 2016-04-15 22:47:06.418172 7f65fbbb0800 1 <cls> >>> cls/version/cls_version.cc:228: Loaded version class! >>> -7> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >>> cls/hello/cls_hello.cc:305: loading cls_hello >>> -6> 2016-04-15 22:47:06.426520 7f65fbbb0800 1 <cls> >>> cls/refcount/cls_refcount.cc:232: Loaded refcount class! >>> -5> 2016-04-15 22:47:06.427217 7f65fbbb0800 1 <cls> >>> cls/user/cls_user.cc:375: Loaded user class! >>> -4> 2016-04-15 22:47:06.428364 7f65fbbb0800 1 <cls> >>> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! >>> -3> 2016-04-15 22:47:06.428970 7f65fbbb0800 1 <cls> >>> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! >>> -2> 2016-04-15 22:47:06.430177 7f65fbbb0800 1 <cls> >>> cls/log/cls_log.cc:317: Loaded log class! >>> -1> 2016-04-15 22:47:06.438063 7f65fbbb0800 1 <cls> >>> cls/rgw/cls_rgw.cc:3206: Loaded rgw class! >>> 0> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >>> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time >>> 2016-04-15 >>> 22:47:06.494680 >>> osd/OSD.h: 885: FAILED assert(ret) >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x82) [0x7f65fb6364f2] >>> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 4: (main()+0x2b05) [0x7f65faed1735] >>> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 6: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- logging levels --- >>> 0/ 5 none >>> 0/ 1 lockdep >>> 0/ 1 context >>> 1/ 1 crush >>> 1/ 5 mds >>> 1/ 5 mds_balancer >>> 1/ 5 mds_locker >>> 1/ 5 mds_log >>> 1/ 5 mds_log_expire >>> 1/ 5 mds_migrator >>> 0/ 1 buffer >>> 0/ 1 timer >>> 0/ 1 filer >>> 0/ 1 striper >>> 0/ 1 objecter >>> 0/ 5 rados >>> 0/ 5 rbd >>> 0/ 5 rbd_mirror >>> 0/ 5 rbd_replay >>> 0/ 5 journaler >>> 0/ 5 objectcacher >>> 0/ 5 client >>> 0/ 5 osd >>> 0/ 5 optracker >>> 0/ 5 objclass >>> 1/ 3 filestore >>> 1/ 3 journal >>> 0/ 5 ms >>> 1/ 5 mon >>> 0/10 monc >>> 1/ 5 paxos >>> 0/ 5 tp >>> 1/ 5 auth >>> 1/ 5 crypto >>> 1/ 1 finisher >>> 1/ 5 heartbeatmap >>> 1/ 5 perfcounter >>> 1/ 5 rgw >>> 1/10 civetweb >>> 1/ 5 javaclient >>> 1/ 5 asok >>> 1/ 1 throttle >>> 0/ 0 refs >>> 1/ 5 xio >>> 1/ 5 compressor >>> 1/ 5 newstore >>> 1/ 5 bluestore >>> 1/ 5 bluefs >>> 1/ 3 bdev >>> 1/ 5 kstore >>> 4/ 5 rocksdb >>> 4/ 5 leveldb >>> 1/ 5 kinetic >>> 1/ 5 fuse >>> -2/-2 (syslog threshold) >>> -1/-1 (stderr threshold) >>> max_recent 10000 >>> max_new 1000 >>> log_file /var/log/ceph/ceph-osd.3.log >>> --- end dump of recent events --- >>> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal (Aborted) ** >>> in thread 7f65fbbb0800 thread_name:ceph-osd >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (()+0x949117) [0x7f65fb52e117] >>> 2: (()+0xf8d0) [0x7f65f9a318d0] >>> 3: (gsignal()+0x37) [0x7f65f7a7b067] >>> 4: (abort()+0x148) [0x7f65f7a7c448] >>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x256) [0x7f65fb6366c6] >>> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 8: (main()+0x2b05) [0x7f65faed1735] >>> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 10: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- begin dump of recent events --- >>> 0> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal >>> (Aborted) ** >>> in thread 7f65fbbb0800 thread_name:ceph-osd >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (()+0x949117) [0x7f65fb52e117] >>> 2: (()+0xf8d0) [0x7f65f9a318d0] >>> 3: (gsignal()+0x37) [0x7f65f7a7b067] >>> 4: (abort()+0x148) [0x7f65f7a7c448] >>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x256) [0x7f65fb6366c6] >>> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 8: (main()+0x2b05) [0x7f65faed1735] >>> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 10: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- logging levels --- >>> 0/ 5 none >>> 0/ 1 lockdep >>> 0/ 1 context >>> 1/ 1 crush >>> 1/ 5 mds >>> 1/ 5 mds_balancer >>> 1/ 5 mds_locker >>> 1/ 5 mds_log >>> 1/ 5 mds_log_expire >>> 1/ 5 mds_migrator >>> 0/ 1 buffer >>> 0/ 1 timer >>> 0/ 1 filer >>> 0/ 1 striper >>> 0/ 1 objecter >>> 0/ 5 rados >>> 0/ 5 rbd >>> 0/ 5 rbd_mirror >>> 0/ 5 rbd_replay >>> 0/ 5 journaler >>> 0/ 5 objectcacher >>> 0/ 5 client >>> 0/ 5 osd >>> 0/ 5 optracker >>> 0/ 5 objclass >>> 1/ 3 filestore >>> 1/ 3 journal >>> 0/ 5 ms >>> 1/ 5 mon >>> 0/10 monc >>> 1/ 5 paxos >>> 0/ 5 tp >>> 1/ 5 auth >>> 1/ 5 crypto >>> 1/ 1 finisher >>> 1/ 5 heartbeatmap >>> 1/ 5 perfcounter >>> 1/ 5 rgw >>> 1/10 civetweb >>> 1/ 5 javaclient >>> 1/ 5 asok >>> 1/ 1 throttle >>> 0/ 0 refs >>> 1/ 5 xio >>> 1/ 5 compressor >>> 1/ 5 newstore >>> 1/ 5 bluestore >>> 1/ 5 bluefs >>> 1/ 3 bdev >>> 1/ 5 kstore >>> 4/ 5 rocksdb >>> 4/ 5 leveldb >>> 1/ 5 kinetic >>> 1/ 5 fuse >>> -2/-2 (syslog threshold) >>> -1/-1 (stderr threshold) >>> max_recent 10000 >>> max_new 1000 >>> log_file /var/log/ceph/ceph-osd.3.log >>> --- end dump of recent events --- >>> >>> What can I try to get this OSD back online? I saw some similar issues on >>> google but I wasn't sure if that was actually the same issue. >>> >>> If I run in to MDS issue after resolving this, I'll send out another one. >>> =) >>> Thanks all! >>> >>> Regards, >>> Hong >> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> >> -- >> Thank you! >> HuangJun >> >> >> > > > > -- > Thank you! > HuangJun > > > > -- Thank you! HuangJun _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |