yes, it's a incremental osdmap, does the file size is correct? you can compare it with the same file in other osd. If it's not the same, you can overwrite it with the right one. 2016-04-16 13:11 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: > Is this it? > > root@OSD2:/var/lib/ceph/osd/ceph-3/current/meta# find ./ | grep osdmap | > grep 16024 > ./DIR_E/DIR_3/inc\uosdmap.16024__0_46887E3E__none > > Regards, > Hong > > > On Friday, April 15, 2016 11:53 PM, huang jun <hjwsm1989@xxxxxxxxx> wrote: > > > First, you should check whether file osdmap.16024 exists in your > osd.3/current/meta dir, > if not, you can copy it from other OSD who has it. > > > 2016-04-16 12:36 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: >> Here is what I get wtih debug_osd = 20. >> >> 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 >> (ceph:ceph) >> 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 >> 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore empty >> --pid-file >> 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 >> ErasureCodePluginSelectJerasure: >> load: jerasure_sse3 >> 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc load: >> isa >> 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting >> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >> 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log #20901 >> 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 #20901 >> >> 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 #20900 >> >> 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot >> 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock >> sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 >> 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] >> lci=[16010,16024]) >> 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes >> 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found journal >> 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new class >> name >> journal 0x7f9caa628808 >> 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from >> /usr/lib/rados-classes/libcls_journal.so >> 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal status 3 >> 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method >> journal.create flags 3 0x7f9c8dadac00 >> 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method >> journal.get_order flags 1 0x7f9c8dada3c0 >> 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method >> journal.get_splay_width flags 1 0x7f9c8dada360 >> 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method >> journal.get_pool_id flags 1 0x7f9c8dadaa30 >> 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method >> journal.get_minimum_set flags 1 0x7f9c8dada9c0 >> 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method >> journal.set_minimum_set flags 3 0x7f9c8dada830 >> 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method >> journal.get_active_set flags 1 0x7f9c8dada7c0 >> 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method >> journal.set_active_set flags 3 0x7f9c8dada630 >> 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method >> journal.get_client flags 1 0x7f9c8dadafb0 >> 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method >> journal.client_register flags 3 0x7f9c8dadc140 >> 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_data flags 3 0x7f9c8dadb730 >> 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_state flags 3 0x7f9c8dadb300 >> 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method >> journal.client_unregister flags 3 0x7f9c8dadf060 >> 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method >> journal.client_commit flags 3 0x7f9c8dadbc40 >> 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method >> journal.client_list flags 1 0x7f9c8dadc9c0 >> 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method >> journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 >> 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method >> journal.get_tag flags 1 0x7f9c8dadf9b0 >> 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method >> journal.tag_create flags 3 0x7f9c8dadfd10 >> 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method >> journal.tag_list flags 1 0x7f9c8dadd290 >> 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method >> journal.guard_append flags 3 0x7f9c8dad97f0 >> 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal success >> 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found statelog >> 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new class >> name >> statelog 0x7f9caa628928 >> 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog from >> /usr/lib/rados-classes/libcls_statelog.so >> 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog status >> 3 >> 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method >> statelog.add >> flags 3 0x7f9c8d8cdb60 >> 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method >> statelog.list >> flags 1 0x7f9c8d8cdef0 >> 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method >> statelog.remove flags 3 0x7f9c8d8cd130 >> 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method >> statelog.check_state flags 1 0x7f9c8d8cc680 >> 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog success >> 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found cephfs >> 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new class >> name >> cephfs 0x7f9caa628a48 >> 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from >> /usr/lib/rados-classes/libcls_cephfs.so >> 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs status 3 >> 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method >> cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 >> 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs success >> 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found rbd >> 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new class >> name >> rbd 0x7f9caa628b68 >> 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from >> /usr/lib/rados-classes/libcls_rbd.so >> 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd status 3 >> 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method rbd.create >> flags 3 0x7f9c8d4a6be0 >> 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method >> rbd.get_features flags 1 0x7f9c8d4af550 >> 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method >> rbd.set_features flags 3 0x7f9c8d4a3390 >> 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method >> rbd.get_size >> flags 1 0x7f9c8d4af940 >> 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method >> rbd.set_size >> flags 3 0x7f9c8d4a2bf0 >> 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapcontext flags 1 0x7f9c8d4aa510 >> 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method >> rbd.get_object_prefix flags 1 0x7f9c8d4a0060 >> 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 >> 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_add flags 3 0x7f9c8d4b2600 >> 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_remove flags 3 0x7f9c8d4b2300 >> 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 >> 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method >> rbd.get_all_features flags 1 0x7f9c8d49d240 >> 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method rbd.copyup >> flags 3 0x7f9c8d49d380 >> 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method >> rbd.get_parent flags 1 0x7f9c8d4b1410 >> 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method >> rbd.set_parent flags 3 0x7f9c8d4a7560 >> 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_parent flags 3 0x7f9c8d4a8cb0 >> 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method >> rbd.set_protection_status flags 3 0x7f9c8d4b00b0 >> 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method >> rbd.get_protection_status flags 1 0x7f9c8d4afdd0 >> 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method >> rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 >> 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method >> rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 >> 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method >> rbd.get_flags >> flags 1 0x7f9c8d4b0880 >> 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method >> rbd.set_flags >> flags 3 0x7f9c8d4b0c30 >> 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_list flags 1 0x7f9c8d4ac290 >> 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_set flags 3 0x7f9c8d4ac9e0 >> 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_remove flags 3 0x7f9c8d49e330 >> 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_get flags 1 0x7f9c8d4a28f0 >> 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method >> rbd.add_child >> flags 3 0x7f9c8d4a9f90 >> 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_child flags 3 0x7f9c8d4abba0 >> 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method >> rbd.get_children flags 1 0x7f9c8d4a9c00 >> 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method rbd.get_id >> flags 1 0x7f9c8d4a2640 >> 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method rbd.set_id >> flags 3 0x7f9c8d4a5f30 >> 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_id flags 1 0x7f9c8d4a0920 >> 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_name flags 1 0x7f9c8d4a0c00 >> 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_list >> flags 1 0x7f9c8d4adc20 >> 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 >> 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_remove_image flags 3 0x7f9c8d4a0770 >> 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_rename_image flags 3 0x7f9c8d4ad770 >> 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_load flags 1 0x7f9c8d4a8090 >> 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_save flags 3 0x7f9c8d4a8710 >> 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_resize flags 3 0x7f9c8d4a81c0 >> 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_update flags 3 0x7f9c8d4a61b0 >> 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 >> 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 >> 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_list >> flags 1 0x7f9c8d4a7d90 >> 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_add >> flags 3 0x7f9c8d4a4230 >> 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_remove flags 3 0x7f9c8d4a52e0 >> 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_rename flags 3 0x7f9c8d4a4b00 >> 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 >> 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 >> 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 >> 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 >> 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 >> 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 >> 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 >> 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 >> 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 >> 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 >> 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 >> 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get flags 1 0x7f9c8d4a1400 >> 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_set flags 3 0x7f9c8d4a4050 >> 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 >> 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success >> 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found version >> 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new class >> name >> version 0x7f9caa628c88 >> 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from >> /usr/lib/rados-classes/libcls_version.so >> 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version status 3 >> 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method version.set >> flags 3 0x7f9c8d28eb60 >> 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method version.inc >> flags 3 0x7f9c8d28fdd0 >> 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method >> version.inc_conds flags 3 0x7f9c8d28fdd0 >> 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method >> version.read >> flags 1 0x7f9c8d28e880 >> 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method >> version.check_conds flags 1 0x7f9c8d28f410 >> 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version success >> 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found kvs >> 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new class >> name >> kvs 0x7f9caa628da8 >> 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from >> /usr/lib/rados-classes/libcls_kvs.so >> 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs status 3 >> 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method >> kvs.get_idata_from_key flags 1 0x7f9c8d07e990 >> 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method >> kvs.get_next_idata flags 1 0x7f9c8d080500 >> 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method >> kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 >> 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method >> kvs.read_many >> flags 1 0x7f9c8d078a40 >> 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method >> kvs.check_writable flags 3 0x7f9c8d0783a0 >> 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method >> kvs.assert_size_in_bound flags 2 0x7f9c8d078640 >> 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_insert flags 2 0x7f9c8d07a870 >> 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method >> kvs.create_with_omap flags 2 0x7f9c8d07c370 >> 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_remove flags 2 0x7f9c8d0799e0 >> 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method >> kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 >> 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success >> 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found hello >> 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new class >> name >> hello 0x7f9caa628fe8 >> 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from >> /usr/lib/rados-classes/libcls_hello.so >> 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> >> cls/hello/cls_hello.cc:305: >> loading cls_hello >> 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello status 3 >> 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method >> hello.say_hello flags 1 0x7f9c8ce6f8f0 >> 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method >> hello.record_hello flags 10 0x7f9c8ce70a50 >> 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method >> hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 >> 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method >> hello.replay >> flags 1 0x7f9c8ce6f880 >> 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method >> hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 >> 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method >> hello.bad_reader flags 2 0x7f9c8ce6f870 >> 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method >> hello.bad_writer flags 1 0x7f9c8ce6f860 >> 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello success >> 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found refcount >> 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new class >> name >> refcount 0x7f9caa629108 >> 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount from >> /usr/lib/rados-classes/libcls_refcount.so >> 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount status >> 3 >> 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method >> refcount.get >> flags 3 0x7f9c8cbc5e80 >> 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method >> refcount.put >> flags 3 0x7f9c8cbc50f0 >> 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method >> refcount.set >> flags 3 0x7f9c8cbc4060 >> 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method >> refcount.read >> flags 1 0x7f9c8cbc57b0 >> 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount success >> 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found user >> 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new class >> name >> user 0x7f9caa629228 >> 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from >> /usr/lib/rados-classes/libcls_user.so >> 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user status 3 >> 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method >> user.set_buckets_info flags 3 0x7f9c8c91edc0 >> 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method >> user.complete_stats_sync flags 3 0x7f9c8c91e5f0 >> 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method >> user.remove_bucket flags 3 0x7f9c8c91d410 >> 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method >> user.list_buckets flags 1 0x7f9c8c91ffb0 >> 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method >> user.get_header flags 1 0x7f9c8c91bee0 >> 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user success >> 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found numops >> 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new class >> name >> numops 0x7f9caa629348 >> 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from >> /usr/lib/rados-classes/libcls_numops.so >> 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops status 3 >> 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method numops.add >> flags 3 0x7f9c8c716860 >> 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method numops.mul >> flags 3 0x7f9c8c715cc0 >> 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops success >> 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found >> replica_log >> 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new class >> name >> replica_log 0x7f9caa629468 >> 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log from >> /usr/lib/rados-classes/libcls_replica_log.so >> 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class replica_log >> status >> 3 >> 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method >> replica_log.set flags 3 0x7f9c8c50eed0 >> 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method >> replica_log.get flags 1 0x7f9c8c50fed0 >> 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method >> replica_log.delete flags 3 0x7f9c8c50f670 >> 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log success >> 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found >> timeindex >> 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new class >> name >> timeindex 0x7f9caa629588 >> 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex from >> /usr/lib/rados-classes/libcls_timeindex.so >> 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class timeindex status >> 3 >> 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method >> timeindex.add >> flags 3 0x7f9c8c306690 >> 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method >> timeindex.list flags 1 0x7f9c8c307200 >> 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method >> timeindex.trim flags 3 0x7f9c8c3068d0 >> 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex success >> 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found lock >> 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new class >> name >> lock 0x7f9caa6296a8 >> 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from >> /usr/lib/rados-classes/libcls_lock.so >> 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock status 3 >> 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method lock.lock >> flags 11 0x7f9c8c0fdb80 >> 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method lock.unlock >> flags 11 0x7f9c8c0fd560 >> 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method >> lock.break_lock flags 3 0x7f9c8c0fcf50 >> 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method >> lock.get_info >> flags 1 0x7f9c8c0fbc20 >> 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method >> lock.list_locks flags 1 0x7f9c8c0fa770 >> 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method >> lock.assert_locked flags 9 0x7f9c8c0fb180 >> 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock success >> 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found log >> 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new class >> name >> log 0x7f9caa6298e8 >> 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from >> /usr/lib/rados-classes/libcls_log.so >> 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log status 3 >> 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method log.add >> flags >> 3 0x7f9c8bef0320 >> 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method log.list >> flags 1 0x7f9c8bef1660 >> 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method log.trim >> flags 3 0x7f9c8bef0d30 >> 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method log.info >> flags 1 0x7f9c8beefc70 >> 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success >> 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found rgw >> 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new class >> name >> rgw 0x7f9caa629a08 >> 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from >> /usr/lib/rados-classes/libcls_rgw.so >> 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw status 3 >> 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_init_index flags 3 0x7f9c8bc0c620 >> 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 >> 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_list flags 1 0x7f9c8bc145e0 >> 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_check_index flags 1 0x7f9c8bc164a0 >> 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 >> 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 >> 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 >> 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 >> 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 >> 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 >> 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 >> 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 >> 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_remove flags 3 0x7f9c8bc138b0 >> 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 >> 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 >> 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 >> 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method rgw.bi_get >> flags 1 0x7f9c8bc0aae0 >> 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method rgw.bi_put >> flags 3 0x7f9c8bc08fc0 >> 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method rgw.bi_list >> flags 1 0x7f9c8bc11ca0 >> 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_list flags 1 0x7f9c8bc0e790 >> 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 >> 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method >> rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 >> 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 >> 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 >> 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 >> 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 >> 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 >> 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method rgw.gc_list >> flags 1 0x7f9c8bc0cff0 >> 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_remove >> flags 3 0x7f9c8bc0c320 >> 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success >> 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - loading >> and decoding 0x7f9caa5f2900 >> 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function >> 'OSDMapRef >> OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time 2016-04-15 >> 23:28:24.870922 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f9ca04e14f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 4: (main()+0x2b05) [0x7f9c9fd7c735] >> 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 6: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> -322> 2016-04-15 23:28:24.401110 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perfcounters_dump hook 0x7f9caa53e030 >> -321> 2016-04-15 23:28:24.401210 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 1 hook 0x7f9caa53e030 >> -320> 2016-04-15 23:28:24.401241 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf dump hook 0x7f9caa53e030 >> -319> 2016-04-15 23:28:24.401270 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perfcounters_schema hook 0x7f9caa53e030 >> -318> 2016-04-15 23:28:24.401296 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 2 hook 0x7f9caa53e030 >> -317> 2016-04-15 23:28:24.401323 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf schema hook 0x7f9caa53e030 >> -316> 2016-04-15 23:28:24.401352 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command perf reset hook 0x7f9caa53e030 >> -315> 2016-04-15 23:28:24.401380 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config show hook 0x7f9caa53e030 >> -314> 2016-04-15 23:28:24.401411 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config set hook 0x7f9caa53e030 >> -313> 2016-04-15 23:28:24.401438 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config get hook 0x7f9caa53e030 >> -312> 2016-04-15 23:28:24.401468 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command config diff hook 0x7f9caa53e030 >> -311> 2016-04-15 23:28:24.401495 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log flush hook 0x7f9caa53e030 >> -310> 2016-04-15 23:28:24.401522 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log dump hook 0x7f9caa53e030 >> -309> 2016-04-15 23:28:24.401549 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command log reopen hook 0x7f9caa53e030 >> -308> 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 >> (ceph:ceph) >> -307> 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 >> -306> 2016-04-15 23:28:24.431394 7f9ca0a5b800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -305> 2016-04-15 23:28:24.431450 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6800/2092 need_addr=0 >> -304> 2016-04-15 23:28:24.431517 7f9ca0a5b800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -303> 2016-04-15 23:28:24.431554 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6800/2092 need_addr=0 >> -302> 2016-04-15 23:28:24.431619 7f9ca0a5b800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -301> 2016-04-15 23:28:24.431877 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6801/2092 need_addr=0 >> -300> 2016-04-15 23:28:24.431974 7f9ca0a5b800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -299> 2016-04-15 23:28:24.432008 7f9ca0a5b800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6801/2092 need_addr=0 >> -298> 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore >> empty --pid-file >> -297> 2016-04-15 23:28:24.440611 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> init >> /var/run/ceph/ceph-osd.3.asok >> -296> 2016-04-15 23:28:24.440650 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> bind_and_listen /var/run/ceph/ceph-osd.3.asok >> -295> 2016-04-15 23:28:24.440983 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command 0 hook 0x7f9caa53a0d8 >> -294> 2016-04-15 23:28:24.441018 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command version hook 0x7f9caa53a0d8 >> -293> 2016-04-15 23:28:24.441038 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command git_version hook 0x7f9caa53a0d8 >> -292> 2016-04-15 23:28:24.441062 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command help hook 0x7f9caa53e1e0 >> -291> 2016-04-15 23:28:24.441084 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command get_command_descriptions hook 0x7f9caa53e1f0 >> -290> 2016-04-15 23:28:24.441316 7f9c99274700 5 asok(0x7f9caa5f6000) >> entry start >> -289> 2016-04-15 23:28:24.441424 7f9ca0a5b800 10 monclient(hunting): >> build_initial_monmap >> -288> 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 >> ErasureCodePluginSelectJerasure: load: jerasure_sse3 >> -287> 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc >> load: isa >> -286> 2016-04-15 23:28:24.470118 7f9ca0a5b800 5 adding auth protocol: >> cephx >> -285> 2016-04-15 23:28:24.470133 7f9ca0a5b800 5 adding auth protocol: >> cephx >> -284> 2016-04-15 23:28:24.470946 7f9ca0a5b800 5 asok(0x7f9caa5f6000) >> register_command objecter_requests hook 0x7f9caa53e2b0 >> -283> 2016-04-15 23:28:24.471128 7f9ca0a5b800 1 -- >> 192.168.1.31:6800/2092 >> messenger.start >> -282> 2016-04-15 23:28:24.471264 7f9ca0a5b800 1 -- :/0 messenger.start >> -281> 2016-04-15 23:28:24.471373 7f9ca0a5b800 1 -- >> 192.168.1.31:6801/2092 >> messenger.start >> -280> 2016-04-15 23:28:24.471482 7f9ca0a5b800 1 -- >> 192.168.2.31:6801/2092 >> messenger.start >> -279> 2016-04-15 23:28:24.471583 7f9ca0a5b800 1 -- >> 192.168.2.31:6800/2092 >> messenger.start >> -278> 2016-04-15 23:28:24.471680 7f9ca0a5b800 1 -- :/0 messenger.start >> -277> 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting >> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >> -276> 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> -275> 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> -274> 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> -273> 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> -272> 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> -271> 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> -270> 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log >> #20901 >> -269> 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 >> #20901 >> >> -268> 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 >> #20900 >> >> -267> 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> -266> 2016-04-15 23:28:24.792679 7f9ca0a5b800 2 journal open >> /var/lib/ceph/osd/ceph-3/journal fsid 4f86a418-6c67-4cb4-83a1-6c123c890036 >> fs_op_seq 9829589 >> -265> 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -264> 2016-04-15 23:28:24.794982 7f9ca0a5b800 2 journal open advancing >> committed_seq 9829584 to fs op_seq 9829589 >> -263> 2016-04-15 23:28:24.796869 7f9ca0a5b800 2 journal read_entry >> 2537717760 : seq 9829585 29509 bytes >> -262> 2016-04-15 23:28:24.797943 7f9ca0a5b800 2 journal read_entry >> 2537750528 : seq 9829586 8134 bytes >> -261> 2016-04-15 23:28:24.798617 7f9ca0a5b800 2 journal read_entry >> 2537762816 : seq 9829587 3064 bytes >> -260> 2016-04-15 23:28:24.798938 7f9ca0a5b800 2 journal read_entry >> 2537766912 : seq 9829588 7647 bytes >> -259> 2016-04-15 23:28:24.799208 7f9ca0a5b800 2 journal read_entry >> 2537775104 : seq 9829589 4737 bytes >> -258> 2016-04-15 23:28:24.799373 7f9ca0a5b800 2 journal No further valid >> entries found, journal is most likely valid >> -257> 2016-04-15 23:28:24.799478 7f9ca0a5b800 2 journal No further valid >> entries found, journal is most likely valid >> -256> 2016-04-15 23:28:24.799542 7f9ca0a5b800 3 journal journal_replay: >> end of journal, done. >> -255> 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -254> 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> -253> 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot >> -252> 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock >> sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 >> 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] >> lci=[16010,16024]) >> -251> 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes >> -250> 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found >> journal >> -249> 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new >> class name journal 0x7f9caa628808 >> -248> 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from >> /usr/lib/rados-classes/libcls_journal.so >> -247> 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal >> status 3 >> -246> 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method >> journal.create flags 3 0x7f9c8dadac00 >> -245> 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method >> journal.get_order flags 1 0x7f9c8dada3c0 >> -244> 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method >> journal.get_splay_width flags 1 0x7f9c8dada360 >> -243> 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method >> journal.get_pool_id flags 1 0x7f9c8dadaa30 >> -242> 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method >> journal.get_minimum_set flags 1 0x7f9c8dada9c0 >> -241> 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method >> journal.set_minimum_set flags 3 0x7f9c8dada830 >> -240> 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method >> journal.get_active_set flags 1 0x7f9c8dada7c0 >> -239> 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method >> journal.set_active_set flags 3 0x7f9c8dada630 >> -238> 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method >> journal.get_client flags 1 0x7f9c8dadafb0 >> -237> 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method >> journal.client_register flags 3 0x7f9c8dadc140 >> -236> 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_data flags 3 0x7f9c8dadb730 >> -235> 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method >> journal.client_update_state flags 3 0x7f9c8dadb300 >> -234> 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method >> journal.client_unregister flags 3 0x7f9c8dadf060 >> -233> 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method >> journal.client_commit flags 3 0x7f9c8dadbc40 >> -232> 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method >> journal.client_list flags 1 0x7f9c8dadc9c0 >> -231> 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method >> journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 >> -230> 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method >> journal.get_tag flags 1 0x7f9c8dadf9b0 >> -229> 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method >> journal.tag_create flags 3 0x7f9c8dadfd10 >> -228> 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method >> journal.tag_list flags 1 0x7f9c8dadd290 >> -227> 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method >> journal.guard_append flags 3 0x7f9c8dad97f0 >> -226> 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal >> success >> -225> 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found >> statelog >> -224> 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new >> class name statelog 0x7f9caa628928 >> -223> 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog >> from >> /usr/lib/rados-classes/libcls_statelog.so >> -222> 2016-04-15 23:28:24.818836 7f9ca0a5b800 1 <cls> >> cls/statelog/cls_statelog.cc:306: Loaded log class! >> -221> 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog >> status 3 >> -220> 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method >> statelog.add flags 3 0x7f9c8d8cdb60 >> -219> 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method >> statelog.list flags 1 0x7f9c8d8cdef0 >> -218> 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method >> statelog.remove flags 3 0x7f9c8d8cd130 >> -217> 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method >> statelog.check_state flags 1 0x7f9c8d8cc680 >> -216> 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog >> success >> -215> 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found >> cephfs >> -214> 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new >> class name cephfs 0x7f9caa628a48 >> -213> 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from >> /usr/lib/rados-classes/libcls_cephfs.so >> -212> 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> -211> 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs >> status 3 >> -210> 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method >> cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 >> -209> 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs >> success >> -208> 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found >> rbd >> -207> 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new >> class name rbd 0x7f9caa628b68 >> -206> 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from >> /usr/lib/rados-classes/libcls_rbd.so >> -205> 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd >> status >> 3 >> -204> 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method >> rbd.create flags 3 0x7f9c8d4a6be0 >> -203> 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method >> rbd.get_features flags 1 0x7f9c8d4af550 >> -202> 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method >> rbd.set_features flags 3 0x7f9c8d4a3390 >> -201> 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method >> rbd.get_size flags 1 0x7f9c8d4af940 >> -200> 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method >> rbd.set_size flags 3 0x7f9c8d4a2bf0 >> -199> 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapcontext flags 1 0x7f9c8d4aa510 >> -198> 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method >> rbd.get_object_prefix flags 1 0x7f9c8d4a0060 >> -197> 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method >> rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 >> -196> 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_add flags 3 0x7f9c8d4b2600 >> -195> 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_remove flags 3 0x7f9c8d4b2300 >> -194> 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method >> rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 >> -193> 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method >> rbd.get_all_features flags 1 0x7f9c8d49d240 >> -192> 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method >> rbd.copyup flags 3 0x7f9c8d49d380 >> -191> 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method >> rbd.get_parent flags 1 0x7f9c8d4b1410 >> -190> 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method >> rbd.set_parent flags 3 0x7f9c8d4a7560 >> -189> 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_parent flags 3 0x7f9c8d4a8cb0 >> -188> 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method >> rbd.set_protection_status flags 3 0x7f9c8d4b00b0 >> -187> 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method >> rbd.get_protection_status flags 1 0x7f9c8d4afdd0 >> -186> 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method >> rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 >> -185> 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method >> rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 >> -184> 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method >> rbd.get_flags flags 1 0x7f9c8d4b0880 >> -183> 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method >> rbd.set_flags flags 3 0x7f9c8d4b0c30 >> -182> 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_list flags 1 0x7f9c8d4ac290 >> -181> 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_set flags 3 0x7f9c8d4ac9e0 >> -180> 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_remove flags 3 0x7f9c8d49e330 >> -179> 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method >> rbd.metadata_get flags 1 0x7f9c8d4a28f0 >> -178> 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method >> rbd.add_child flags 3 0x7f9c8d4a9f90 >> -177> 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method >> rbd.remove_child flags 3 0x7f9c8d4abba0 >> -176> 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method >> rbd.get_children flags 1 0x7f9c8d4a9c00 >> -175> 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method >> rbd.get_id flags 1 0x7f9c8d4a2640 >> -174> 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method >> rbd.set_id flags 3 0x7f9c8d4a5f30 >> -173> 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_id flags 1 0x7f9c8d4a0920 >> -172> 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_get_name flags 1 0x7f9c8d4a0c00 >> -171> 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_list flags 1 0x7f9c8d4adc20 >> -170> 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 >> -169> 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_remove_image flags 3 0x7f9c8d4a0770 >> -168> 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method >> rbd.dir_rename_image flags 3 0x7f9c8d4ad770 >> -167> 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_load flags 1 0x7f9c8d4a8090 >> -166> 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_save flags 3 0x7f9c8d4a8710 >> -165> 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_resize flags 3 0x7f9c8d4a81c0 >> -164> 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_update flags 3 0x7f9c8d4a61b0 >> -163> 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 >> -162> 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method >> rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 >> -161> 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_list flags 1 0x7f9c8d4a7d90 >> -160> 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_add flags 3 0x7f9c8d4a4230 >> -159> 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_remove flags 3 0x7f9c8d4a52e0 >> -158> 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method >> rbd.snap_rename flags 3 0x7f9c8d4a4b00 >> -157> 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 >> -156> 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 >> -155> 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 >> -154> 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 >> -153> 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 >> -152> 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 >> -151> 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 >> -150> 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 >> -149> 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 >> -148> 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 >> -147> 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 >> -146> 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_get flags 1 0x7f9c8d4a1400 >> -145> 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_set flags 3 0x7f9c8d4a4050 >> -144> 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method >> rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 >> -143> 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success >> -142> 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found >> version >> -141> 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new >> class name version 0x7f9caa628c88 >> -140> 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from >> /usr/lib/rados-classes/libcls_version.so >> -139> 2016-04-15 23:28:24.833724 7f9ca0a5b800 1 <cls> >> cls/version/cls_version.cc:228: Loaded version class! >> -138> 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version >> status 3 >> -137> 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method >> version.set flags 3 0x7f9c8d28eb60 >> -136> 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method >> version.inc flags 3 0x7f9c8d28fdd0 >> -135> 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method >> version.inc_conds flags 3 0x7f9c8d28fdd0 >> -134> 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method >> version.read flags 1 0x7f9c8d28e880 >> -133> 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method >> version.check_conds flags 1 0x7f9c8d28f410 >> -132> 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version >> success >> -131> 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found >> kvs >> -130> 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new >> class name kvs 0x7f9caa628da8 >> -129> 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from >> /usr/lib/rados-classes/libcls_kvs.so >> -128> 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs >> status >> 3 >> -127> 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method >> kvs.get_idata_from_key flags 1 0x7f9c8d07e990 >> -126> 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method >> kvs.get_next_idata flags 1 0x7f9c8d080500 >> -125> 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method >> kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 >> -124> 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method >> kvs.read_many flags 1 0x7f9c8d078a40 >> -123> 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method >> kvs.check_writable flags 3 0x7f9c8d0783a0 >> -122> 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method >> kvs.assert_size_in_bound flags 2 0x7f9c8d078640 >> -121> 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_insert flags 2 0x7f9c8d07a870 >> -120> 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method >> kvs.create_with_omap flags 2 0x7f9c8d07c370 >> -119> 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method >> kvs.omap_remove flags 2 0x7f9c8d0799e0 >> -118> 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method >> kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 >> -117> 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success >> -116> 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found >> hello >> -115> 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new >> class name hello 0x7f9caa628fe8 >> -114> 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from >> /usr/lib/rados-classes/libcls_hello.so >> -113> 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> >> cls/hello/cls_hello.cc:305: loading cls_hello >> -112> 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello >> status 3 >> -111> 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method >> hello.say_hello flags 1 0x7f9c8ce6f8f0 >> -110> 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method >> hello.record_hello flags 10 0x7f9c8ce70a50 >> -109> 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method >> hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 >> -108> 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method >> hello.replay flags 1 0x7f9c8ce6f880 >> -107> 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method >> hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 >> -106> 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method >> hello.bad_reader flags 2 0x7f9c8ce6f870 >> -105> 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method >> hello.bad_writer flags 1 0x7f9c8ce6f860 >> -104> 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello >> success >> -103> 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found >> refcount >> -102> 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new >> class name refcount 0x7f9caa629108 >> -101> 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount >> from >> /usr/lib/rados-classes/libcls_refcount.so >> -100> 2016-04-15 23:28:24.849726 7f9ca0a5b800 1 <cls> >> cls/refcount/cls_refcount.cc:232: Loaded refcount class! >> -99> 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount >> status 3 >> -98> 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method >> refcount.get flags 3 0x7f9c8cbc5e80 >> -97> 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method >> refcount.put flags 3 0x7f9c8cbc50f0 >> -96> 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method >> refcount.set flags 3 0x7f9c8cbc4060 >> -95> 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method >> refcount.read flags 1 0x7f9c8cbc57b0 >> -94> 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount >> success >> -93> 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found >> user >> -92> 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new >> class name user 0x7f9caa629228 >> -91> 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from >> /usr/lib/rados-classes/libcls_user.so >> -90> 2016-04-15 23:28:24.851663 7f9ca0a5b800 1 <cls> >> cls/user/cls_user.cc:375: Loaded user class! >> -89> 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user >> status 3 >> -88> 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method >> user.set_buckets_info flags 3 0x7f9c8c91edc0 >> -87> 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method >> user.complete_stats_sync flags 3 0x7f9c8c91e5f0 >> -86> 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method >> user.remove_bucket flags 3 0x7f9c8c91d410 >> -85> 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method >> user.list_buckets flags 1 0x7f9c8c91ffb0 >> -84> 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method >> user.get_header flags 1 0x7f9c8c91bee0 >> -83> 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user >> success >> -82> 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found >> numops >> -81> 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new >> class name numops 0x7f9caa629348 >> -80> 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from >> /usr/lib/rados-classes/libcls_numops.so >> -79> 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops >> status 3 >> -78> 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method >> numops.add flags 3 0x7f9c8c716860 >> -77> 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method >> numops.mul flags 3 0x7f9c8c715cc0 >> -76> 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops >> success >> -75> 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found >> replica_log >> -74> 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new >> class name replica_log 0x7f9caa629468 >> -73> 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log >> from /usr/lib/rados-classes/libcls_replica_log.so >> -72> 2016-04-15 23:28:24.854977 7f9ca0a5b800 1 <cls> >> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! >> -71> 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class >> replica_log status 3 >> -70> 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method >> replica_log.set flags 3 0x7f9c8c50eed0 >> -69> 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method >> replica_log.get flags 1 0x7f9c8c50fed0 >> -68> 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method >> replica_log.delete flags 3 0x7f9c8c50f670 >> -67> 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log >> success >> -66> 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found >> timeindex >> -65> 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new >> class name timeindex 0x7f9caa629588 >> -64> 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex >> from /usr/lib/rados-classes/libcls_timeindex.so >> -63> 2016-04-15 23:28:24.856621 7f9ca0a5b800 1 <cls> >> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! >> -62> 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class >> timeindex >> status 3 >> -61> 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method >> timeindex.add flags 3 0x7f9c8c306690 >> -60> 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method >> timeindex.list flags 1 0x7f9c8c307200 >> -59> 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method >> timeindex.trim flags 3 0x7f9c8c3068d0 >> -58> 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex >> success >> -57> 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found >> lock >> -56> 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new >> class name lock 0x7f9caa6296a8 >> -55> 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from >> /usr/lib/rados-classes/libcls_lock.so >> -54> 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock >> status 3 >> -53> 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method >> lock.lock flags 11 0x7f9c8c0fdb80 >> -52> 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method >> lock.unlock flags 11 0x7f9c8c0fd560 >> -51> 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method >> lock.break_lock flags 3 0x7f9c8c0fcf50 >> -50> 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method >> lock.get_info flags 1 0x7f9c8c0fbc20 >> -49> 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method >> lock.list_locks flags 1 0x7f9c8c0fa770 >> -48> 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method >> lock.assert_locked flags 9 0x7f9c8c0fb180 >> -47> 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock >> success >> -46> 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found >> log >> -45> 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new >> class name log 0x7f9caa6298e8 >> -44> 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from >> /usr/lib/rados-classes/libcls_log.so >> -43> 2016-04-15 23:28:24.860294 7f9ca0a5b800 1 <cls> >> cls/log/cls_log.cc:317: Loaded log class! >> -42> 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log >> status >> 3 >> -41> 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method >> log.add flags 3 0x7f9c8bef0320 >> -40> 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method >> log.list flags 1 0x7f9c8bef1660 >> -39> 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method >> log.trim flags 3 0x7f9c8bef0d30 >> -38> 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method >> log.info flags 1 0x7f9c8beefc70 >> -37> 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success >> -36> 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found >> rgw >> -35> 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new >> class name rgw 0x7f9caa629a08 >> -34> 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from >> /usr/lib/rados-classes/libcls_rgw.so >> -33> 2016-04-15 23:28:24.870017 7f9ca0a5b800 1 <cls> >> cls/rgw/cls_rgw.cc:3206: Loaded rgw class! >> -32> 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw >> status >> 3 >> -31> 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_init_index flags 3 0x7f9c8bc0c620 >> -30> 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 >> -29> 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_list flags 1 0x7f9c8bc145e0 >> -28> 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_check_index flags 1 0x7f9c8bc164a0 >> -27> 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 >> -26> 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 >> -25> 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 >> -24> 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 >> -23> 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 >> -22> 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 >> -21> 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 >> -20> 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method >> rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 >> -19> 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_remove flags 3 0x7f9c8bc138b0 >> -18> 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 >> -17> 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 >> -16> 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method >> rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 >> -15> 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_get flags 1 0x7f9c8bc0aae0 >> -14> 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_put flags 3 0x7f9c8bc08fc0 >> -13> 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_list flags 1 0x7f9c8bc11ca0 >> -12> 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_list flags 1 0x7f9c8bc0e790 >> -11> 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method >> rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 >> -10> 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method >> rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 >> -9> 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 >> -8> 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 >> -7> 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method >> rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 >> -6> 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 >> -5> 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 >> -4> 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_list flags 1 0x7f9c8bc0cff0 >> -3> 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method >> rgw.gc_remove flags 3 0x7f9c8bc0c320 >> -2> 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success >> -1> 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - >> loading and decoding 0x7f9caa5f2900 >> 0> 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function >> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time >> 2016-04-15 >> 23:28:24.870922 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f9ca04e14f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 4: (main()+0x2b05) [0x7f9c9fd7c735] >> 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 6: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 20/20 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal (Aborted) ** >> in thread 7f9ca0a5b800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f9ca03d9117] >> 2: (()+0xf8d0) [0x7f9c9e8dc8d0] >> 3: (gsignal()+0x37) [0x7f9c9c926067] >> 4: (abort()+0x148) [0x7f9c9c927448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f9ca04e16c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 8: (main()+0x2b05) [0x7f9c9fd7c735] >> 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 10: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> 0> 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal >> (Aborted) ** >> in thread 7f9ca0a5b800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f9ca03d9117] >> 2: (()+0xf8d0) [0x7f9c9e8dc8d0] >> 3: (gsignal()+0x37) [0x7f9c9c926067] >> 4: (abort()+0x148) [0x7f9c9c927448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f9ca04e16c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] >> 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] >> 8: (main()+0x2b05) [0x7f9c9fd7c735] >> 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] >> 10: (()+0x337197) [0x7f9c9fdc7197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 20/20 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> >> >> >> On Friday, April 15, 2016 11:24 PM, huang jun <hjwsm1989@xxxxxxxxx> wrote: >> >> >> Can you set 'debug_osd = 20' in ceph.conf and restart the osd again, >> and post the corrupt log. >> I doubt it's problem related to "0 byte osdmap" decode problem? >> >> 2016-04-16 12:14 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: >>> I've been successfully running cephfs on my Debian Jessies for a while >>> and >>> one day after power outage, MDS wasn't happy. MDS crashing after it was >>> done loading, increasing the memory utilization quite a bit. I was >>> running >>> infernalis 9.2.0 and did successful upgrade from Hammer before... so I >>> thought I may have hit a bug and decided to try 9.2.1. >>> >>> In 9.2.1, it was not happy that my journal didn't have permission for >>> user >>> ceph. So corrected it. Then all of my OSDs are no longer starting. >>> Failing with similar messages as below. I upgraded to Jewel, as I didn't >>> see too much more complexitiy to upgrade from Infernalis and am still >>> seeing >>> these errors. >>> >>> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to 1001:1001 >>> (ceph:ceph) >>> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >>> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >>> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore empty >>> --pid-file >>> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >>> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >>> ioctl is disabled via 'filestore fiemap' config option >>> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >>> option >>> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >>> is >>> supported >>> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> syncfs(2) >>> syscall fully supported (by glibc and kernel) >>> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >>> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >>> disabled by conf >>> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log #20899 >>> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 #20899 >>> >>> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 #20898 >>> >>> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >>> mode: >>> checkpoint is not enabled >>> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >>> filestore(/var/lib/ceph/osd/ceph-3) upgrade >>> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >>> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >>> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >>> cls/hello/cls_hello.cc:305: >>> loading cls_hello >>> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >>> 'OSDMapRef >>> OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time 2016-04-15 >>> 22:47:06.494680 >>> osd/OSD.h: 885: FAILED assert(ret) >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x82) [0x7f65fb6364f2] >>> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 4: (main()+0x2b05) [0x7f65faed1735] >>> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 6: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- begin dump of recent events --- >>> -78> 2016-04-15 22:47:04.873688 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_dump hook 0x7f66067e2030 >>> -77> 2016-04-15 22:47:04.873771 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 1 hook 0x7f66067e2030 >>> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf dump hook 0x7f66067e2030 >>> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_schema hook 0x7f66067e2030 >>> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf dump hook 0x7f66067e2030 >>> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perfcounters_schema hook 0x7f66067e2030 >>> -74> 2016-04-15 22:47:04.873860 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 2 hook 0x7f66067e2030 >>> -73> 2016-04-15 22:47:04.873886 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf schema hook 0x7f66067e2030 >>> -72> 2016-04-15 22:47:04.873916 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command perf reset hook 0x7f66067e2030 >>> -71> 2016-04-15 22:47:04.873943 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config show hook 0x7f66067e2030 >>> -70> 2016-04-15 22:47:04.873974 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config set hook 0x7f66067e2030 >>> -69> 2016-04-15 22:47:04.874000 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config get hook 0x7f66067e2030 >>> -68> 2016-04-15 22:47:04.874029 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command config diff hook 0x7f66067e2030 >>> -67> 2016-04-15 22:47:04.874055 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log flush hook 0x7f66067e2030 >>> -66> 2016-04-15 22:47:04.874082 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log dump hook 0x7f66067e2030 >>> -65> 2016-04-15 22:47:04.874109 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command log reopen hook 0x7f66067e2030 >>> -64> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to >>> 1001:1001 >>> (ceph:ceph) >>> -63> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >>> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >>> -62> 2016-04-15 22:47:04.900224 7f65fbbb0800 1 -- 192.168.1.31:0/0 >>> learned my addr 192.168.1.31:0/0 >>> -61> 2016-04-15 22:47:04.900286 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.1.31:6802/1284 need_addr=0 >>> -60> 2016-04-15 22:47:04.900350 7f65fbbb0800 1 -- 192.168.2.31:0/0 >>> learned my addr 192.168.2.31:0/0 >>> -59> 2016-04-15 22:47:04.900375 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.2.31:6802/1284 need_addr=0 >>> -58> 2016-04-15 22:47:04.900443 7f65fbbb0800 1 -- 192.168.2.31:0/0 >>> learned my addr 192.168.2.31:0/0 >>> -57> 2016-04-15 22:47:04.900475 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.2.31:6803/1284 need_addr=0 >>> -56> 2016-04-15 22:47:04.900538 7f65fbbb0800 1 -- 192.168.1.31:0/0 >>> learned my addr 192.168.1.31:0/0 >>> -55> 2016-04-15 22:47:04.900562 7f65fbbb0800 1 accepter.accepter.bind >>> my_inst.addr is 192.168.1.31:6803/1284 need_addr=0 >>> -54> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore >>> empty --pid-file >>> -53> 2016-04-15 22:47:04.909743 7f65fbbb0800 5 asok(0x7f660689a000) >>> init >>> /var/run/ceph/ceph-osd.3.asok >>> -52> 2016-04-15 22:47:04.909792 7f65fbbb0800 5 asok(0x7f660689a000) >>> bind_and_listen /var/run/ceph/ceph-osd.3.asok >>> -51> 2016-04-15 22:47:04.909891 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command 0 hook 0x7f66067de0d8 >>> -50> 2016-04-15 22:47:04.909928 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command version hook 0x7f66067de0d8 >>> -49> 2016-04-15 22:47:04.909955 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command git_version hook 0x7f66067de0d8 >>> -48> 2016-04-15 22:47:04.909988 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command help hook 0x7f66067e21e0 >>> -47> 2016-04-15 22:47:04.910015 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command get_command_descriptions hook 0x7f66067e21f0 >>> -46> 2016-04-15 22:47:04.910205 7f65f43c9700 5 asok(0x7f660689a000) >>> entry start >>> -45> 2016-04-15 22:47:04.910330 7f65fbbb0800 10 monclient(hunting): >>> build_initial_monmap >>> -44> 2016-04-15 22:47:04.939070 7f65fbbb0800 5 adding auth protocol: >>> cephx >>> -43> 2016-04-15 22:47:04.939118 7f65fbbb0800 5 adding auth protocol: >>> cephx >>> -42> 2016-04-15 22:47:04.939986 7f65fbbb0800 5 asok(0x7f660689a000) >>> register_command objecter_requests hook 0x7f66067e22b0 >>> -41> 2016-04-15 22:47:04.940256 7f65fbbb0800 1 -- >>> 192.168.1.31:6802/1284 >>> messenger.start >>> -40> 2016-04-15 22:47:04.940413 7f65fbbb0800 1 -- :/0 messenger.start >>> -39> 2016-04-15 22:47:04.940557 7f65fbbb0800 1 -- >>> 192.168.1.31:6803/1284 >>> messenger.start >>> -38> 2016-04-15 22:47:04.940686 7f65fbbb0800 1 -- >>> 192.168.2.31:6803/1284 >>> messenger.start >>> -37> 2016-04-15 22:47:04.940798 7f65fbbb0800 1 -- >>> 192.168.2.31:6802/1284 >>> messenger.start >>> -36> 2016-04-15 22:47:04.940899 7f65fbbb0800 1 -- :/0 messenger.start >>> -35> 2016-04-15 22:47:04.941223 7f65fbbb0800 2 osd.3 0 mounting >>> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >>> -34> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >>> -33> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >>> ioctl is disabled via 'filestore fiemap' config option >>> -32> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >>> option >>> -31> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >>> is >>> supported >>> -30> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >>> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >>> syncfs(2) >>> syscall fully supported (by glibc and kernel) >>> -29> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >>> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >>> disabled by conf >>> -28> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering >>> log >>> #20899 >>> -27> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 >>> #20899 >>> >>> -26> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 >>> #20898 >>> >>> -25> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >>> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >>> mode: >>> checkpoint is not enabled >>> -24> 2016-04-15 22:47:06.377715 7f65fbbb0800 2 journal open >>> /var/lib/ceph/osd/ceph-3/journal fsid >>> 4f86a418-6c67-4cb4-83a1-6c123c890036 >>> fs_op_seq 9829589 >>> -23> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> -22> 2016-04-15 22:47:06.379811 7f65fbbb0800 2 journal open advancing >>> committed_seq 9829584 to fs op_seq 9829589 >>> -21> 2016-04-15 22:47:06.380757 7f65fbbb0800 2 journal read_entry >>> 2537717760 : seq 9829585 29509 bytes >>> -20> 2016-04-15 22:47:06.380996 7f65fbbb0800 2 journal read_entry >>> 2537750528 : seq 9829586 8134 bytes >>> -19> 2016-04-15 22:47:06.381091 7f65fbbb0800 2 journal read_entry >>> 2537762816 : seq 9829587 3064 bytes >>> -18> 2016-04-15 22:47:06.381155 7f65fbbb0800 2 journal read_entry >>> 2537766912 : seq 9829588 7647 bytes >>> -17> 2016-04-15 22:47:06.381219 7f65fbbb0800 2 journal read_entry >>> 2537775104 : seq 9829589 4737 bytes >>> -16> 2016-04-15 22:47:06.381257 7f65fbbb0800 2 journal No further >>> valid >>> entries found, journal is most likely valid >>> -15> 2016-04-15 22:47:06.381287 7f65fbbb0800 2 journal No further >>> valid >>> entries found, journal is most likely valid >>> -14> 2016-04-15 22:47:06.381302 7f65fbbb0800 3 journal >>> journal_replay: >>> end of journal, done. >>> -13> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >>> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size >>> 4096 >>> bytes, directio = 1, aio = 1 >>> -12> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >>> filestore(/var/lib/ceph/osd/ceph-3) upgrade >>> -11> 2016-04-15 22:47:06.385071 7f65fbbb0800 2 osd.3 0 boot >>> -10> 2016-04-15 22:47:06.415253 7f65fbbb0800 1 <cls> >>> cls/statelog/cls_statelog.cc:306: Loaded log class! >>> -9> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >>> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >>> -8> 2016-04-15 22:47:06.418172 7f65fbbb0800 1 <cls> >>> cls/version/cls_version.cc:228: Loaded version class! >>> -7> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >>> cls/hello/cls_hello.cc:305: loading cls_hello >>> -6> 2016-04-15 22:47:06.426520 7f65fbbb0800 1 <cls> >>> cls/refcount/cls_refcount.cc:232: Loaded refcount class! >>> -5> 2016-04-15 22:47:06.427217 7f65fbbb0800 1 <cls> >>> cls/user/cls_user.cc:375: Loaded user class! >>> -4> 2016-04-15 22:47:06.428364 7f65fbbb0800 1 <cls> >>> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! >>> -3> 2016-04-15 22:47:06.428970 7f65fbbb0800 1 <cls> >>> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! >>> -2> 2016-04-15 22:47:06.430177 7f65fbbb0800 1 <cls> >>> cls/log/cls_log.cc:317: Loaded log class! >>> -1> 2016-04-15 22:47:06.438063 7f65fbbb0800 1 <cls> >>> cls/rgw/cls_rgw.cc:3206: Loaded rgw class! >>> 0> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >>> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time >>> 2016-04-15 >>> 22:47:06.494680 >>> osd/OSD.h: 885: FAILED assert(ret) >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x82) [0x7f65fb6364f2] >>> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 4: (main()+0x2b05) [0x7f65faed1735] >>> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 6: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- logging levels --- >>> 0/ 5 none >>> 0/ 1 lockdep >>> 0/ 1 context >>> 1/ 1 crush >>> 1/ 5 mds >>> 1/ 5 mds_balancer >>> 1/ 5 mds_locker >>> 1/ 5 mds_log >>> 1/ 5 mds_log_expire >>> 1/ 5 mds_migrator >>> 0/ 1 buffer >>> 0/ 1 timer >>> 0/ 1 filer >>> 0/ 1 striper >>> 0/ 1 objecter >>> 0/ 5 rados >>> 0/ 5 rbd >>> 0/ 5 rbd_mirror >>> 0/ 5 rbd_replay >>> 0/ 5 journaler >>> 0/ 5 objectcacher >>> 0/ 5 client >>> 0/ 5 osd >>> 0/ 5 optracker >>> 0/ 5 objclass >>> 1/ 3 filestore >>> 1/ 3 journal >>> 0/ 5 ms >>> 1/ 5 mon >>> 0/10 monc >>> 1/ 5 paxos >>> 0/ 5 tp >>> 1/ 5 auth >>> 1/ 5 crypto >>> 1/ 1 finisher >>> 1/ 5 heartbeatmap >>> 1/ 5 perfcounter >>> 1/ 5 rgw >>> 1/10 civetweb >>> 1/ 5 javaclient >>> 1/ 5 asok >>> 1/ 1 throttle >>> 0/ 0 refs >>> 1/ 5 xio >>> 1/ 5 compressor >>> 1/ 5 newstore >>> 1/ 5 bluestore >>> 1/ 5 bluefs >>> 1/ 3 bdev >>> 1/ 5 kstore >>> 4/ 5 rocksdb >>> 4/ 5 leveldb >>> 1/ 5 kinetic >>> 1/ 5 fuse >>> -2/-2 (syslog threshold) >>> -1/-1 (stderr threshold) >>> max_recent 10000 >>> max_new 1000 >>> log_file /var/log/ceph/ceph-osd.3.log >>> --- end dump of recent events --- >>> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal (Aborted) ** >>> in thread 7f65fbbb0800 thread_name:ceph-osd >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (()+0x949117) [0x7f65fb52e117] >>> 2: (()+0xf8d0) [0x7f65f9a318d0] >>> 3: (gsignal()+0x37) [0x7f65f7a7b067] >>> 4: (abort()+0x148) [0x7f65f7a7c448] >>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x256) [0x7f65fb6366c6] >>> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 8: (main()+0x2b05) [0x7f65faed1735] >>> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 10: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- begin dump of recent events --- >>> 0> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal >>> (Aborted) ** >>> in thread 7f65fbbb0800 thread_name:ceph-osd >>> >>> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >>> 1: (()+0x949117) [0x7f65fb52e117] >>> 2: (()+0xf8d0) [0x7f65f9a318d0] >>> 3: (gsignal()+0x37) [0x7f65f7a7b067] >>> 4: (abort()+0x148) [0x7f65f7a7c448] >>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>> const*)+0x256) [0x7f65fb6366c6] >>> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >>> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >>> 8: (main()+0x2b05) [0x7f65faed1735] >>> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >>> 10: (()+0x337197) [0x7f65faf1c197] >>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >>> to >>> interpret this. >>> >>> --- logging levels --- >>> 0/ 5 none >>> 0/ 1 lockdep >>> 0/ 1 context >>> 1/ 1 crush >>> 1/ 5 mds >>> 1/ 5 mds_balancer >>> 1/ 5 mds_locker >>> 1/ 5 mds_log >>> 1/ 5 mds_log_expire >>> 1/ 5 mds_migrator >>> 0/ 1 buffer >>> 0/ 1 timer >>> 0/ 1 filer >>> 0/ 1 striper >>> 0/ 1 objecter >>> 0/ 5 rados >>> 0/ 5 rbd >>> 0/ 5 rbd_mirror >>> 0/ 5 rbd_replay >>> 0/ 5 journaler >>> 0/ 5 objectcacher >>> 0/ 5 client >>> 0/ 5 osd >>> 0/ 5 optracker >>> 0/ 5 objclass >>> 1/ 3 filestore >>> 1/ 3 journal >>> 0/ 5 ms >>> 1/ 5 mon >>> 0/10 monc >>> 1/ 5 paxos >>> 0/ 5 tp >>> 1/ 5 auth >>> 1/ 5 crypto >>> 1/ 1 finisher >>> 1/ 5 heartbeatmap >>> 1/ 5 perfcounter >>> 1/ 5 rgw >>> 1/10 civetweb >>> 1/ 5 javaclient >>> 1/ 5 asok >>> 1/ 1 throttle >>> 0/ 0 refs >>> 1/ 5 xio >>> 1/ 5 compressor >>> 1/ 5 newstore >>> 1/ 5 bluestore >>> 1/ 5 bluefs >>> 1/ 3 bdev >>> 1/ 5 kstore >>> 4/ 5 rocksdb >>> 4/ 5 leveldb >>> 1/ 5 kinetic >>> 1/ 5 fuse >>> -2/-2 (syslog threshold) >>> -1/-1 (stderr threshold) >>> max_recent 10000 >>> max_new 1000 >>> log_file /var/log/ceph/ceph-osd.3.log >>> --- end dump of recent events --- >>> >>> What can I try to get this OSD back online? I saw some similar issues on >>> google but I wasn't sure if that was actually the same issue. >>> >>> If I run in to MDS issue after resolving this, I'll send out another one. >>> =) >>> Thanks all! >>> >>> Regards, >>> Hong >> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> >> -- >> Thank you! >> HuangJun >> >> >> > > > > -- > Thank you! > HuangJun > > -- Thank you! HuangJun _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |