First, you should check whether file osdmap.16024 exists in your osd.3/current/meta dir, if not, you can copy it from other OSD who has it. 2016-04-16 12:36 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: > Here is what I get wtih debug_osd = 20. > > 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 > (ceph:ceph) > 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 > (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 > 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore empty > --pid-file > 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 ErasureCodePluginSelectJerasure: > load: jerasure_sse3 > 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc load: > isa > 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting > /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 > filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP > ioctl is disabled via 'filestore fiemap' config option > 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: > SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option > 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice is > supported > 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) > syscall fully supported (by glibc and kernel) > 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 > xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is > disabled by conf > 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log #20901 > 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 #20901 > > 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 #20900 > > 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 > filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: > checkpoint is not enabled > 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 > filestore(/var/lib/ceph/osd/ceph-3) upgrade > 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot > 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock > sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 > 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] lci=[16010,16024]) > 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes > 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found journal > 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new class name > journal 0x7f9caa628808 > 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from > /usr/lib/rados-classes/libcls_journal.so > 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal status 3 > 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method > journal.create flags 3 0x7f9c8dadac00 > 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method > journal.get_order flags 1 0x7f9c8dada3c0 > 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method > journal.get_splay_width flags 1 0x7f9c8dada360 > 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method > journal.get_pool_id flags 1 0x7f9c8dadaa30 > 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method > journal.get_minimum_set flags 1 0x7f9c8dada9c0 > 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method > journal.set_minimum_set flags 3 0x7f9c8dada830 > 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method > journal.get_active_set flags 1 0x7f9c8dada7c0 > 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method > journal.set_active_set flags 3 0x7f9c8dada630 > 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method > journal.get_client flags 1 0x7f9c8dadafb0 > 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method > journal.client_register flags 3 0x7f9c8dadc140 > 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method > journal.client_update_data flags 3 0x7f9c8dadb730 > 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method > journal.client_update_state flags 3 0x7f9c8dadb300 > 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method > journal.client_unregister flags 3 0x7f9c8dadf060 > 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method > journal.client_commit flags 3 0x7f9c8dadbc40 > 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method > journal.client_list flags 1 0x7f9c8dadc9c0 > 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method > journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 > 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method > journal.get_tag flags 1 0x7f9c8dadf9b0 > 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method > journal.tag_create flags 3 0x7f9c8dadfd10 > 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method > journal.tag_list flags 1 0x7f9c8dadd290 > 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method > journal.guard_append flags 3 0x7f9c8dad97f0 > 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal success > 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found statelog > 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new class name > statelog 0x7f9caa628928 > 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog from > /usr/lib/rados-classes/libcls_statelog.so > 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog status 3 > 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method statelog.add > flags 3 0x7f9c8d8cdb60 > 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method statelog.list > flags 1 0x7f9c8d8cdef0 > 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method > statelog.remove flags 3 0x7f9c8d8cd130 > 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method > statelog.check_state flags 1 0x7f9c8d8cc680 > 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog success > 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found cephfs > 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new class name > cephfs 0x7f9caa628a48 > 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from > /usr/lib/rados-classes/libcls_cephfs.so > 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> > cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan > 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs status 3 > 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method > cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 > 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs success > 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found rbd > 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new class name > rbd 0x7f9caa628b68 > 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from > /usr/lib/rados-classes/libcls_rbd.so > 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd status 3 > 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method rbd.create > flags 3 0x7f9c8d4a6be0 > 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method > rbd.get_features flags 1 0x7f9c8d4af550 > 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method > rbd.set_features flags 3 0x7f9c8d4a3390 > 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method rbd.get_size > flags 1 0x7f9c8d4af940 > 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method rbd.set_size > flags 3 0x7f9c8d4a2bf0 > 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method > rbd.get_snapcontext flags 1 0x7f9c8d4aa510 > 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method > rbd.get_object_prefix flags 1 0x7f9c8d4a0060 > 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method > rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 > 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_add flags 3 0x7f9c8d4b2600 > 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_remove flags 3 0x7f9c8d4b2300 > 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 > 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method > rbd.get_all_features flags 1 0x7f9c8d49d240 > 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method rbd.copyup > flags 3 0x7f9c8d49d380 > 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method > rbd.get_parent flags 1 0x7f9c8d4b1410 > 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method > rbd.set_parent flags 3 0x7f9c8d4a7560 > 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method > rbd.remove_parent flags 3 0x7f9c8d4a8cb0 > 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method > rbd.set_protection_status flags 3 0x7f9c8d4b00b0 > 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method > rbd.get_protection_status flags 1 0x7f9c8d4afdd0 > 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method > rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 > 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method > rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 > 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method rbd.get_flags > flags 1 0x7f9c8d4b0880 > 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method rbd.set_flags > flags 3 0x7f9c8d4b0c30 > 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_list flags 1 0x7f9c8d4ac290 > 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_set flags 3 0x7f9c8d4ac9e0 > 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_remove flags 3 0x7f9c8d49e330 > 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_get flags 1 0x7f9c8d4a28f0 > 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method rbd.add_child > flags 3 0x7f9c8d4a9f90 > 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method > rbd.remove_child flags 3 0x7f9c8d4abba0 > 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method > rbd.get_children flags 1 0x7f9c8d4a9c00 > 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method rbd.get_id > flags 1 0x7f9c8d4a2640 > 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method rbd.set_id > flags 3 0x7f9c8d4a5f30 > 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method > rbd.dir_get_id flags 1 0x7f9c8d4a0920 > 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method > rbd.dir_get_name flags 1 0x7f9c8d4a0c00 > 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method rbd.dir_list > flags 1 0x7f9c8d4adc20 > 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method > rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 > 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method > rbd.dir_remove_image flags 3 0x7f9c8d4a0770 > 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method > rbd.dir_rename_image flags 3 0x7f9c8d4ad770 > 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_load flags 1 0x7f9c8d4a8090 > 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_save flags 3 0x7f9c8d4a8710 > 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_resize flags 3 0x7f9c8d4a81c0 > 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_update flags 3 0x7f9c8d4a61b0 > 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 > 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 > 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method rbd.snap_list > flags 1 0x7f9c8d4a7d90 > 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method rbd.snap_add > flags 3 0x7f9c8d4a4230 > 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method > rbd.snap_remove flags 3 0x7f9c8d4a52e0 > 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method > rbd.snap_rename flags 3 0x7f9c8d4a4b00 > 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 > 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 > 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 > 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 > 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 > 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 > 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 > 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 > 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 > 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 > 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 > 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_get flags 1 0x7f9c8d4a1400 > 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_set flags 3 0x7f9c8d4a4050 > 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 > 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success > 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found version > 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new class name > version 0x7f9caa628c88 > 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from > /usr/lib/rados-classes/libcls_version.so > 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version status 3 > 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method version.set > flags 3 0x7f9c8d28eb60 > 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method version.inc > flags 3 0x7f9c8d28fdd0 > 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method > version.inc_conds flags 3 0x7f9c8d28fdd0 > 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method version.read > flags 1 0x7f9c8d28e880 > 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method > version.check_conds flags 1 0x7f9c8d28f410 > 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version success > 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found kvs > 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new class name > kvs 0x7f9caa628da8 > 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from > /usr/lib/rados-classes/libcls_kvs.so > 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs status 3 > 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method > kvs.get_idata_from_key flags 1 0x7f9c8d07e990 > 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method > kvs.get_next_idata flags 1 0x7f9c8d080500 > 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method > kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 > 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method kvs.read_many > flags 1 0x7f9c8d078a40 > 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method > kvs.check_writable flags 3 0x7f9c8d0783a0 > 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method > kvs.assert_size_in_bound flags 2 0x7f9c8d078640 > 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method > kvs.omap_insert flags 2 0x7f9c8d07a870 > 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method > kvs.create_with_omap flags 2 0x7f9c8d07c370 > 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method > kvs.omap_remove flags 2 0x7f9c8d0799e0 > 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method > kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 > 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success > 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found hello > 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new class name > hello 0x7f9caa628fe8 > 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from > /usr/lib/rados-classes/libcls_hello.so > 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> cls/hello/cls_hello.cc:305: > loading cls_hello > 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello status 3 > 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method > hello.say_hello flags 1 0x7f9c8ce6f8f0 > 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method > hello.record_hello flags 10 0x7f9c8ce70a50 > 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method > hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 > 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method hello.replay > flags 1 0x7f9c8ce6f880 > 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method > hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 > 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method > hello.bad_reader flags 2 0x7f9c8ce6f870 > 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method > hello.bad_writer flags 1 0x7f9c8ce6f860 > 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello success > 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found refcount > 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new class name > refcount 0x7f9caa629108 > 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount from > /usr/lib/rados-classes/libcls_refcount.so > 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount status 3 > 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method refcount.get > flags 3 0x7f9c8cbc5e80 > 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method refcount.put > flags 3 0x7f9c8cbc50f0 > 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method refcount.set > flags 3 0x7f9c8cbc4060 > 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method refcount.read > flags 1 0x7f9c8cbc57b0 > 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount success > 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found user > 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new class name > user 0x7f9caa629228 > 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from > /usr/lib/rados-classes/libcls_user.so > 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user status 3 > 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method > user.set_buckets_info flags 3 0x7f9c8c91edc0 > 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method > user.complete_stats_sync flags 3 0x7f9c8c91e5f0 > 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method > user.remove_bucket flags 3 0x7f9c8c91d410 > 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method > user.list_buckets flags 1 0x7f9c8c91ffb0 > 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method > user.get_header flags 1 0x7f9c8c91bee0 > 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user success > 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found numops > 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new class name > numops 0x7f9caa629348 > 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from > /usr/lib/rados-classes/libcls_numops.so > 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops status 3 > 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method numops.add > flags 3 0x7f9c8c716860 > 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method numops.mul > flags 3 0x7f9c8c715cc0 > 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops success > 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found > replica_log > 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new class name > replica_log 0x7f9caa629468 > 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log from > /usr/lib/rados-classes/libcls_replica_log.so > 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class replica_log status > 3 > 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method > replica_log.set flags 3 0x7f9c8c50eed0 > 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method > replica_log.get flags 1 0x7f9c8c50fed0 > 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method > replica_log.delete flags 3 0x7f9c8c50f670 > 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log success > 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found timeindex > 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new class name > timeindex 0x7f9caa629588 > 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex from > /usr/lib/rados-classes/libcls_timeindex.so > 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class timeindex status 3 > 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method timeindex.add > flags 3 0x7f9c8c306690 > 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method > timeindex.list flags 1 0x7f9c8c307200 > 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method > timeindex.trim flags 3 0x7f9c8c3068d0 > 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex success > 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found lock > 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new class name > lock 0x7f9caa6296a8 > 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from > /usr/lib/rados-classes/libcls_lock.so > 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock status 3 > 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method lock.lock > flags 11 0x7f9c8c0fdb80 > 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method lock.unlock > flags 11 0x7f9c8c0fd560 > 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method > lock.break_lock flags 3 0x7f9c8c0fcf50 > 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method lock.get_info > flags 1 0x7f9c8c0fbc20 > 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method > lock.list_locks flags 1 0x7f9c8c0fa770 > 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method > lock.assert_locked flags 9 0x7f9c8c0fb180 > 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock success > 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found log > 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new class name > log 0x7f9caa6298e8 > 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from > /usr/lib/rados-classes/libcls_log.so > 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log status 3 > 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method log.add flags > 3 0x7f9c8bef0320 > 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method log.list > flags 1 0x7f9c8bef1660 > 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method log.trim > flags 3 0x7f9c8bef0d30 > 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method log.info > flags 1 0x7f9c8beefc70 > 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success > 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found rgw > 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new class name > rgw 0x7f9caa629a08 > 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from > /usr/lib/rados-classes/libcls_rgw.so > 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw status 3 > 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_init_index flags 3 0x7f9c8bc0c620 > 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 > 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_list flags 1 0x7f9c8bc145e0 > 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_check_index flags 1 0x7f9c8bc164a0 > 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 > 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 > 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 > 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 > 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 > 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 > 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 > 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 > 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method > rgw.obj_remove flags 3 0x7f9c8bc138b0 > 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method > rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 > 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method > rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 > 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method > rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 > 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method rgw.bi_get > flags 1 0x7f9c8bc0aae0 > 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method rgw.bi_put > flags 3 0x7f9c8bc08fc0 > 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method rgw.bi_list > flags 1 0x7f9c8bc11ca0 > 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method > rgw.bi_log_list flags 1 0x7f9c8bc0e790 > 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method > rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 > 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method > rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 > 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 > 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 > 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 > 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method > rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 > 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method > rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 > 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method rgw.gc_list > flags 1 0x7f9c8bc0cff0 > 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method rgw.gc_remove > flags 3 0x7f9c8bc0c320 > 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success > 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - loading > and decoding 0x7f9caa5f2900 > 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function 'OSDMapRef > OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time 2016-04-15 > 23:28:24.870922 > osd/OSD.h: 885: FAILED assert(ret) > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x82) [0x7f9ca04e14f2] > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] > 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] > 4: (main()+0x2b05) [0x7f9c9fd7c735] > 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] > 6: (()+0x337197) [0x7f9c9fdc7197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- begin dump of recent events --- > -322> 2016-04-15 23:28:24.401110 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command perfcounters_dump hook 0x7f9caa53e030 > -321> 2016-04-15 23:28:24.401210 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command 1 hook 0x7f9caa53e030 > -320> 2016-04-15 23:28:24.401241 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command perf dump hook 0x7f9caa53e030 > -319> 2016-04-15 23:28:24.401270 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command perfcounters_schema hook 0x7f9caa53e030 > -318> 2016-04-15 23:28:24.401296 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command 2 hook 0x7f9caa53e030 > -317> 2016-04-15 23:28:24.401323 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command perf schema hook 0x7f9caa53e030 > -316> 2016-04-15 23:28:24.401352 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command perf reset hook 0x7f9caa53e030 > -315> 2016-04-15 23:28:24.401380 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command config show hook 0x7f9caa53e030 > -314> 2016-04-15 23:28:24.401411 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command config set hook 0x7f9caa53e030 > -313> 2016-04-15 23:28:24.401438 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command config get hook 0x7f9caa53e030 > -312> 2016-04-15 23:28:24.401468 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command config diff hook 0x7f9caa53e030 > -311> 2016-04-15 23:28:24.401495 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command log flush hook 0x7f9caa53e030 > -310> 2016-04-15 23:28:24.401522 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command log dump hook 0x7f9caa53e030 > -309> 2016-04-15 23:28:24.401549 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command log reopen hook 0x7f9caa53e030 > -308> 2016-04-15 23:28:24.429063 7f9ca0a5b800 0 set uid:gid to 1001:1001 > (ceph:ceph) > -307> 2016-04-15 23:28:24.429167 7f9ca0a5b800 0 ceph version 10.1.2 > (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 2092 > -306> 2016-04-15 23:28:24.431394 7f9ca0a5b800 1 -- 192.168.1.31:0/0 > learned my addr 192.168.1.31:0/0 > -305> 2016-04-15 23:28:24.431450 7f9ca0a5b800 1 accepter.accepter.bind > my_inst.addr is 192.168.1.31:6800/2092 need_addr=0 > -304> 2016-04-15 23:28:24.431517 7f9ca0a5b800 1 -- 192.168.2.31:0/0 > learned my addr 192.168.2.31:0/0 > -303> 2016-04-15 23:28:24.431554 7f9ca0a5b800 1 accepter.accepter.bind > my_inst.addr is 192.168.2.31:6800/2092 need_addr=0 > -302> 2016-04-15 23:28:24.431619 7f9ca0a5b800 1 -- 192.168.2.31:0/0 > learned my addr 192.168.2.31:0/0 > -301> 2016-04-15 23:28:24.431877 7f9ca0a5b800 1 accepter.accepter.bind > my_inst.addr is 192.168.2.31:6801/2092 need_addr=0 > -300> 2016-04-15 23:28:24.431974 7f9ca0a5b800 1 -- 192.168.1.31:0/0 > learned my addr 192.168.1.31:0/0 > -299> 2016-04-15 23:28:24.432008 7f9ca0a5b800 1 accepter.accepter.bind > my_inst.addr is 192.168.1.31:6801/2092 need_addr=0 > -298> 2016-04-15 23:28:24.432034 7f9ca0a5b800 0 pidfile_write: ignore > empty --pid-file > -297> 2016-04-15 23:28:24.440611 7f9ca0a5b800 5 asok(0x7f9caa5f6000) init > /var/run/ceph/ceph-osd.3.asok > -296> 2016-04-15 23:28:24.440650 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > bind_and_listen /var/run/ceph/ceph-osd.3.asok > -295> 2016-04-15 23:28:24.440983 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command 0 hook 0x7f9caa53a0d8 > -294> 2016-04-15 23:28:24.441018 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command version hook 0x7f9caa53a0d8 > -293> 2016-04-15 23:28:24.441038 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command git_version hook 0x7f9caa53a0d8 > -292> 2016-04-15 23:28:24.441062 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command help hook 0x7f9caa53e1e0 > -291> 2016-04-15 23:28:24.441084 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command get_command_descriptions hook 0x7f9caa53e1f0 > -290> 2016-04-15 23:28:24.441316 7f9c99274700 5 asok(0x7f9caa5f6000) > entry start > -289> 2016-04-15 23:28:24.441424 7f9ca0a5b800 10 monclient(hunting): > build_initial_monmap > -288> 2016-04-15 23:28:24.459417 7f9ca0a5b800 10 > ErasureCodePluginSelectJerasure: load: jerasure_sse3 > -287> 2016-04-15 23:28:24.470016 7f9ca0a5b800 10 load: jerasure load: lrc > load: isa > -286> 2016-04-15 23:28:24.470118 7f9ca0a5b800 5 adding auth protocol: > cephx > -285> 2016-04-15 23:28:24.470133 7f9ca0a5b800 5 adding auth protocol: > cephx > -284> 2016-04-15 23:28:24.470946 7f9ca0a5b800 5 asok(0x7f9caa5f6000) > register_command objecter_requests hook 0x7f9caa53e2b0 > -283> 2016-04-15 23:28:24.471128 7f9ca0a5b800 1 -- 192.168.1.31:6800/2092 > messenger.start > -282> 2016-04-15 23:28:24.471264 7f9ca0a5b800 1 -- :/0 messenger.start > -281> 2016-04-15 23:28:24.471373 7f9ca0a5b800 1 -- 192.168.1.31:6801/2092 > messenger.start > -280> 2016-04-15 23:28:24.471482 7f9ca0a5b800 1 -- 192.168.2.31:6801/2092 > messenger.start > -279> 2016-04-15 23:28:24.471583 7f9ca0a5b800 1 -- 192.168.2.31:6800/2092 > messenger.start > -278> 2016-04-15 23:28:24.471680 7f9ca0a5b800 1 -- :/0 messenger.start > -277> 2016-04-15 23:28:24.472013 7f9ca0a5b800 2 osd.3 0 mounting > /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > -276> 2016-04-15 23:28:24.472292 7f9ca0a5b800 0 > filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > -275> 2016-04-15 23:28:24.473496 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP > ioctl is disabled via 'filestore fiemap' config option > -274> 2016-04-15 23:28:24.473541 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: > SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option > -273> 2016-04-15 23:28:24.473615 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice is > supported > -272> 2016-04-15 23:28:24.494485 7f9ca0a5b800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) > syscall fully supported (by glibc and kernel) > -271> 2016-04-15 23:28:24.494802 7f9ca0a5b800 0 > xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is > disabled by conf > -270> 2016-04-15 23:28:24.499066 7f9ca0a5b800 1 leveldb: Recovering log > #20901 > -269> 2016-04-15 23:28:24.782188 7f9ca0a5b800 1 leveldb: Delete type=0 > #20901 > > -268> 2016-04-15 23:28:24.782420 7f9ca0a5b800 1 leveldb: Delete type=3 > #20900 > > -267> 2016-04-15 23:28:24.784810 7f9ca0a5b800 0 > filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: > checkpoint is not enabled > -266> 2016-04-15 23:28:24.792679 7f9ca0a5b800 2 journal open > /var/lib/ceph/osd/ceph-3/journal fsid 4f86a418-6c67-4cb4-83a1-6c123c890036 > fs_op_seq 9829589 > -265> 2016-04-15 23:28:24.792918 7f9ca0a5b800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > -264> 2016-04-15 23:28:24.794982 7f9ca0a5b800 2 journal open advancing > committed_seq 9829584 to fs op_seq 9829589 > -263> 2016-04-15 23:28:24.796869 7f9ca0a5b800 2 journal read_entry > 2537717760 : seq 9829585 29509 bytes > -262> 2016-04-15 23:28:24.797943 7f9ca0a5b800 2 journal read_entry > 2537750528 : seq 9829586 8134 bytes > -261> 2016-04-15 23:28:24.798617 7f9ca0a5b800 2 journal read_entry > 2537762816 : seq 9829587 3064 bytes > -260> 2016-04-15 23:28:24.798938 7f9ca0a5b800 2 journal read_entry > 2537766912 : seq 9829588 7647 bytes > -259> 2016-04-15 23:28:24.799208 7f9ca0a5b800 2 journal read_entry > 2537775104 : seq 9829589 4737 bytes > -258> 2016-04-15 23:28:24.799373 7f9ca0a5b800 2 journal No further valid > entries found, journal is most likely valid > -257> 2016-04-15 23:28:24.799478 7f9ca0a5b800 2 journal No further valid > entries found, journal is most likely valid > -256> 2016-04-15 23:28:24.799542 7f9ca0a5b800 3 journal journal_replay: > end of journal, done. > -255> 2016-04-15 23:28:24.800583 7f9ca0a5b800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > -254> 2016-04-15 23:28:24.808144 7f9ca0a5b800 1 > filestore(/var/lib/ceph/osd/ceph-3) upgrade > -253> 2016-04-15 23:28:24.808540 7f9ca0a5b800 2 osd.3 0 boot > -252> 2016-04-15 23:28:24.809265 7f9ca0a5b800 10 osd.3 0 read_superblock > sb(9b2c9bca-112e-48b0-86fc-587ef9a52948 osd.3 > 4f86a418-6c67-4cb4-83a1-6c123c890036 e16024 [15332,16024] lci=[16010,16024]) > -251> 2016-04-15 23:28:24.810029 7f9ca0a5b800 10 open_all_classes > -250> 2016-04-15 23:28:24.810433 7f9ca0a5b800 10 open_all_classes found > journal > -249> 2016-04-15 23:28:24.810746 7f9ca0a5b800 10 _get_class adding new > class name journal 0x7f9caa628808 > -248> 2016-04-15 23:28:24.811059 7f9ca0a5b800 10 _load_class journal from > /usr/lib/rados-classes/libcls_journal.so > -247> 2016-04-15 23:28:24.814498 7f9ca0a5b800 10 register_class journal > status 3 > -246> 2016-04-15 23:28:24.814650 7f9ca0a5b800 10 register_cxx_method > journal.create flags 3 0x7f9c8dadac00 > -245> 2016-04-15 23:28:24.814745 7f9ca0a5b800 10 register_cxx_method > journal.get_order flags 1 0x7f9c8dada3c0 > -244> 2016-04-15 23:28:24.814838 7f9ca0a5b800 10 register_cxx_method > journal.get_splay_width flags 1 0x7f9c8dada360 > -243> 2016-04-15 23:28:24.814925 7f9ca0a5b800 10 register_cxx_method > journal.get_pool_id flags 1 0x7f9c8dadaa30 > -242> 2016-04-15 23:28:24.815062 7f9ca0a5b800 10 register_cxx_method > journal.get_minimum_set flags 1 0x7f9c8dada9c0 > -241> 2016-04-15 23:28:24.815162 7f9ca0a5b800 10 register_cxx_method > journal.set_minimum_set flags 3 0x7f9c8dada830 > -240> 2016-04-15 23:28:24.815246 7f9ca0a5b800 10 register_cxx_method > journal.get_active_set flags 1 0x7f9c8dada7c0 > -239> 2016-04-15 23:28:24.815336 7f9ca0a5b800 10 register_cxx_method > journal.set_active_set flags 3 0x7f9c8dada630 > -238> 2016-04-15 23:28:24.815417 7f9ca0a5b800 10 register_cxx_method > journal.get_client flags 1 0x7f9c8dadafb0 > -237> 2016-04-15 23:28:24.815501 7f9ca0a5b800 10 register_cxx_method > journal.client_register flags 3 0x7f9c8dadc140 > -236> 2016-04-15 23:28:24.815589 7f9ca0a5b800 10 register_cxx_method > journal.client_update_data flags 3 0x7f9c8dadb730 > -235> 2016-04-15 23:28:24.815679 7f9ca0a5b800 10 register_cxx_method > journal.client_update_state flags 3 0x7f9c8dadb300 > -234> 2016-04-15 23:28:24.815771 7f9ca0a5b800 10 register_cxx_method > journal.client_unregister flags 3 0x7f9c8dadf060 > -233> 2016-04-15 23:28:24.815854 7f9ca0a5b800 10 register_cxx_method > journal.client_commit flags 3 0x7f9c8dadbc40 > -232> 2016-04-15 23:28:24.815934 7f9ca0a5b800 10 register_cxx_method > journal.client_list flags 1 0x7f9c8dadc9c0 > -231> 2016-04-15 23:28:24.816019 7f9ca0a5b800 10 register_cxx_method > journal.get_next_tag_tid flags 1 0x7f9c8dada5c0 > -230> 2016-04-15 23:28:24.816102 7f9ca0a5b800 10 register_cxx_method > journal.get_tag flags 1 0x7f9c8dadf9b0 > -229> 2016-04-15 23:28:24.816190 7f9ca0a5b800 10 register_cxx_method > journal.tag_create flags 3 0x7f9c8dadfd10 > -228> 2016-04-15 23:28:24.816277 7f9ca0a5b800 10 register_cxx_method > journal.tag_list flags 1 0x7f9c8dadd290 > -227> 2016-04-15 23:28:24.816358 7f9ca0a5b800 10 register_cxx_method > journal.guard_append flags 3 0x7f9c8dad97f0 > -226> 2016-04-15 23:28:24.816444 7f9ca0a5b800 10 _load_class journal > success > -225> 2016-04-15 23:28:24.816523 7f9ca0a5b800 10 open_all_classes found > statelog > -224> 2016-04-15 23:28:24.816602 7f9ca0a5b800 10 _get_class adding new > class name statelog 0x7f9caa628928 > -223> 2016-04-15 23:28:24.816685 7f9ca0a5b800 10 _load_class statelog from > /usr/lib/rados-classes/libcls_statelog.so > -222> 2016-04-15 23:28:24.818836 7f9ca0a5b800 1 <cls> > cls/statelog/cls_statelog.cc:306: Loaded log class! > -221> 2016-04-15 23:28:24.819037 7f9ca0a5b800 10 register_class statelog > status 3 > -220> 2016-04-15 23:28:24.819342 7f9ca0a5b800 10 register_cxx_method > statelog.add flags 3 0x7f9c8d8cdb60 > -219> 2016-04-15 23:28:24.819592 7f9ca0a5b800 10 register_cxx_method > statelog.list flags 1 0x7f9c8d8cdef0 > -218> 2016-04-15 23:28:24.819878 7f9ca0a5b800 10 register_cxx_method > statelog.remove flags 3 0x7f9c8d8cd130 > -217> 2016-04-15 23:28:24.820129 7f9ca0a5b800 10 register_cxx_method > statelog.check_state flags 1 0x7f9c8d8cc680 > -216> 2016-04-15 23:28:24.820400 7f9ca0a5b800 10 _load_class statelog > success > -215> 2016-04-15 23:28:24.820658 7f9ca0a5b800 10 open_all_classes found > cephfs > -214> 2016-04-15 23:28:24.820917 7f9ca0a5b800 10 _get_class adding new > class name cephfs 0x7f9caa628a48 > -213> 2016-04-15 23:28:24.821192 7f9ca0a5b800 10 _load_class cephfs from > /usr/lib/rados-classes/libcls_cephfs.so > -212> 2016-04-15 23:28:24.823518 7f9ca0a5b800 0 <cls> > cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan > -211> 2016-04-15 23:28:24.823829 7f9ca0a5b800 10 register_class cephfs > status 3 > -210> 2016-04-15 23:28:24.823993 7f9ca0a5b800 10 register_cxx_method > cephfs.accumulate_inode_metadata flags 3 0x7f9c8d6c5770 > -209> 2016-04-15 23:28:24.824104 7f9ca0a5b800 10 _load_class cephfs > success > -208> 2016-04-15 23:28:24.824192 7f9ca0a5b800 10 open_all_classes found > rbd > -207> 2016-04-15 23:28:24.824280 7f9ca0a5b800 10 _get_class adding new > class name rbd 0x7f9caa628b68 > -206> 2016-04-15 23:28:24.824366 7f9ca0a5b800 10 _load_class rbd from > /usr/lib/rados-classes/libcls_rbd.so > -205> 2016-04-15 23:28:24.826685 7f9ca0a5b800 10 register_class rbd status > 3 > -204> 2016-04-15 23:28:24.826878 7f9ca0a5b800 10 register_cxx_method > rbd.create flags 3 0x7f9c8d4a6be0 > -203> 2016-04-15 23:28:24.826988 7f9ca0a5b800 10 register_cxx_method > rbd.get_features flags 1 0x7f9c8d4af550 > -202> 2016-04-15 23:28:24.827082 7f9ca0a5b800 10 register_cxx_method > rbd.set_features flags 3 0x7f9c8d4a3390 > -201> 2016-04-15 23:28:24.827185 7f9ca0a5b800 10 register_cxx_method > rbd.get_size flags 1 0x7f9c8d4af940 > -200> 2016-04-15 23:28:24.827329 7f9ca0a5b800 10 register_cxx_method > rbd.set_size flags 3 0x7f9c8d4a2bf0 > -199> 2016-04-15 23:28:24.827420 7f9ca0a5b800 10 register_cxx_method > rbd.get_snapcontext flags 1 0x7f9c8d4aa510 > -198> 2016-04-15 23:28:24.827489 7f9ca0a5b800 10 register_cxx_method > rbd.get_object_prefix flags 1 0x7f9c8d4a0060 > -197> 2016-04-15 23:28:24.827557 7f9ca0a5b800 10 register_cxx_method > rbd.get_snapshot_name flags 1 0x7f9c8d4b1900 > -196> 2016-04-15 23:28:24.827637 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_add flags 3 0x7f9c8d4b2600 > -195> 2016-04-15 23:28:24.827722 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_remove flags 3 0x7f9c8d4b2300 > -194> 2016-04-15 23:28:24.827800 7f9ca0a5b800 10 register_cxx_method > rbd.snapshot_rename flags 3 0x7f9c8d4b1b50 > -193> 2016-04-15 23:28:24.827869 7f9ca0a5b800 10 register_cxx_method > rbd.get_all_features flags 1 0x7f9c8d49d240 > -192> 2016-04-15 23:28:24.827940 7f9ca0a5b800 10 register_cxx_method > rbd.copyup flags 3 0x7f9c8d49d380 > -191> 2016-04-15 23:28:24.828017 7f9ca0a5b800 10 register_cxx_method > rbd.get_parent flags 1 0x7f9c8d4b1410 > -190> 2016-04-15 23:28:24.828099 7f9ca0a5b800 10 register_cxx_method > rbd.set_parent flags 3 0x7f9c8d4a7560 > -189> 2016-04-15 23:28:24.828192 7f9ca0a5b800 10 register_cxx_method > rbd.remove_parent flags 3 0x7f9c8d4a8cb0 > -188> 2016-04-15 23:28:24.828281 7f9ca0a5b800 10 register_cxx_method > rbd.set_protection_status flags 3 0x7f9c8d4b00b0 > -187> 2016-04-15 23:28:24.828385 7f9ca0a5b800 10 register_cxx_method > rbd.get_protection_status flags 1 0x7f9c8d4afdd0 > -186> 2016-04-15 23:28:24.828484 7f9ca0a5b800 10 register_cxx_method > rbd.get_stripe_unit_count flags 1 0x7f9c8d49f640 > -185> 2016-04-15 23:28:24.828584 7f9ca0a5b800 10 register_cxx_method > rbd.set_stripe_unit_count flags 3 0x7f9c8d4a5950 > -184> 2016-04-15 23:28:24.828674 7f9ca0a5b800 10 register_cxx_method > rbd.get_flags flags 1 0x7f9c8d4b0880 > -183> 2016-04-15 23:28:24.828800 7f9ca0a5b800 10 register_cxx_method > rbd.set_flags flags 3 0x7f9c8d4b0c30 > -182> 2016-04-15 23:28:24.828906 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_list flags 1 0x7f9c8d4ac290 > -181> 2016-04-15 23:28:24.828997 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_set flags 3 0x7f9c8d4ac9e0 > -180> 2016-04-15 23:28:24.829100 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_remove flags 3 0x7f9c8d49e330 > -179> 2016-04-15 23:28:24.829179 7f9ca0a5b800 10 register_cxx_method > rbd.metadata_get flags 1 0x7f9c8d4a28f0 > -178> 2016-04-15 23:28:24.829261 7f9ca0a5b800 10 register_cxx_method > rbd.add_child flags 3 0x7f9c8d4a9f90 > -177> 2016-04-15 23:28:24.829341 7f9ca0a5b800 10 register_cxx_method > rbd.remove_child flags 3 0x7f9c8d4abba0 > -176> 2016-04-15 23:28:24.829432 7f9ca0a5b800 10 register_cxx_method > rbd.get_children flags 1 0x7f9c8d4a9c00 > -175> 2016-04-15 23:28:24.829522 7f9ca0a5b800 10 register_cxx_method > rbd.get_id flags 1 0x7f9c8d4a2640 > -174> 2016-04-15 23:28:24.829605 7f9ca0a5b800 10 register_cxx_method > rbd.set_id flags 3 0x7f9c8d4a5f30 > -173> 2016-04-15 23:28:24.829683 7f9ca0a5b800 10 register_cxx_method > rbd.dir_get_id flags 1 0x7f9c8d4a0920 > -172> 2016-04-15 23:28:24.829761 7f9ca0a5b800 10 register_cxx_method > rbd.dir_get_name flags 1 0x7f9c8d4a0c00 > -171> 2016-04-15 23:28:24.829852 7f9ca0a5b800 10 register_cxx_method > rbd.dir_list flags 1 0x7f9c8d4adc20 > -170> 2016-04-15 23:28:24.829998 7f9ca0a5b800 10 register_cxx_method > rbd.dir_add_image flags 3 0x7f9c8d4ad9d0 > -169> 2016-04-15 23:28:24.830111 7f9ca0a5b800 10 register_cxx_method > rbd.dir_remove_image flags 3 0x7f9c8d4a0770 > -168> 2016-04-15 23:28:24.830193 7f9ca0a5b800 10 register_cxx_method > rbd.dir_rename_image flags 3 0x7f9c8d4ad770 > -167> 2016-04-15 23:28:24.830273 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_load flags 1 0x7f9c8d4a8090 > -166> 2016-04-15 23:28:24.830361 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_save flags 3 0x7f9c8d4a8710 > -165> 2016-04-15 23:28:24.830439 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_resize flags 3 0x7f9c8d4a81c0 > -164> 2016-04-15 23:28:24.830525 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_update flags 3 0x7f9c8d4a61b0 > -163> 2016-04-15 23:28:24.830613 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_snap_add flags 3 0x7f9c8d4a8520 > -162> 2016-04-15 23:28:24.830711 7f9ca0a5b800 10 register_cxx_method > rbd.object_map_snap_remove flags 3 0x7f9c8d4a8940 > -161> 2016-04-15 23:28:24.830799 7f9ca0a5b800 10 register_cxx_method > rbd.snap_list flags 1 0x7f9c8d4a7d90 > -160> 2016-04-15 23:28:24.830961 7f9ca0a5b800 10 register_cxx_method > rbd.snap_add flags 3 0x7f9c8d4a4230 > -159> 2016-04-15 23:28:24.831068 7f9ca0a5b800 10 register_cxx_method > rbd.snap_remove flags 3 0x7f9c8d4a52e0 > -158> 2016-04-15 23:28:24.831162 7f9ca0a5b800 10 register_cxx_method > rbd.snap_rename flags 3 0x7f9c8d4a4b00 > -157> 2016-04-15 23:28:24.831254 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_uuid_get flags 1 0x7f9c8d49fd30 > -156> 2016-04-15 23:28:24.831338 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_uuid_set flags 3 0x7f9c8d4a4850 > -155> 2016-04-15 23:28:24.831430 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_mode_get flags 1 0x7f9c8d49fae0 > -154> 2016-04-15 23:28:24.831525 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_mode_set flags 3 0x7f9c8d4aae80 > -153> 2016-04-15 23:28:24.831610 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_list flags 1 0x7f9c8d4ab340 > -152> 2016-04-15 23:28:24.831697 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_add flags 3 0x7f9c8d4ab570 > -151> 2016-04-15 23:28:24.831780 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_remove flags 3 0x7f9c8d49e580 > -150> 2016-04-15 23:28:24.831860 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_set_client flags 3 0x7f9c8d4a1a60 > -149> 2016-04-15 23:28:24.831942 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_peer_set_cluster flags 3 0x7f9c8d4a1d20 > -148> 2016-04-15 23:28:24.832018 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_list flags 1 0x7f9c8d4ae6c0 > -147> 2016-04-15 23:28:24.832092 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_get_image_id flags 1 0x7f9c8d4a0ee0 > -146> 2016-04-15 23:28:24.832168 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_get flags 1 0x7f9c8d4a1400 > -145> 2016-04-15 23:28:24.832249 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_set flags 3 0x7f9c8d4a4050 > -144> 2016-04-15 23:28:24.832340 7f9ca0a5b800 10 register_cxx_method > rbd.mirror_image_remove flags 3 0x7f9c8d4a2500 > -143> 2016-04-15 23:28:24.832435 7f9ca0a5b800 10 _load_class rbd success > -142> 2016-04-15 23:28:24.832523 7f9ca0a5b800 10 open_all_classes found > version > -141> 2016-04-15 23:28:24.832604 7f9ca0a5b800 10 _get_class adding new > class name version 0x7f9caa628c88 > -140> 2016-04-15 23:28:24.832697 7f9ca0a5b800 10 _load_class version from > /usr/lib/rados-classes/libcls_version.so > -139> 2016-04-15 23:28:24.833724 7f9ca0a5b800 1 <cls> > cls/version/cls_version.cc:228: Loaded version class! > -138> 2016-04-15 23:28:24.833816 7f9ca0a5b800 10 register_class version > status 3 > -137> 2016-04-15 23:28:24.833947 7f9ca0a5b800 10 register_cxx_method > version.set flags 3 0x7f9c8d28eb60 > -136> 2016-04-15 23:28:24.834052 7f9ca0a5b800 10 register_cxx_method > version.inc flags 3 0x7f9c8d28fdd0 > -135> 2016-04-15 23:28:24.834147 7f9ca0a5b800 10 register_cxx_method > version.inc_conds flags 3 0x7f9c8d28fdd0 > -134> 2016-04-15 23:28:24.834237 7f9ca0a5b800 10 register_cxx_method > version.read flags 1 0x7f9c8d28e880 > -133> 2016-04-15 23:28:24.834347 7f9ca0a5b800 10 register_cxx_method > version.check_conds flags 1 0x7f9c8d28f410 > -132> 2016-04-15 23:28:24.834441 7f9ca0a5b800 10 _load_class version > success > -131> 2016-04-15 23:28:24.834533 7f9ca0a5b800 10 open_all_classes found > kvs > -130> 2016-04-15 23:28:24.834624 7f9ca0a5b800 10 _get_class adding new > class name kvs 0x7f9caa628da8 > -129> 2016-04-15 23:28:24.834709 7f9ca0a5b800 10 _load_class kvs from > /usr/lib/rados-classes/libcls_kvs.so > -128> 2016-04-15 23:28:24.836078 7f9ca0a5b800 10 register_class kvs status > 3 > -127> 2016-04-15 23:28:24.836240 7f9ca0a5b800 10 register_cxx_method > kvs.get_idata_from_key flags 1 0x7f9c8d07e990 > -126> 2016-04-15 23:28:24.836344 7f9ca0a5b800 10 register_cxx_method > kvs.get_next_idata flags 1 0x7f9c8d080500 > -125> 2016-04-15 23:28:24.836431 7f9ca0a5b800 10 register_cxx_method > kvs.get_prev_idata flags 1 0x7f9c8d07d0f0 > -124> 2016-04-15 23:28:24.836521 7f9ca0a5b800 10 register_cxx_method > kvs.read_many flags 1 0x7f9c8d078a40 > -123> 2016-04-15 23:28:24.836606 7f9ca0a5b800 10 register_cxx_method > kvs.check_writable flags 3 0x7f9c8d0783a0 > -122> 2016-04-15 23:28:24.836700 7f9ca0a5b800 10 register_cxx_method > kvs.assert_size_in_bound flags 2 0x7f9c8d078640 > -121> 2016-04-15 23:28:24.836846 7f9ca0a5b800 10 register_cxx_method > kvs.omap_insert flags 2 0x7f9c8d07a870 > -120> 2016-04-15 23:28:24.836957 7f9ca0a5b800 10 register_cxx_method > kvs.create_with_omap flags 2 0x7f9c8d07c370 > -119> 2016-04-15 23:28:24.837052 7f9ca0a5b800 10 register_cxx_method > kvs.omap_remove flags 2 0x7f9c8d0799e0 > -118> 2016-04-15 23:28:24.837197 7f9ca0a5b800 10 register_cxx_method > kvs.maybe_read_for_balance flags 1 0x7f9c8d07b690 > -117> 2016-04-15 23:28:24.837291 7f9ca0a5b800 10 _load_class kvs success > -116> 2016-04-15 23:28:24.837369 7f9ca0a5b800 10 open_all_classes found > hello > -115> 2016-04-15 23:28:24.837447 7f9ca0a5b800 10 _get_class adding new > class name hello 0x7f9caa628fe8 > -114> 2016-04-15 23:28:24.837527 7f9ca0a5b800 10 _load_class hello from > /usr/lib/rados-classes/libcls_hello.so > -113> 2016-04-15 23:28:24.838591 7f9ca0a5b800 0 <cls> > cls/hello/cls_hello.cc:305: loading cls_hello > -112> 2016-04-15 23:28:24.838757 7f9ca0a5b800 10 register_class hello > status 3 > -111> 2016-04-15 23:28:24.838851 7f9ca0a5b800 10 register_cxx_method > hello.say_hello flags 1 0x7f9c8ce6f8f0 > -110> 2016-04-15 23:28:24.838948 7f9ca0a5b800 10 register_cxx_method > hello.record_hello flags 10 0x7f9c8ce70a50 > -109> 2016-04-15 23:28:24.839384 7f9ca0a5b800 10 register_cxx_method > hello.writes_dont_return_data flags 2 0x7f9c8ce6fd20 > -108> 2016-04-15 23:28:24.839491 7f9ca0a5b800 10 register_cxx_method > hello.replay flags 1 0x7f9c8ce6f880 > -107> 2016-04-15 23:28:24.839581 7f9ca0a5b800 10 register_cxx_method > hello.turn_it_to_11 flags 11 0x7f9c8ce6ffe0 > -106> 2016-04-15 23:28:24.839669 7f9ca0a5b800 10 register_cxx_method > hello.bad_reader flags 2 0x7f9c8ce6f870 > -105> 2016-04-15 23:28:24.839750 7f9ca0a5b800 10 register_cxx_method > hello.bad_writer flags 1 0x7f9c8ce6f860 > -104> 2016-04-15 23:28:24.839839 7f9ca0a5b800 10 _load_class hello success > -103> 2016-04-15 23:28:24.839926 7f9ca0a5b800 10 open_all_classes found > refcount > -102> 2016-04-15 23:28:24.840019 7f9ca0a5b800 10 _get_class adding new > class name refcount 0x7f9caa629108 > -101> 2016-04-15 23:28:24.840110 7f9ca0a5b800 10 _load_class refcount from > /usr/lib/rados-classes/libcls_refcount.so > -100> 2016-04-15 23:28:24.849726 7f9ca0a5b800 1 <cls> > cls/refcount/cls_refcount.cc:232: Loaded refcount class! > -99> 2016-04-15 23:28:24.849814 7f9ca0a5b800 10 register_class refcount > status 3 > -98> 2016-04-15 23:28:24.849994 7f9ca0a5b800 10 register_cxx_method > refcount.get flags 3 0x7f9c8cbc5e80 > -97> 2016-04-15 23:28:24.850131 7f9ca0a5b800 10 register_cxx_method > refcount.put flags 3 0x7f9c8cbc50f0 > -96> 2016-04-15 23:28:24.850228 7f9ca0a5b800 10 register_cxx_method > refcount.set flags 3 0x7f9c8cbc4060 > -95> 2016-04-15 23:28:24.850316 7f9ca0a5b800 10 register_cxx_method > refcount.read flags 1 0x7f9c8cbc57b0 > -94> 2016-04-15 23:28:24.850397 7f9ca0a5b800 10 _load_class refcount > success > -93> 2016-04-15 23:28:24.850480 7f9ca0a5b800 10 open_all_classes found > user > -92> 2016-04-15 23:28:24.850567 7f9ca0a5b800 10 _get_class adding new > class name user 0x7f9caa629228 > -91> 2016-04-15 23:28:24.850648 7f9ca0a5b800 10 _load_class user from > /usr/lib/rados-classes/libcls_user.so > -90> 2016-04-15 23:28:24.851663 7f9ca0a5b800 1 <cls> > cls/user/cls_user.cc:375: Loaded user class! > -89> 2016-04-15 23:28:24.851749 7f9ca0a5b800 10 register_class user > status 3 > -88> 2016-04-15 23:28:24.851874 7f9ca0a5b800 10 register_cxx_method > user.set_buckets_info flags 3 0x7f9c8c91edc0 > -87> 2016-04-15 23:28:24.851965 7f9ca0a5b800 10 register_cxx_method > user.complete_stats_sync flags 3 0x7f9c8c91e5f0 > -86> 2016-04-15 23:28:24.852041 7f9ca0a5b800 10 register_cxx_method > user.remove_bucket flags 3 0x7f9c8c91d410 > -85> 2016-04-15 23:28:24.852121 7f9ca0a5b800 10 register_cxx_method > user.list_buckets flags 1 0x7f9c8c91ffb0 > -84> 2016-04-15 23:28:24.852201 7f9ca0a5b800 10 register_cxx_method > user.get_header flags 1 0x7f9c8c91bee0 > -83> 2016-04-15 23:28:24.852274 7f9ca0a5b800 10 _load_class user success > -82> 2016-04-15 23:28:24.852348 7f9ca0a5b800 10 open_all_classes found > numops > -81> 2016-04-15 23:28:24.852421 7f9ca0a5b800 10 _get_class adding new > class name numops 0x7f9caa629348 > -80> 2016-04-15 23:28:24.852495 7f9ca0a5b800 10 _load_class numops from > /usr/lib/rados-classes/libcls_numops.so > -79> 2016-04-15 23:28:24.853410 7f9ca0a5b800 10 register_class numops > status 3 > -78> 2016-04-15 23:28:24.853551 7f9ca0a5b800 10 register_cxx_method > numops.add flags 3 0x7f9c8c716860 > -77> 2016-04-15 23:28:24.853662 7f9ca0a5b800 10 register_cxx_method > numops.mul flags 3 0x7f9c8c715cc0 > -76> 2016-04-15 23:28:24.853760 7f9ca0a5b800 10 _load_class numops > success > -75> 2016-04-15 23:28:24.853837 7f9ca0a5b800 10 open_all_classes found > replica_log > -74> 2016-04-15 23:28:24.853924 7f9ca0a5b800 10 _get_class adding new > class name replica_log 0x7f9caa629468 > -73> 2016-04-15 23:28:24.854006 7f9ca0a5b800 10 _load_class replica_log > from /usr/lib/rados-classes/libcls_replica_log.so > -72> 2016-04-15 23:28:24.854977 7f9ca0a5b800 1 <cls> > cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! > -71> 2016-04-15 23:28:24.855055 7f9ca0a5b800 10 register_class > replica_log status 3 > -70> 2016-04-15 23:28:24.855167 7f9ca0a5b800 10 register_cxx_method > replica_log.set flags 3 0x7f9c8c50eed0 > -69> 2016-04-15 23:28:24.855261 7f9ca0a5b800 10 register_cxx_method > replica_log.get flags 1 0x7f9c8c50fed0 > -68> 2016-04-15 23:28:24.855348 7f9ca0a5b800 10 register_cxx_method > replica_log.delete flags 3 0x7f9c8c50f670 > -67> 2016-04-15 23:28:24.855428 7f9ca0a5b800 10 _load_class replica_log > success > -66> 2016-04-15 23:28:24.855507 7f9ca0a5b800 10 open_all_classes found > timeindex > -65> 2016-04-15 23:28:24.855591 7f9ca0a5b800 10 _get_class adding new > class name timeindex 0x7f9caa629588 > -64> 2016-04-15 23:28:24.855677 7f9ca0a5b800 10 _load_class timeindex > from /usr/lib/rados-classes/libcls_timeindex.so > -63> 2016-04-15 23:28:24.856621 7f9ca0a5b800 1 <cls> > cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! > -62> 2016-04-15 23:28:24.856699 7f9ca0a5b800 10 register_class timeindex > status 3 > -61> 2016-04-15 23:28:24.856858 7f9ca0a5b800 10 register_cxx_method > timeindex.add flags 3 0x7f9c8c306690 > -60> 2016-04-15 23:28:24.856963 7f9ca0a5b800 10 register_cxx_method > timeindex.list flags 1 0x7f9c8c307200 > -59> 2016-04-15 23:28:24.857040 7f9ca0a5b800 10 register_cxx_method > timeindex.trim flags 3 0x7f9c8c3068d0 > -58> 2016-04-15 23:28:24.857118 7f9ca0a5b800 10 _load_class timeindex > success > -57> 2016-04-15 23:28:24.857202 7f9ca0a5b800 10 open_all_classes found > lock > -56> 2016-04-15 23:28:24.857279 7f9ca0a5b800 10 _get_class adding new > class name lock 0x7f9caa6296a8 > -55> 2016-04-15 23:28:24.857353 7f9ca0a5b800 10 _load_class lock from > /usr/lib/rados-classes/libcls_lock.so > -54> 2016-04-15 23:28:24.858409 7f9ca0a5b800 10 register_class lock > status 3 > -53> 2016-04-15 23:28:24.858567 7f9ca0a5b800 10 register_cxx_method > lock.lock flags 11 0x7f9c8c0fdb80 > -52> 2016-04-15 23:28:24.858672 7f9ca0a5b800 10 register_cxx_method > lock.unlock flags 11 0x7f9c8c0fd560 > -51> 2016-04-15 23:28:24.858760 7f9ca0a5b800 10 register_cxx_method > lock.break_lock flags 3 0x7f9c8c0fcf50 > -50> 2016-04-15 23:28:24.858850 7f9ca0a5b800 10 register_cxx_method > lock.get_info flags 1 0x7f9c8c0fbc20 > -49> 2016-04-15 23:28:24.858942 7f9ca0a5b800 10 register_cxx_method > lock.list_locks flags 1 0x7f9c8c0fa770 > -48> 2016-04-15 23:28:24.859027 7f9ca0a5b800 10 register_cxx_method > lock.assert_locked flags 9 0x7f9c8c0fb180 > -47> 2016-04-15 23:28:24.859114 7f9ca0a5b800 10 _load_class lock success > -46> 2016-04-15 23:28:24.859193 7f9ca0a5b800 10 open_all_classes found > log > -45> 2016-04-15 23:28:24.859275 7f9ca0a5b800 10 _get_class adding new > class name log 0x7f9caa6298e8 > -44> 2016-04-15 23:28:24.859361 7f9ca0a5b800 10 _load_class log from > /usr/lib/rados-classes/libcls_log.so > -43> 2016-04-15 23:28:24.860294 7f9ca0a5b800 1 <cls> > cls/log/cls_log.cc:317: Loaded log class! > -42> 2016-04-15 23:28:24.860369 7f9ca0a5b800 10 register_class log status > 3 > -41> 2016-04-15 23:28:24.860486 7f9ca0a5b800 10 register_cxx_method > log.add flags 3 0x7f9c8bef0320 > -40> 2016-04-15 23:28:24.860592 7f9ca0a5b800 10 register_cxx_method > log.list flags 1 0x7f9c8bef1660 > -39> 2016-04-15 23:28:24.860680 7f9ca0a5b800 10 register_cxx_method > log.trim flags 3 0x7f9c8bef0d30 > -38> 2016-04-15 23:28:24.860807 7f9ca0a5b800 10 register_cxx_method > log.info flags 1 0x7f9c8beefc70 > -37> 2016-04-15 23:28:24.860915 7f9ca0a5b800 10 _load_class log success > -36> 2016-04-15 23:28:24.860998 7f9ca0a5b800 10 open_all_classes found > rgw > -35> 2016-04-15 23:28:24.861086 7f9ca0a5b800 10 _get_class adding new > class name rgw 0x7f9caa629a08 > -34> 2016-04-15 23:28:24.861167 7f9ca0a5b800 10 _load_class rgw from > /usr/lib/rados-classes/libcls_rgw.so > -33> 2016-04-15 23:28:24.870017 7f9ca0a5b800 1 <cls> > cls/rgw/cls_rgw.cc:3206: Loaded rgw class! > -32> 2016-04-15 23:28:24.870067 7f9ca0a5b800 10 register_class rgw status > 3 > -31> 2016-04-15 23:28:24.870080 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_init_index flags 3 0x7f9c8bc0c620 > -30> 2016-04-15 23:28:24.870103 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_set_tag_timeout flags 3 0x7f9c8bc15750 > -29> 2016-04-15 23:28:24.870127 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_list flags 1 0x7f9c8bc145e0 > -28> 2016-04-15 23:28:24.870142 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_check_index flags 1 0x7f9c8bc164a0 > -27> 2016-04-15 23:28:24.870157 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_rebuild_index flags 3 0x7f9c8bc16300 > -26> 2016-04-15 23:28:24.870173 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_prepare_op flags 3 0x7f9c8bc196f0 > -25> 2016-04-15 23:28:24.870187 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_complete_op flags 3 0x7f9c8bc1a320 > -24> 2016-04-15 23:28:24.870202 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_link_olh flags 3 0x7f9c8bc1cce0 > -23> 2016-04-15 23:28:24.870219 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_unlink_instance flags 3 0x7f9c8bc1b8a0 > -22> 2016-04-15 23:28:24.870236 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_read_olh_log flags 1 0x7f9c8bc1e490 > -21> 2016-04-15 23:28:24.870253 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_trim_olh_log flags 3 0x7f9c8bc17f50 > -20> 2016-04-15 23:28:24.870268 7f9ca0a5b800 10 register_cxx_method > rgw.bucket_clear_olh flags 3 0x7f9c8bc17540 > -19> 2016-04-15 23:28:24.870285 7f9ca0a5b800 10 register_cxx_method > rgw.obj_remove flags 3 0x7f9c8bc138b0 > -18> 2016-04-15 23:28:24.870299 7f9ca0a5b800 10 register_cxx_method > rgw.obj_store_pg_ver flags 2 0x7f9c8bc09b60 > -17> 2016-04-15 23:28:24.870314 7f9ca0a5b800 10 register_cxx_method > rgw.obj_check_attrs_prefix flags 1 0x7f9c8bc0c910 > -16> 2016-04-15 23:28:24.870328 7f9ca0a5b800 10 register_cxx_method > rgw.obj_check_mtime flags 1 0x7f9c8bc07d70 > -15> 2016-04-15 23:28:24.870341 7f9ca0a5b800 10 register_cxx_method > rgw.bi_get flags 1 0x7f9c8bc0aae0 > -14> 2016-04-15 23:28:24.870359 7f9ca0a5b800 10 register_cxx_method > rgw.bi_put flags 3 0x7f9c8bc08fc0 > -13> 2016-04-15 23:28:24.870373 7f9ca0a5b800 10 register_cxx_method > rgw.bi_list flags 1 0x7f9c8bc11ca0 > -12> 2016-04-15 23:28:24.870392 7f9ca0a5b800 10 register_cxx_method > rgw.bi_log_list flags 1 0x7f9c8bc0e790 > -11> 2016-04-15 23:28:24.870410 7f9ca0a5b800 10 register_cxx_method > rgw.bi_log_trim flags 3 0x7f9c8bc0eec0 > -10> 2016-04-15 23:28:24.870426 7f9ca0a5b800 10 register_cxx_method > rgw.dir_suggest_changes flags 3 0x7f9c8bc16740 > -9> 2016-04-15 23:28:24.870441 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_add flags 3 0x7f9c8bc10cc0 > -8> 2016-04-15 23:28:24.870458 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_read flags 1 0x7f9c8bc100c0 > -7> 2016-04-15 23:28:24.870475 7f9ca0a5b800 10 register_cxx_method > rgw.user_usage_log_trim flags 3 0x7f9c8bc10720 > -6> 2016-04-15 23:28:24.870495 7f9ca0a5b800 10 register_cxx_method > rgw.gc_set_entry flags 3 0x7f9c8bc0beb0 > -5> 2016-04-15 23:28:24.870510 7f9ca0a5b800 10 register_cxx_method > rgw.gc_defer_entry flags 3 0x7f9c8bc0b970 > -4> 2016-04-15 23:28:24.870525 7f9ca0a5b800 10 register_cxx_method > rgw.gc_list flags 1 0x7f9c8bc0cff0 > -3> 2016-04-15 23:28:24.870540 7f9ca0a5b800 10 register_cxx_method > rgw.gc_remove flags 3 0x7f9c8bc0c320 > -2> 2016-04-15 23:28:24.870555 7f9ca0a5b800 10 _load_class rgw success > -1> 2016-04-15 23:28:24.870639 7f9ca0a5b800 20 osd.3 0 get_map 16024 - > loading and decoding 0x7f9caa5f2900 > 0> 2016-04-15 23:28:24.874608 7f9ca0a5b800 -1 osd/OSD.h: In function > 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f9ca0a5b800 time 2016-04-15 > 23:28:24.870922 > osd/OSD.h: 885: FAILED assert(ret) > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x82) [0x7f9ca04e14f2] > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] > 3: (OSD::init()+0x1862) [0x7f9c9fe16a52] > 4: (main()+0x2b05) [0x7f9c9fd7c735] > 5: (__libc_start_main()+0xf5) [0x7f9c9c912b45] > 6: (()+0x337197) [0x7f9c9fdc7197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 20/20 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 newstore > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 1/ 5 kinetic > 1/ 5 fuse > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal (Aborted) ** > in thread 7f9ca0a5b800 thread_name:ceph-osd > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (()+0x949117) [0x7f9ca03d9117] > 2: (()+0xf8d0) [0x7f9c9e8dc8d0] > 3: (gsignal()+0x37) [0x7f9c9c926067] > 4: (abort()+0x148) [0x7f9c9c927448] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x256) [0x7f9ca04e16c6] > 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] > 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] > 8: (main()+0x2b05) [0x7f9c9fd7c735] > 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] > 10: (()+0x337197) [0x7f9c9fdc7197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- begin dump of recent events --- > 0> 2016-04-15 23:28:24.885310 7f9ca0a5b800 -1 *** Caught signal > (Aborted) ** > in thread 7f9ca0a5b800 thread_name:ceph-osd > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (()+0x949117) [0x7f9ca03d9117] > 2: (()+0xf8d0) [0x7f9c9e8dc8d0] > 3: (gsignal()+0x37) [0x7f9c9c926067] > 4: (abort()+0x148) [0x7f9c9c927448] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x256) [0x7f9ca04e16c6] > 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f9c9fe6883d] > 7: (OSD::init()+0x1862) [0x7f9c9fe16a52] > 8: (main()+0x2b05) [0x7f9c9fd7c735] > 9: (__libc_start_main()+0xf5) [0x7f9c9c912b45] > 10: (()+0x337197) [0x7f9c9fdc7197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 20/20 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 newstore > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 1/ 5 kinetic > 1/ 5 fuse > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > > > > On Friday, April 15, 2016 11:24 PM, huang jun <hjwsm1989@xxxxxxxxx> wrote: > > > Can you set 'debug_osd = 20' in ceph.conf and restart the osd again, > and post the corrupt log. > I doubt it's problem related to "0 byte osdmap" decode problem? > > 2016-04-16 12:14 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: >> I've been successfully running cephfs on my Debian Jessies for a while and >> one day after power outage, MDS wasn't happy. MDS crashing after it was >> done loading, increasing the memory utilization quite a bit. I was >> running >> infernalis 9.2.0 and did successful upgrade from Hammer before... so I >> thought I may have hit a bug and decided to try 9.2.1. >> >> In 9.2.1, it was not happy that my journal didn't have permission for user >> ceph. So corrected it. Then all of my OSDs are no longer starting. >> Failing with similar messages as below. I upgraded to Jewel, as I didn't >> see too much more complexitiy to upgrade from Infernalis and am still >> seeing >> these errors. >> >> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to 1001:1001 >> (ceph:ceph) >> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore empty >> --pid-file >> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log #20899 >> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 #20899 >> >> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 #20898 >> >> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >> cls/hello/cls_hello.cc:305: >> loading cls_hello >> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >> 'OSDMapRef >> OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time 2016-04-15 >> 22:47:06.494680 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f65fb6364f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >> 4: (main()+0x2b05) [0x7f65faed1735] >> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >> 6: (()+0x337197) [0x7f65faf1c197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> -78> 2016-04-15 22:47:04.873688 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perfcounters_dump hook 0x7f66067e2030 >> -77> 2016-04-15 22:47:04.873771 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command 1 hook 0x7f66067e2030 >> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perf dump hook 0x7f66067e2030 >> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perfcounters_schema hook 0x7f66067e2030 >> -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perf dump hook 0x7f66067e2030 >> -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perfcounters_schema hook 0x7f66067e2030 >> -74> 2016-04-15 22:47:04.873860 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command 2 hook 0x7f66067e2030 >> -73> 2016-04-15 22:47:04.873886 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perf schema hook 0x7f66067e2030 >> -72> 2016-04-15 22:47:04.873916 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command perf reset hook 0x7f66067e2030 >> -71> 2016-04-15 22:47:04.873943 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command config show hook 0x7f66067e2030 >> -70> 2016-04-15 22:47:04.873974 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command config set hook 0x7f66067e2030 >> -69> 2016-04-15 22:47:04.874000 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command config get hook 0x7f66067e2030 >> -68> 2016-04-15 22:47:04.874029 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command config diff hook 0x7f66067e2030 >> -67> 2016-04-15 22:47:04.874055 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command log flush hook 0x7f66067e2030 >> -66> 2016-04-15 22:47:04.874082 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command log dump hook 0x7f66067e2030 >> -65> 2016-04-15 22:47:04.874109 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command log reopen hook 0x7f66067e2030 >> -64> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to >> 1001:1001 >> (ceph:ceph) >> -63> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 >> (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 >> -62> 2016-04-15 22:47:04.900224 7f65fbbb0800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -61> 2016-04-15 22:47:04.900286 7f65fbbb0800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6802/1284 need_addr=0 >> -60> 2016-04-15 22:47:04.900350 7f65fbbb0800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -59> 2016-04-15 22:47:04.900375 7f65fbbb0800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6802/1284 need_addr=0 >> -58> 2016-04-15 22:47:04.900443 7f65fbbb0800 1 -- 192.168.2.31:0/0 >> learned my addr 192.168.2.31:0/0 >> -57> 2016-04-15 22:47:04.900475 7f65fbbb0800 1 accepter.accepter.bind >> my_inst.addr is 192.168.2.31:6803/1284 need_addr=0 >> -56> 2016-04-15 22:47:04.900538 7f65fbbb0800 1 -- 192.168.1.31:0/0 >> learned my addr 192.168.1.31:0/0 >> -55> 2016-04-15 22:47:04.900562 7f65fbbb0800 1 accepter.accepter.bind >> my_inst.addr is 192.168.1.31:6803/1284 need_addr=0 >> -54> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore >> empty --pid-file >> -53> 2016-04-15 22:47:04.909743 7f65fbbb0800 5 asok(0x7f660689a000) >> init >> /var/run/ceph/ceph-osd.3.asok >> -52> 2016-04-15 22:47:04.909792 7f65fbbb0800 5 asok(0x7f660689a000) >> bind_and_listen /var/run/ceph/ceph-osd.3.asok >> -51> 2016-04-15 22:47:04.909891 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command 0 hook 0x7f66067de0d8 >> -50> 2016-04-15 22:47:04.909928 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command version hook 0x7f66067de0d8 >> -49> 2016-04-15 22:47:04.909955 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command git_version hook 0x7f66067de0d8 >> -48> 2016-04-15 22:47:04.909988 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command help hook 0x7f66067e21e0 >> -47> 2016-04-15 22:47:04.910015 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command get_command_descriptions hook 0x7f66067e21f0 >> -46> 2016-04-15 22:47:04.910205 7f65f43c9700 5 asok(0x7f660689a000) >> entry start >> -45> 2016-04-15 22:47:04.910330 7f65fbbb0800 10 monclient(hunting): >> build_initial_monmap >> -44> 2016-04-15 22:47:04.939070 7f65fbbb0800 5 adding auth protocol: >> cephx >> -43> 2016-04-15 22:47:04.939118 7f65fbbb0800 5 adding auth protocol: >> cephx >> -42> 2016-04-15 22:47:04.939986 7f65fbbb0800 5 asok(0x7f660689a000) >> register_command objecter_requests hook 0x7f66067e22b0 >> -41> 2016-04-15 22:47:04.940256 7f65fbbb0800 1 -- >> 192.168.1.31:6802/1284 >> messenger.start >> -40> 2016-04-15 22:47:04.940413 7f65fbbb0800 1 -- :/0 messenger.start >> -39> 2016-04-15 22:47:04.940557 7f65fbbb0800 1 -- >> 192.168.1.31:6803/1284 >> messenger.start >> -38> 2016-04-15 22:47:04.940686 7f65fbbb0800 1 -- >> 192.168.2.31:6803/1284 >> messenger.start >> -37> 2016-04-15 22:47:04.940798 7f65fbbb0800 1 -- >> 192.168.2.31:6802/1284 >> messenger.start >> -36> 2016-04-15 22:47:04.940899 7f65fbbb0800 1 -- :/0 messenger.start >> -35> 2016-04-15 22:47:04.941223 7f65fbbb0800 2 osd.3 0 mounting >> /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal >> -34> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 >> filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) >> -33> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP >> ioctl is disabled via 'filestore fiemap' config option >> -32> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config >> option >> -31> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice >> is >> supported >> -30> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 >> genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: >> syncfs(2) >> syscall fully supported (by glibc and kernel) >> -29> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 >> xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is >> disabled by conf >> -28> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log >> #20899 >> -27> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 >> #20899 >> >> -26> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 >> #20898 >> >> -25> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 >> filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal >> mode: >> checkpoint is not enabled >> -24> 2016-04-15 22:47:06.377715 7f65fbbb0800 2 journal open >> /var/lib/ceph/osd/ceph-3/journal fsid 4f86a418-6c67-4cb4-83a1-6c123c890036 >> fs_op_seq 9829589 >> -23> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -22> 2016-04-15 22:47:06.379811 7f65fbbb0800 2 journal open advancing >> committed_seq 9829584 to fs op_seq 9829589 >> -21> 2016-04-15 22:47:06.380757 7f65fbbb0800 2 journal read_entry >> 2537717760 : seq 9829585 29509 bytes >> -20> 2016-04-15 22:47:06.380996 7f65fbbb0800 2 journal read_entry >> 2537750528 : seq 9829586 8134 bytes >> -19> 2016-04-15 22:47:06.381091 7f65fbbb0800 2 journal read_entry >> 2537762816 : seq 9829587 3064 bytes >> -18> 2016-04-15 22:47:06.381155 7f65fbbb0800 2 journal read_entry >> 2537766912 : seq 9829588 7647 bytes >> -17> 2016-04-15 22:47:06.381219 7f65fbbb0800 2 journal read_entry >> 2537775104 : seq 9829589 4737 bytes >> -16> 2016-04-15 22:47:06.381257 7f65fbbb0800 2 journal No further >> valid >> entries found, journal is most likely valid >> -15> 2016-04-15 22:47:06.381287 7f65fbbb0800 2 journal No further >> valid >> entries found, journal is most likely valid >> -14> 2016-04-15 22:47:06.381302 7f65fbbb0800 3 journal journal_replay: >> end of journal, done. >> -13> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open >> /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 >> bytes, directio = 1, aio = 1 >> -12> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 >> filestore(/var/lib/ceph/osd/ceph-3) upgrade >> -11> 2016-04-15 22:47:06.385071 7f65fbbb0800 2 osd.3 0 boot >> -10> 2016-04-15 22:47:06.415253 7f65fbbb0800 1 <cls> >> cls/statelog/cls_statelog.cc:306: Loaded log class! >> -9> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> >> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan >> -8> 2016-04-15 22:47:06.418172 7f65fbbb0800 1 <cls> >> cls/version/cls_version.cc:228: Loaded version class! >> -7> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> >> cls/hello/cls_hello.cc:305: loading cls_hello >> -6> 2016-04-15 22:47:06.426520 7f65fbbb0800 1 <cls> >> cls/refcount/cls_refcount.cc:232: Loaded refcount class! >> -5> 2016-04-15 22:47:06.427217 7f65fbbb0800 1 <cls> >> cls/user/cls_user.cc:375: Loaded user class! >> -4> 2016-04-15 22:47:06.428364 7f65fbbb0800 1 <cls> >> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! >> -3> 2016-04-15 22:47:06.428970 7f65fbbb0800 1 <cls> >> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! >> -2> 2016-04-15 22:47:06.430177 7f65fbbb0800 1 <cls> >> cls/log/cls_log.cc:317: Loaded log class! >> -1> 2016-04-15 22:47:06.438063 7f65fbbb0800 1 <cls> >> cls/rgw/cls_rgw.cc:3206: Loaded rgw class! >> 0> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function >> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time >> 2016-04-15 >> 22:47:06.494680 >> osd/OSD.h: 885: FAILED assert(ret) >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x82) [0x7f65fb6364f2] >> 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >> 3: (OSD::init()+0x1862) [0x7f65faf6ba52] >> 4: (main()+0x2b05) [0x7f65faed1735] >> 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >> 6: (()+0x337197) [0x7f65faf1c197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 0/ 5 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal (Aborted) ** >> in thread 7f65fbbb0800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f65fb52e117] >> 2: (()+0xf8d0) [0x7f65f9a318d0] >> 3: (gsignal()+0x37) [0x7f65f7a7b067] >> 4: (abort()+0x148) [0x7f65f7a7c448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f65fb6366c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >> 8: (main()+0x2b05) [0x7f65faed1735] >> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >> 10: (()+0x337197) [0x7f65faf1c197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- begin dump of recent events --- >> 0> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal >> (Aborted) ** >> in thread 7f65fbbb0800 thread_name:ceph-osd >> >> ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) >> 1: (()+0x949117) [0x7f65fb52e117] >> 2: (()+0xf8d0) [0x7f65f9a318d0] >> 3: (gsignal()+0x37) [0x7f65f7a7b067] >> 4: (abort()+0x148) [0x7f65f7a7c448] >> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x256) [0x7f65fb6366c6] >> 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] >> 7: (OSD::init()+0x1862) [0x7f65faf6ba52] >> 8: (main()+0x2b05) [0x7f65faed1735] >> 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] >> 10: (()+0x337197) [0x7f65faf1c197] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to >> interpret this. >> >> --- logging levels --- >> 0/ 5 none >> 0/ 1 lockdep >> 0/ 1 context >> 1/ 1 crush >> 1/ 5 mds >> 1/ 5 mds_balancer >> 1/ 5 mds_locker >> 1/ 5 mds_log >> 1/ 5 mds_log_expire >> 1/ 5 mds_migrator >> 0/ 1 buffer >> 0/ 1 timer >> 0/ 1 filer >> 0/ 1 striper >> 0/ 1 objecter >> 0/ 5 rados >> 0/ 5 rbd >> 0/ 5 rbd_mirror >> 0/ 5 rbd_replay >> 0/ 5 journaler >> 0/ 5 objectcacher >> 0/ 5 client >> 0/ 5 osd >> 0/ 5 optracker >> 0/ 5 objclass >> 1/ 3 filestore >> 1/ 3 journal >> 0/ 5 ms >> 1/ 5 mon >> 0/10 monc >> 1/ 5 paxos >> 0/ 5 tp >> 1/ 5 auth >> 1/ 5 crypto >> 1/ 1 finisher >> 1/ 5 heartbeatmap >> 1/ 5 perfcounter >> 1/ 5 rgw >> 1/10 civetweb >> 1/ 5 javaclient >> 1/ 5 asok >> 1/ 1 throttle >> 0/ 0 refs >> 1/ 5 xio >> 1/ 5 compressor >> 1/ 5 newstore >> 1/ 5 bluestore >> 1/ 5 bluefs >> 1/ 3 bdev >> 1/ 5 kstore >> 4/ 5 rocksdb >> 4/ 5 leveldb >> 1/ 5 kinetic >> 1/ 5 fuse >> -2/-2 (syslog threshold) >> -1/-1 (stderr threshold) >> max_recent 10000 >> max_new 1000 >> log_file /var/log/ceph/ceph-osd.3.log >> --- end dump of recent events --- >> >> What can I try to get this OSD back online? I saw some similar issues on >> google but I wasn't sure if that was actually the same issue. >> >> If I run in to MDS issue after resolving this, I'll send out another one. >> =) >> Thanks all! >> >> Regards, >> Hong > >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > > -- > Thank you! > HuangJun > > > -- Thank you! HuangJun _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |