Hi cephers,
I have a problem with my ceph cluster, i have upgraded to Lumious
12.2.10, (maybe from jewel or old lumious, i forget), but osd can not
start now.
I used ceph-objectstore-tool to cat pg info, but failed with decode errors.
OSD start also display the same errors.
I used one monitor and one osd , monitor data is destroyed, I used
ceph-montortool to restore monitor data, with mismatched crc errors.
Follows are commands execute process:
root@san01:~# ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-0/ --type filestore --pgid 4.3f --op info
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what(): buffer::malformed_input: void
object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
*** Caught signal (Aborted) **
in thread 7f73f81aa700 thread_name:ceph-objectstor
ceph version 12.2.10 (fc2b1783e3727b66315cc667af9d663d30fe7ed4)
luminous (stable)
1: (()+0x94a174) [0x55ea7ff15174]
2: (()+0x110c0) [0x7f73eced30c0]
3: (gsignal()+0xcf) [0x7f73eba6cfff]
4: (abort()+0x16a) [0x7f73eba6e42a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f73ec3850ad]
6: (()+0x8f066) [0x7f73ec383066]
7: (()+0x8f0b1) [0x7f73ec3830b1]
8: (()+0x8f2c9) [0x7f73ec3832c9]
9: (object_stat_sum_t::decode(ceph::buffer::list::iterator&)+0x5c3)
[0x55ea7fb9d893]
10: (object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x67)
[0x55ea7fbb6a27]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x1eb) [0x55ea7fbb799b]
12: (pg_info_t::decode(ceph::buffer::list::iterator&)+0x147) [0x55ea7fbc0be7]
13: (PG::read_info(ObjectStore*, spg_t, coll_t const&,
ceph::buffer::list&, pg_info_t&, PastIntervals&, unsigned
char&)+0x215) [0x55ea7fa03835]
14: (main()+0x6fb7) [0x55ea7f83b7d7]
15: (__libc_start_main()+0xf1) [0x7f73eba5a2e1]
16: (_start()+0x2a) [0x55ea7f8c1f5a]
Aborted
root@san01:~#
root@san01:~#
root@san01:~# vim /etc/ceph/ceph.conf
-bash: vim: command not found
root@san01:~# vi /etc/ceph/ceph.conf
root@san01:~# ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-0/ --op update-mon-db ^C
root@san01:~# mkdir /tmp/monstore
root@san01:~# ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-0/ --op update-mon-db --mon-store-path
/tmp/monstore
mismatched full crc: 3704629568 != 33348385
root@san01:~# ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-0/ --op update-mon-db --mon-store-path
/tmp/monstore --debug
2019-01-08 03:07:39.641860 7fdf8eb14700 0
filestore(/var/lib/ceph/osd/ceph-0/) backend xfs (magic 0x58465342)
2019-01-08 03:07:39.642118 7fdf8eb14700 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-0/) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2019-01-08 03:07:39.642134 7fdf8eb14700 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-0/) detect_features:
SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config
option
2019-01-08 03:07:39.642135 7fdf8eb14700 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-0/) detect_features:
splice() is disabled via 'filestore splice' config option
2019-01-08 03:07:39.642430 7fdf8eb14700 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-0/) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2019-01-08 03:07:39.642471 7fdf8eb14700 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-0/) detect_feature: extsize
is disabled by conf
2019-01-08 03:07:39.642850 7fdf8eb14700 0
filestore(/var/lib/ceph/osd/ceph-0/) start omap initiation
2019-01-08 03:07:39.644509 7fdf8eb14700 0
filestore(/var/lib/ceph/osd/ceph-0/) mount(1759): enabling WRITEAHEAD
journal mode: checkpoint is not enabled
2019-01-08 03:07:39.646097 7fdf8eb14700 1
filestore(/var/lib/ceph/osd/ceph-0/) upgrade(1366)
Cluster fsid=1efd00b0-fda7-4ac2-90de-0e93e09488ec
Supported features: compat={},rocompat={},incompat={1=initial feature
set(~v.18),2=pginfo object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,11=sharded
objects,12=transaction hints,13=pg meta object,14=explicit missing
set,15=fastinfo pg attr,16=deletes in missing set}
On-disk features: compat={},rocompat={},incompat={1=initial feature
set(~v.18),2=pginfo object,3=object
locator,4=last_epoch_clean,5=categories,6=hobjectpool,7=biginfo,8=leveldbinfo,9=leveldblog,10=snapmapper,12=transaction
hints,13=pg meta object,14=explicit missing set,15=fastinfo pg
attr,16=deletes in missing set}
2019-01-08 03:07:39.646479 7fdf8eb14700 0 set rocksdb option
compression = kNoCompression
2019-01-08 03:07:39.646487 7fdf8eb14700 0 set rocksdb option
level_compaction_dynamic_level_bytes = true
2019-01-08 03:07:39.646492 7fdf8eb14700 0 set rocksdb option
write_buffer_size = 33554432
2019-01-08 03:07:39.646516 7fdf8eb14700 0 set rocksdb option
compression = kNoCompression
2019-01-08 03:07:39.646519 7fdf8eb14700 0 set rocksdb option
level_compaction_dynamic_level_bytes = true
2019-01-08 03:07:39.646522 7fdf8eb14700 0 set rocksdb option
write_buffer_size = 33554432
2019-01-08 03:07:39.646616 7fdf8eb14700 4 rocksdb: RocksDB version: 5.4.0
2019-01-08 03:07:39.646623 7fdf8eb14700 4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2019-01-08 03:07:39.646624 7fdf8eb14700 4 rocksdb: Compile date Dec 6 2018
2019-01-08 03:07:39.646625 7fdf8eb14700 4 rocksdb: DB SUMMARY
2019-01-08 03:07:39.646662 7fdf8eb14700 4 rocksdb: CURRENT file: CURRENT
2019-01-08 03:07:39.646665 7fdf8eb14700 4 rocksdb: IDENTITY file: IDENTITY
2019-01-08 03:07:39.646668 7fdf8eb14700 4 rocksdb: MANIFEST file:
MANIFEST-000001 size: 13 Bytes
2019-01-08 03:07:39.646670 7fdf8eb14700 4 rocksdb: SST files in
/tmp/monstore/store.db dir, Total Num: 0, files:
2019-01-08 03:07:39.646671 7fdf8eb14700 4 rocksdb: Write Ahead Log
file in /tmp/monstore/store.db: 000003.log size: 261 ;
2019-01-08 03:07:39.646680 7fdf8eb14700 4 rocksdb:
Options.error_if_exists: 0
2019-01-08 03:07:39.646681 7fdf8eb14700 4 rocksdb:
Options.create_if_missing: 1
2019-01-08 03:07:39.646681 7fdf8eb14700 4 rocksdb:
Options.paranoid_checks: 1
2019-01-08 03:07:39.646682 7fdf8eb14700 4 rocksdb:
Options.env: 0x55f070e86ac0
2019-01-08 03:07:39.646691 7fdf8eb14700 4 rocksdb:
Options.info_log: 0x55f07287b860
2019-01-08 03:07:39.646692 7fdf8eb14700 4 rocksdb:
Options.max_open_files: -1
2019-01-08 03:07:39.646692 7fdf8eb14700 4 rocksdb:
Options.max_file_opening_threads: 16
2019-01-08 03:07:39.646693 7fdf8eb14700 4 rocksdb:
Options.use_fsync: 0
2019-01-08 03:07:39.646694 7fdf8eb14700 4 rocksdb:
Options.max_log_file_size: 0
2019-01-08 03:07:39.646694 7fdf8eb14700 4 rocksdb:
Options.max_manifest_file_size: 18446744073709551615
2019-01-08 03:07:39.646695 7fdf8eb14700 4 rocksdb:
Options.log_file_time_to_roll: 0
2019-01-08 03:07:39.646696 7fdf8eb14700 4 rocksdb:
Options.keep_log_file_num: 1000
2019-01-08 03:07:39.646696 7fdf8eb14700 4 rocksdb:
Options.recycle_log_file_num: 0
2019-01-08 03:07:39.646697 7fdf8eb14700 4 rocksdb:
Options.allow_fallocate: 1
2019-01-08 03:07:39.646697 7fdf8eb14700 4 rocksdb:
Options.allow_mmap_reads: 0
2019-01-08 03:07:39.646704 7fdf8eb14700 4 rocksdb:
Options.allow_mmap_writes: 0
2019-01-08 03:07:39.646705 7fdf8eb14700 4 rocksdb:
Options.use_direct_reads: 0
2019-01-08 03:07:39.646706 7fdf8eb14700 4 rocksdb:
Options.use_direct_io_for_flush_and_compaction: 0
2019-01-08 03:07:39.646706 7fdf8eb14700 4 rocksdb:
Options.create_missing_column_families: 0
2019-01-08 03:07:39.646707 7fdf8eb14700 4 rocksdb:
Options.db_log_dir:
2019-01-08 03:07:39.646708 7fdf8eb14700 4 rocksdb:
Options.wal_dir: /tmp/monstore/store.db
2019-01-08 03:07:39.646708 7fdf8eb14700 4 rocksdb:
Options.table_cache_numshardbits: 6
2019-01-08 03:07:39.646709 7fdf8eb14700 4 rocksdb:
Options.max_subcompactions: 1
2019-01-08 03:07:39.646709 7fdf8eb14700 4 rocksdb:
Options.max_background_flushes: 1
2019-01-08 03:07:39.646710 7fdf8eb14700 4 rocksdb:
Options.WAL_ttl_seconds: 0
2019-01-08 03:07:39.646711 7fdf8eb14700 4 rocksdb:
Options.WAL_size_limit_MB: 0
2019-01-08 03:07:39.646711 7fdf8eb14700 4 rocksdb:
Options.manifest_preallocation_size: 4194304
2019-01-08 03:07:39.646712 7fdf8eb14700 4 rocksdb:
Options.is_fd_close_on_exec: 1
2019-01-08 03:07:39.646713 7fdf8eb14700 4 rocksdb:
Options.advise_random_on_open: 1
2019-01-08 03:07:39.646713 7fdf8eb14700 4 rocksdb:
Options.db_write_buffer_size: 0
2019-01-08 03:07:39.646714 7fdf8eb14700 4 rocksdb:
Options.access_hint_on_compaction_start: 1
2019-01-08 03:07:39.646714 7fdf8eb14700 4 rocksdb:
Options.new_table_reader_for_compaction_inputs: 0
2019-01-08 03:07:39.646715 7fdf8eb14700 4 rocksdb:
Options.compaction_readahead_size: 0
2019-01-08 03:07:39.646715 7fdf8eb14700 4 rocksdb:
Options.random_access_max_buffer_size: 1048576
2019-01-08 03:07:39.646716 7fdf8eb14700 4 rocksdb:
Options.writable_file_max_buffer_size: 1048576
2019-01-08 03:07:39.646716 7fdf8eb14700 4 rocksdb:
Options.use_adaptive_mutex: 0
2019-01-08 03:07:39.646717 7fdf8eb14700 4 rocksdb:
Options.rate_limiter: (nil)
2019-01-08 03:07:39.646718 7fdf8eb14700 4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
2019-01-08 03:07:39.646718 7fdf8eb14700 4 rocksdb:
Options.bytes_per_sync: 0
2019-01-08 03:07:39.646720 7fdf8eb14700 4 rocksdb:
Options.wal_bytes_per_sync: 0
2019-01-08 03:07:39.646721 7fdf8eb14700 4 rocksdb:
Options.wal_recovery_mode: 2
2019-01-08 03:07:39.646721 7fdf8eb14700 4 rocksdb:
Options.enable_thread_tracking: 0
2019-01-08 03:07:39.646722 7fdf8eb14700 4 rocksdb:
Options.allow_concurrent_memtable_write: 1
2019-01-08 03:07:39.646723 7fdf8eb14700 4 rocksdb:
Options.enable_write_thread_adaptive_yield: 1
2019-01-08 03:07:39.646723 7fdf8eb14700 4 rocksdb:
Options.write_thread_max_yield_usec: 100
2019-01-08 03:07:39.646724 7fdf8eb14700 4 rocksdb:
Options.write_thread_slow_yield_usec: 3
2019-01-08 03:07:39.646724 7fdf8eb14700 4 rocksdb:
Options.row_cache: None
2019-01-08 03:07:39.646725 7fdf8eb14700 4 rocksdb:
Options.wal_filter: None
2019-01-08 03:07:39.646726 7fdf8eb14700 4 rocksdb:
Options.avoid_flush_during_recovery: 0
2019-01-08 03:07:39.646726 7fdf8eb14700 4 rocksdb:
Options.base_background_compactions: 1
2019-01-08 03:07:39.646727 7fdf8eb14700 4 rocksdb:
Options.max_background_compactions: 1
2019-01-08 03:07:39.646728 7fdf8eb14700 4 rocksdb:
Options.avoid_flush_during_shutdown: 0
2019-01-08 03:07:39.646728 7fdf8eb14700 4 rocksdb:
Options.delayed_write_rate : 16777216
2019-01-08 03:07:39.646729 7fdf8eb14700 4 rocksdb:
Options.max_total_wal_size: 0
2019-01-08 03:07:39.646729 7fdf8eb14700 4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
2019-01-08 03:07:39.646730 7fdf8eb14700 4 rocksdb:
Options.stats_dump_period_sec: 600
2019-01-08 03:07:39.646731 7fdf8eb14700 4 rocksdb: Compression
algorithms supported:
2019-01-08 03:07:39.646732 7fdf8eb14700 4 rocksdb: Snappy supported: 1
2019-01-08 03:07:39.646732 7fdf8eb14700 4 rocksdb: Zlib supported: 0
2019-01-08 03:07:39.646733 7fdf8eb14700 4 rocksdb: Bzip supported: 0
2019-01-08 03:07:39.646733 7fdf8eb14700 4 rocksdb: LZ4 supported: 0
2019-01-08 03:07:39.646734 7fdf8eb14700 4 rocksdb: ZSTD supported: 0
2019-01-08 03:07:39.646735 7fdf8eb14700 4 rocksdb: Fast CRC32 supported: 1
2019-01-08 03:07:39.649517 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/version_set.cc:2609]
Recovering from manifest file: MANIFEST-000001
2019-01-08 03:07:39.649621 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/column_family.cc:407]
--------------- Options for column family [default]:
2019-01-08 03:07:39.649626 7fdf8eb14700 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
2019-01-08 03:07:39.649627 7fdf8eb14700 4 rocksdb:
Options.merge_operator:
2019-01-08 03:07:39.649628 7fdf8eb14700 4 rocksdb:
Options.compaction_filter: None
2019-01-08 03:07:39.649629 7fdf8eb14700 4 rocksdb:
Options.compaction_filter_factory: None
2019-01-08 03:07:39.649629 7fdf8eb14700 4 rocksdb:
Options.memtable_factory: SkipListFactory
2019-01-08 03:07:39.649630 7fdf8eb14700 4 rocksdb:
Options.table_factory: BlockBasedTable
2019-01-08 03:07:39.649644 7fdf8eb14700 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x55f07246e1b0)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x55f072880390
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
2019-01-08 03:07:39.649646 7fdf8eb14700 4 rocksdb:
Options.write_buffer_size: 33554432
2019-01-08 03:07:39.649647 7fdf8eb14700 4 rocksdb:
Options.max_write_buffer_number: 2
2019-01-08 03:07:39.649648 7fdf8eb14700 4 rocksdb:
Options.compression: NoCompression
2019-01-08 03:07:39.649649 7fdf8eb14700 4 rocksdb:
Options.bottommost_compression: Disabled
2019-01-08 03:07:39.649649 7fdf8eb14700 4 rocksdb:
Options.prefix_extractor: nullptr
2019-01-08 03:07:39.649650 7fdf8eb14700 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2019-01-08 03:07:39.649650 7fdf8eb14700 4 rocksdb:
Options.num_levels: 7
2019-01-08 03:07:39.649651 7fdf8eb14700 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
2019-01-08 03:07:39.649651 7fdf8eb14700 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
2019-01-08 03:07:39.649652 7fdf8eb14700 4 rocksdb:
Options.compression_opts.window_bits: -14
2019-01-08 03:07:39.649653 7fdf8eb14700 4 rocksdb:
Options.compression_opts.level: -1
2019-01-08 03:07:39.649653 7fdf8eb14700 4 rocksdb:
Options.compression_opts.strategy: 0
2019-01-08 03:07:39.649654 7fdf8eb14700 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
2019-01-08 03:07:39.649655 7fdf8eb14700 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
2019-01-08 03:07:39.649656 7fdf8eb14700 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
2019-01-08 03:07:39.649656 7fdf8eb14700 4 rocksdb:
Options.level0_stop_writes_trigger: 36
2019-01-08 03:07:39.649657 7fdf8eb14700 4 rocksdb:
Options.target_file_size_base: 67108864
2019-01-08 03:07:39.649658 7fdf8eb14700 4 rocksdb:
Options.target_file_size_multiplier: 1
2019-01-08 03:07:39.649658 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
2019-01-08 03:07:39.649659 7fdf8eb14700 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 1
2019-01-08 03:07:39.649659 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
2019-01-08 03:07:39.649669 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2019-01-08 03:07:39.649670 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2019-01-08 03:07:39.649670 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2019-01-08 03:07:39.649676 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2019-01-08 03:07:39.649677 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2019-01-08 03:07:39.649678 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2019-01-08 03:07:39.649678 7fdf8eb14700 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2019-01-08 03:07:39.649679 7fdf8eb14700 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
2019-01-08 03:07:39.649679 7fdf8eb14700 4 rocksdb:
Options.max_compaction_bytes: 1677721600
2019-01-08 03:07:39.649680 7fdf8eb14700 4 rocksdb:
Options.arena_block_size: 4194304
2019-01-08 03:07:39.649680 7fdf8eb14700 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
2019-01-08 03:07:39.649681 7fdf8eb14700 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
2019-01-08 03:07:39.649681 7fdf8eb14700 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
2019-01-08 03:07:39.649682 7fdf8eb14700 4 rocksdb:
Options.disable_auto_compactions: 0
2019-01-08 03:07:39.649683 7fdf8eb14700 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
2019-01-08 03:07:39.649684 7fdf8eb14700 4 rocksdb:
Options.compaction_pri: kByCompensatedSize
2019-01-08 03:07:39.649685 7fdf8eb14700 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
2019-01-08 03:07:39.649685 7fdf8eb14700 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2019-01-08 03:07:39.649686 7fdf8eb14700 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2019-01-08 03:07:39.649686 7fdf8eb14700 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent:
200
2019-01-08 03:07:39.649687 7fdf8eb14700 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2019-01-08 03:07:39.649687 7fdf8eb14700 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2019-01-08 03:07:39.649688 7fdf8eb14700 4 rocksdb:
Options.table_properties_collectors:
2019-01-08 03:07:39.649689 7fdf8eb14700 4 rocksdb:
Options.inplace_update_support: 0
2019-01-08 03:07:39.649689 7fdf8eb14700 4 rocksdb:
Options.inplace_update_num_locks: 10000
2019-01-08 03:07:39.649690 7fdf8eb14700 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
2019-01-08 03:07:39.649697 7fdf8eb14700 4 rocksdb:
Options.memtable_huge_page_size: 0
2019-01-08 03:07:39.649698 7fdf8eb14700 4 rocksdb:
Options.bloom_locality: 0
2019-01-08 03:07:39.649698 7fdf8eb14700 4 rocksdb:
Options.max_successive_merges: 0
2019-01-08 03:07:39.649699 7fdf8eb14700 4 rocksdb:
Options.optimize_filters_for_hits: 0
2019-01-08 03:07:39.649699 7fdf8eb14700 4 rocksdb:
Options.paranoid_file_checks: 0
2019-01-08 03:07:39.649700 7fdf8eb14700 4 rocksdb:
Options.force_consistency_checks: 0
2019-01-08 03:07:39.649700 7fdf8eb14700 4 rocksdb:
Options.report_bg_io_stats: 0
2019-01-08 03:07:39.651643 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/version_set.cc:2859]
Recovered from manifest file:/tmp/monstore/store.db/MANIFEST-000001
succeeded,manifest_file_number is 1, next_file_number is 3,
last_sequence is 0, log_number is 0,prev_log_number is
0,max_column_family is 0
2019-01-08 03:07:39.651652 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/version_set.cc:2867]
Column family [default] (ID 0), log number is 0
2019-01-08 03:07:39.651699 7fdf8eb14700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1546913259651695, "job": 1, "event":
"recovery_started", "log_files": [3]}
2019-01-08 03:07:39.651704 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/db_impl_open.cc:482]
Recovering log #3 mode 2
2019-01-08 03:07:39.663188 7fdf8eb14700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1546913259663180, "cf_name": "default", "job": 1,
"event": "table_file_creation", "file_number": 4, "file_size": 1215,
"table_properties": {"data_size": 258, "index_size": 28,
"filter_size": 20, "raw_key_size": 99, "raw_average_key_size": 24,
"raw_value_size": 144, "raw_average_value_size": 36,
"num_data_blocks": 1, "num_entries": 4, "filter_policy_name":
"rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands":
"0"}}
2019-01-08 03:07:39.663212 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/version_set.cc:2395]
Creating manifest 5
2019-01-08 03:07:39.671222 7fdf8eb14700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1546913259671219, "job": 1, "event":
"recovery_finished"}
2019-01-08 03:07:39.679958 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/db_impl_open.cc:1063]
DB pointer 0x55f0727e0000
mismatched full crc: 3704629568 != 33348385 ---------》ERROR!!!!
2019-01-08 03:07:39.687414 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/db_impl.cc:217]
Shutdown: canceling all background work
2019-01-08 03:07:39.687538 7fdf8eb14700 4 rocksdb:
[/home/builder/source/ceph-12.2.10/src/rocksdb/db/db_impl.cc:343]
Shutdown complete
ceph-osd -i 0
ceph version 12.2.10 (fc2b1783e3727b66315cc667af9d663d30fe7ed4)
luminous (stable)
1: (()+0xa56bd4) [0x556337cefbd4]
2: (()+0x110c0) [0x7fa38f9300c0]
3: (gsignal()+0xcf) [0x7fa38e8f7fff]
4: (abort()+0x16a) [0x7fa38e8f942a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7fa38f2100ad]
6: (()+0x8f066) [0x7fa38f20e066]
7: (()+0x8f0b1) [0x7fa38f20e0b1]
8: (()+0x8f2c9) [0x7fa38f20e2c9]
9: (object_stat_sum_t::decode(ceph::buffer::list::iterator&)+0x5c3)
[0x55633796e003]
10: (object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x67)
[0x556337987197]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x1eb) [0x55633798810b]
12: (pg_info_t::decode(ceph::buffer::list::iterator&)+0x147) [0x556337991357]
13: (PG::read_info(ObjectStore*, spg_t, coll_t const&,
ceph::buffer::list&, pg_info_t&, PastIntervals&, unsigned
char&)+0x215) [0x5563377cd395]
14: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x6d) [0x5563377d0b8d]
15: (OSD::load_pgs()+0x8b8) [0x55633771bfc8]
16: (OSD::init()+0x2237) [0x55633773ce37]
17: (main()+0x3092) [0x5563376443c2]
18: (__libc_start_main()+0xf1) [0x7fa38e8e52e1]
19: (_start()+0x2a) [0x5563376d0f9a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Any advice is appreciated.
