Experts :
Sorry to bother you , I can't start a new thread from website , have to seek your help with email .
We met one question :
Cell version : 13.2.5-68-g53a67b5 (53a67b54fa57bcdfe2ce45c460e06adf62fe2eb2) mimic (stable)
Problem : MON can't startup , the cluster down .
Details description : the cell version was upgraded on the end of Mar . And system run well even we did reboot test after upgrade . But we found system crash after we reboot one node after HDD replacement . It seems that mon can't start with below error mesg :
2021-11-23 14:44:50.832 7fa548806700 -1 mon.host15@4(synchronizing).mgr e5166 Failed to load mgr commands: (2) No such file or directory
2021-11-23 14:44:50.833 7fa548806700 -1 mon.host15@4(probing) e1 handle_probe_reply existing cluster has not completed a full luminous scrub to purge legacy snapdir objects; please scrub before upgrading beyond luminous.
We did some investigation from google , and found that it may be caused by upgrade without deep scrub . We are not sure whether it is right , and how to fix that if the reason is that .
Your kindly help is much appreciated !
Best Regar
-yanyong
2021-11-23 14:44:38.324 7fa560ad6a00 0 set uid:gid to 167:167 (ceph:ceph)
2021-11-23 14:44:38.324 7fa560ad6a00 0 ceph version 13.2.5-68-g53a67b5 (53a67b54fa57bcdfe2ce45c460e06adf62fe2eb2) mimic (stable), process ceph-mon, pid 2862392
2021-11-23 14:44:38.330 7fa560ad6a00 0 pidfile_write: ignore empty --pid-file
2021-11-23 14:44:38.414 7fa560ad6a00 0 load: jerasure load: lrc load: isa
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option compression = kNoCompression
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option level_compaction_dynamic_level_bytes = true
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option write_buffer_size = 33554432
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option compression = kNoCompression
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option level_compaction_dynamic_level_bytes = true
2021-11-23 14:44:38.427 7fa560ad6a00 0 set rocksdb option write_buffer_size = 33554432
2021-11-23 14:44:38.435 7fa560ad6a00 1 rocksdb: do_open column families: [default]
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: RocksDB version: 5.13.0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Compile date Nov 24 2020
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: DB SUMMARY
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: CURRENT file: CURRENT
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: IDENTITY file: IDENTITY
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: MANIFEST file: MANIFEST-5296634 size: 4904 Bytes
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: SST files in /var/lib/ceph/mon/ucsm-host15/store.db dir, Total Num: 15, files: 5296560.sst 5296615.sst 5296616.sst 5296628.sst 5296629.sst 5296662.sst 5296663.sst 5296665.sst 5296666.sst
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ucsm-host15/store.db: 5296682.log size: 2022147 ; 5296684.log size: 0 ;
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.error_if_exists: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.create_if_missing: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.paranoid_checks: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.env: 0x5564533f78c0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.info_log: 0x556454288260
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_file_opening_threads: 16
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.statistics: (nil)
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.use_fsync: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_log_file_size: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.log_file_time_to_roll: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.keep_log_file_num: 1000
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.recycle_log_file_num: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.allow_fallocate: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.allow_mmap_reads: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.allow_mmap_writes: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.use_direct_reads: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.create_missing_column_families: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.db_log_dir:
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ucsm-host15/store.db
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.table_cache_numshardbits: 6
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_subcompactions: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_background_flushes: -1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.WAL_ttl_seconds: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.WAL_size_limit_MB: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.manifest_preallocation_size: 4194304
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.is_fd_close_on_exec: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.advise_random_on_open: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.db_write_buffer_size: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.write_buffer_manager: 0x556454232c90
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.access_hint_on_compaction_start: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.use_adaptive_mutex: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.rate_limiter: (nil)
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.wal_recovery_mode: 2
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.enable_thread_tracking: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.enable_pipelined_write: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.write_thread_max_yield_usec: 100
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.row_cache: None
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.wal_filter: None
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.avoid_flush_during_recovery: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.allow_ingest_behind: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.preserve_deletes: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.two_write_queues: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.manual_wal_flush: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_background_jobs: 2
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_background_compactions: -1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.delayed_write_rate : 16777216
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_total_wal_size: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.stats_dump_period_sec: 600
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.max_open_files: -1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.bytes_per_sync: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.wal_bytes_per_sync: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Options.compaction_readahead_size: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Compression algorithms supported:
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kZSTDNotFinalCompression supported: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kZSTD supported: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kXpressCompression supported: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kLZ4HCCompression supported: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kLZ4Compression supported: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kBZip2Compression supported: 0
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kZlibCompression supported: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: kSnappyCompression supported: 1
2021-11-23 14:44:38.435 7fa560ad6a00 4 rocksdb: Fast CRC32 supported: Supported on x86
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-5296634
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/column_family.cc:474] --------------- Options for column family [default]:
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.merge_operator:
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_filter: None
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_filter_factory: None
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.memtable_factory: SkipListFactory
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.table_factory: BlockBasedTable
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556453ea9450)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x556453f03dd0
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.write_buffer_size: 33554432
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_write_buffer_number: 2
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression: NoCompression
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.bottommost_compression: Disabled
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.prefix_extractor: nullptr
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.num_levels: 7
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression_opts.window_bits: -14
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression_opts.level: -1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression_opts.strategy: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.level0_stop_writes_trigger: 36
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.target_file_size_base: 67108864
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.target_file_size_multiplier: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_compaction_bytes: 1677721600
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.arena_block_size: 4194304
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.disable_auto_compactions: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.compaction_options_fifo.ttl: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.table_properties_collectors:
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.inplace_update_support: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.inplace_update_num_locks: 10000
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.memtable_huge_page_size: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.bloom_locality: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.max_successive_merges: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.optimize_filters_for_hits: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.paranoid_file_checks: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.force_consistency_checks: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.report_bg_io_stats: 0
2021-11-23 14:44:38.436 7fa560ad6a00 4 rocksdb: Options.ttl: 0
2021-11-23 14:44:38.484 7fa560ad6a00 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:38.484 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:3362] Recovered from manifest file:/var/lib/ceph/mon/ucsm-host15/store.db/MANIFEST-5296634 succeeded,manifest_file_number is 5296634, next_file_number is 5296685, last_sequence is 1315351375, log_number is 0,prev_log_number is 0,max_column_family is 0,deleted_log_number is 5296680
2021-11-23 14:44:38.484 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:3370] Column family [default] (ID 0), log number is 5296682
2021-11-23 14:44:38.484 7fa560ad6a00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649878485665, "job": 1, "event": "recovery_started", "log_files": [5296682, 5296684]}
2021-11-23 14:44:38.484 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_open.cc:551] Recovering log #5296682 mode 2
2021-11-23 14:44:38.510 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_open.cc:551] Recovering log #5296684 mode 2
2021-11-23 14:44:38.525 7fa560ad6a00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649878526341, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 5296685, "file_size": 1880154, "table_properties": {"data_size": 1870303, "index_size": 3624, "filter_size": 5244, "raw_key_size": 9115, "raw_average_key_size": 23, "raw_value_size": 1862236, "raw_average_value_size": 4874, "num_data_blocks": 103, "num_entries": 382, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "254", "kMergeOperands": "0"}}
2021-11-23 14:44:38.525 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2863] Creating manifest 5296686
2021-11-23 14:44:38.525 7fa560ad6a00 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:38.526 7fa560ad6a00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649878527977, "job": 1, "event": "recovery_finished"}
2021-11-23 14:44:38.536 7fa560ad6a00 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:38.537 7fa560ad6a00 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:38.546 7fa560ad6a00 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_open.cc:1218] DB pointer 0x556454340000
2021-11-23 14:44:38.546 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl.cc:490] ------- DUMPING STATS -------
2021-11-23 14:44:38.546 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl.cc:491]
** DB Stats **
Uptime(secs): 0.1 total, 0.1 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 4/0 94.66 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 122.6 0 1 0.015 0 0
L2 4/0 253.60 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L3 4/0 245.54 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L4 4/0 134.69 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 16/0 728.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 122.6 0 1 0.015 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 122.6 0 1 0.015 0 0
Uptime(secs): 0.1 total, 0.1 interval
Flush(GB): cumulative 0.002, interval 0.002
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 16.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 16.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 4/0 94.66 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 122.6 0 1 0.015 0 0
L2 4/0 253.60 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L3 4/0 245.54 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
L4 4/0 134.69 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 16/0 728.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 122.6 0 1 0.015 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Uptime(secs): 0.1 total, 0.0 interval
Flush(GB): cumulative 0.002, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 16.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
2021-11-23 14:44:38.546 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 3] Compacting 4@0 + 4@2 files to L2, score 1.00
2021-11-23 14:44:38.546 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 4 Base level 0, inputs: [5296685(1836KB) 5296683(30MB) 5296681(31MB) 5296679(31MB)], [5296662(64MB) 5296663(64MB) 5296676(64MB) 5296677(60MB)]
2021-11-23 14:44:38.546 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649878548058, "job": 3, "event": "compaction_started", "files_L0": [5296685, 5296683, 5296681, 5296679], "files_L2": [5296662, 5296663, 5296676, 5296677], "score": 1, "input_data_size": 365176059}
2021-11-23 14:44:39.126 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296690: 2333 keys, 67389858 bytes
2021-11-23 14:44:39.126 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649879127868, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296690, "file_size": 67389858, "table_properties": {"data_size": 67223249, "index_size": 25033, "filter_size": 140584, "raw_key_size": 51907, "raw_average_key_size": 22, "raw_value_size": 67172592, "raw_average_value_size": 28792, "num_data_blocks": 664, "num_entries": 2333, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "254", "kMergeOperands": "0"}}
2021-11-23 14:44:39.175 7fa560ad6a00 0 starting mon.host15 rank 4 at public addr 10.1.222.15:6789/0 at bind addr 10.1.222.15:6789/0 mon_data /var/lib/ceph/mon/ucsm-host15 fsid b3cf36fc-733c-4129-a812-15c115166f96
2021-11-23 14:44:39.175 7fa560ad6a00 0 starting mon.host15 rank 4 at 10.1.222.15:6789/0 mon_data /var/lib/ceph/mon/ucsm-host15 fsid b3cf36fc-733c-4129-a812-15c115166f96
2021-11-23 14:44:39.729 7fa560ad6a00 0 mon.host15@-1(probing).mds e1056 new map
2021-11-23 14:44:39.733 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296691: 729 keys, 67370543 bytes
2021-11-23 14:44:39.733 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649879735087, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296691, "file_size": 67370543, "table_properties": {"data_size": 67202687, "index_size": 29047, "filter_size": 137818, "raw_key_size": 18954, "raw_average_key_size": 26, "raw_value_size": 67170611, "raw_average_value_size": 92140, "num_data_blocks": 729, "num_entries": 729, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:39.729 7fa560ad6a00 0 mon.host15@-1(probing).mds e1056 print_map
e1056
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 1045
flags c
created 2019-03-08 18:38:50.945451
modified 2021-03-12 10:58:03.138480
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 112760
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=3282386930}
failed
damaged
stopped
data_pools [2]
metadata_pool 3
inline_data disabled
balancer
standby_count_wanted 1
3282386930: 10.1.222.13:6800/2879194895 'host13' mds.0.1042 up:active seq 9
Standby daemons:
3683837653: 10.1.222.15:6800/3608600498 'host15' mds.-1.0 up:standby seq 2
2021-11-23 14:44:40.328 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296692: 671 keys, 67443188 bytes
2021-11-23 14:44:40.328 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649880329959, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296692, "file_size": 67443188, "table_properties": {"data_size": 67278024, "index_size": 26729, "filter_size": 137444, "raw_key_size": 17446, "raw_average_key_size": 26, "raw_value_size": 67248500, "raw_average_value_size": 100221, "num_data_blocks": 671, "num_entries": 671, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:40.362 7fa560ad6a00 0 mon.host15@-1(probing).osd e114412 crush map has features 2303210029056, adjusting msgr requires
2021-11-23 14:44:40.362 7fa560ad6a00 0 mon.host15@-1(probing).osd e114412 crush map has features 2303210029056, adjusting msgr requires
2021-11-23 14:44:40.362 7fa560ad6a00 0 mon.host15@-1(probing).osd e114412 crush map has features 720578243589324800, adjusting msgr requires
2021-11-23 14:44:40.362 7fa560ad6a00 0 mon.host15@-1(probing).osd e114412 crush map has features 2303210029056, adjusting msgr requires
2021-11-23 14:44:40.985 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296693: 684 keys, 67414609 bytes
2021-11-23 14:44:40.985 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649880986487, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296693, "file_size": 67414609, "table_properties": {"data_size": 67248860, "index_size": 27253, "filter_size": 137505, "raw_key_size": 17784, "raw_average_key_size": 26, "raw_value_size": 67218764, "raw_average_value_size": 98273, "num_data_blocks": 684, "num_entries": 684, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:41.153 7fa560ad6a00 -1 mon.host15@-1(probing).mgr e5123 Failed to load mgr commands: (2) No such file or directory
2021-11-23 14:44:41.162 7fa560ad6a00 0 mon.host15@-1(probing) e1 my rank is now 4 (was -1)
2021-11-23 14:44:41.168 7fa544fff700 0 -- 10.1.222.15:6789/0 >> 10.1.222.17:6789/0 conn(0x556454dc0a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=234401710 cs=1 l=0).process missed message? skipped from seq 0 to 1861684816
2021-11-23 14:44:41.169 7fa546001700 0 -- 10.1.222.15:6789/0 >> 10.1.222.16:6789/0 conn(0x556454dc0400 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=234202689 cs=1 l=0).process missed message? skipped from seq 0 to 1863016889
2021-11-23 14:44:41.194 7fa545800700 0 -- 10.1.222.15:6789/0 >> 10.1.222.12:6789/0 conn(0x556454dbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=234548774 cs=1 l=0).process missed message? skipped from seq 0 to 240757177
2021-11-23 14:44:41.258 7fa544fff700 0 -- 10.1.222.15:6789/0 >> 10.1.222.14:6789/0 conn(0x556454286a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=234487345 cs=1 l=0).process missed message? skipped from seq 0 to 1594724737
2021-11-23 14:44:41.640 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296694: 837 keys, 67302702 bytes
2021-11-23 14:44:41.640 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649881641551, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296694, "file_size": 67302702, "table_properties": {"data_size": 67131060, "index_size": 32019, "filter_size": 138632, "raw_key_size": 20718, "raw_average_key_size": 24, "raw_value_size": 67095519, "raw_average_value_size": 80161, "num_data_blocks": 835, "num_entries": 837, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:42.149 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 3] Generated table #5296695: 351 keys, 28253327 bytes
2021-11-23 14:44:42.149 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649882151210, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 5296695, "file_size": 28253327, "table_properties": {"data_size": 28181842, "index_size": 12358, "filter_size": 58138, "raw_key_size": 8091, "raw_average_key_size": 23, "raw_value_size": 28167811, "raw_average_value_size": 80250, "num_data_blocks": 341, "num_entries": 351, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:42.150 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 3] Compacted 4@0 + 4@2 files to L2 => 365174227 bytes
2021-11-23 14:44:42.150 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:42.152644) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 2 max bytes base 268435456 files[0 0 6 4 4 0 0] max score 1.36, MB/sec: 101.3 rd, 101.3 wr, level 2, files in(4, 4) out(6) MB in(94.7, 253.6) out(348.3), read-write-amplify(7.4) write-amplify(3.7) OK, records in: 5605, records dropped: 0 output_compression: NoCompression
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:42.152664) EVENT_LOG_v1 {"time_micros": 1637649882152655, "job": 3, "event": "compaction_finished", "compaction_time_micros": 3603807, "output_level": 2, "num_output_files": 6, "total_output_size": 365174227, "num_input_records": 5605, "num_output_records": 5605, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 6, 4, 4, 0, 0]}
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_picker.cc:483] [default] Expanding@2 1+1(67443188+54955071 bytes) to 4+1 (230413826+54955071bytes)
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 4] Compacting 4@2 + 1@3 files to L3, score 1.36
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 5 Base level 2, inputs: [5296692(64MB) 5296693(64MB) 5296694(64MB) 5296695(26MB)], [5296665(52MB)]
2021-11-23 14:44:42.151 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649882152752, "job": 4, "event": "compaction_started", "files_L2": [5296692, 5296693, 5296694, 5296695], "files_L3": [5296665], "score": 1.36038, "input_data_size": 285368897}
2021-11-23 14:44:42.874 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 4] Generated table #5296696: 722 keys, 67452555 bytes
2021-11-23 14:44:42.874 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649882875703, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 5296696, "file_size": 67452555, "table_properties": {"data_size": 67284883, "index_size": 28766, "filter_size": 137915, "raw_key_size": 18772, "raw_average_key_size": 26, "raw_value_size": 67253115, "raw_average_value_size": 93148, "num_data_blocks": 722, "num_entries": 722, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:43.291 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 4] Generated table #5296697: 671 keys, 67459376 bytes
2021-11-23 14:44:43.291 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649883292775, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 5296697, "file_size": 67459376, "table_properties": {"data_size": 67294175, "index_size": 26734, "filter_size": 137476, "raw_key_size": 17446, "raw_average_key_size": 26, "raw_value_size": 67264651, "raw_average_value_size": 100245, "num_data_blocks": 671, "num_entries": 671, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:43.653 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 4] Generated table #5296698: 683 keys, 67451267 bytes
2021-11-23 14:44:43.653 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649883654310, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 5296698, "file_size": 67451267, "table_properties": {"data_size": 67285497, "index_size": 27211, "filter_size": 137568, "raw_key_size": 17758, "raw_average_key_size": 26, "raw_value_size": 67255445, "raw_average_value_size": 98470, "num_data_blocks": 683, "num_entries": 683, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:44.057 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 4] Generated table #5296699: 853 keys, 67452232 bytes
2021-11-23 14:44:44.057 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884059107, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 5296699, "file_size": 67452232, "table_properties": {"data_size": 67280022, "index_size": 32151, "filter_size": 139068, "raw_key_size": 20687, "raw_average_key_size": 24, "raw_value_size": 67244284, "raw_average_value_size": 78832, "num_data_blocks": 851, "num_entries": 853, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:44.171 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 4] Generated table #5296700: 216 keys, 15549111 bytes
2021-11-23 14:44:44.171 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884172704, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 5296700, "file_size": 15549111, "table_properties": {"data_size": 15508484, "index_size": 7467, "filter_size": 32171, "raw_key_size": 4986, "raw_average_key_size": 23, "raw_value_size": 15499937, "raw_average_value_size": 71758, "num_data_blocks": 206, "num_entries": 216, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:44.172 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 4] Compacted 4@2 + 1@3 files to L3 => 285364541 bytes
2021-11-23 14:44:44.172 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:44.173 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.174586) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 2 max bytes base 268435456 files[0 0 2 8 4 0 0] max score 1.82, MB/sec: 141.2 rd, 141.2 wr, level 3, files in(4, 1) out(5) MB in(219.7, 52.4) out(272.1), read-write-amplify(2.5) write-amplify(1.2) OK, records in: 3412, records dropped: 267 output_compression: NoCompression
2021-11-23 14:44:44.173 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.174606) EVENT_LOG_v1 {"time_micros": 1637649884174597, "job": 4, "event": "compaction_finished", "compaction_time_micros": 2020743, "output_level": 3, "num_output_files": 5, "total_output_size": 285364541, "num_input_records": 3412, "num_output_records": 3145, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 2, 8, 4, 0, 0]}
2021-11-23 14:44:44.179 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884180954, "job": 4, "event": "table_file_deletion", "file_number": 5296695}
2021-11-23 14:44:44.194 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884195622, "job": 4, "event": "table_file_deletion", "file_number": 5296694}
2021-11-23 14:44:44.208 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884209316, "job": 4, "event": "table_file_deletion", "file_number": 5296693}
2021-11-23 14:44:44.221 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884222639, "job": 4, "event": "table_file_deletion", "file_number": 5296692}
2021-11-23 14:44:44.221 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.222685) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:1807] [default] Moving #5296666 to level-4 67611962 bytes
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.223697) EVENT_LOG_v1 {"time_micros": 1637649884223692, "job": 5, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 67611962}
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.223706) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:1836] [default] Moved #1 files to level-4 67611962 bytes OK: base level 2 max bytes base 268435456 files[0 0 2 7 5 0 0] max score 1.57
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 6] Compacting 1@3 + 1@4 files to L4, score 1.57
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 7 Base level 3, inputs: [5296667(64MB)], [5296615(64MB)]
2021-11-23 14:44:44.222 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884223758, "job": 6, "event": "compaction_started", "files_L3": [5296667], "files_L4": [5296615], "score": 1.56562, "input_data_size": 135001369}
2021-11-23 14:44:44.591 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 6] Generated table #5296701: 1136 keys, 67459541 bytes
2021-11-23 14:44:44.591 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884592988, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 5296701, "file_size": 67459541, "table_properties": {"data_size": 67308021, "index_size": 14868, "filter_size": 135662, "raw_key_size": 23856, "raw_average_key_size": 21, "raw_value_size": 67282127, "raw_average_value_size": 59227, "num_data_blocks": 432, "num_entries": 1136, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:44.623 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 6] Generated table #5296702: 668 keys, 5650048 bytes
2021-11-23 14:44:44.623 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884624718, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 5296702, "file_size": 5650048, "table_properties": {"data_size": 5633955, "index_size": 2270, "filter_size": 12837, "raw_key_size": 14028, "raw_average_key_size": 21, "raw_value_size": 5623211, "raw_average_value_size": 8417, "num_data_blocks": 67, "num_entries": 668, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:44.642 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 6] Compacted 1@3 + 1@4 files to L4 => 73109589 bytes
2021-11-23 14:44:44.642 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:44.647 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.649165) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 2 max bytes base 268435456 files[0 0 2 6 6 0 0] max score 1.31, MB/sec: 321.8 rd, 174.3 wr, level 4, files in(1, 1) out(2) MB in(64.3, 64.4) out(69.7), read-write-amplify(3.1) write-amplify(1.1) OK, records in: 2854, records dropped: 1050 output_compression: NoCompression
2021-11-23 14:44:44.647 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:44.649190) EVENT_LOG_v1 {"time_micros": 1637649884649180, "job": 6, "event": "compaction_finished", "compaction_time_micros": 419514, "output_level": 4, "num_output_files": 2, "total_output_size": 73109589, "num_input_records": 2854, "num_output_records": 1804, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 2, 6, 6, 0, 0]}
2021-11-23 14:44:44.648 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_picker.cc:483] [default] Expanding@3 1+2(67459376+71933446 bytes) to 2+2 (134911931+71933446bytes)
2021-11-23 14:44:44.648 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 7] Compacting 2@3 + 2@4 files to L4, score 1.31
2021-11-23 14:44:44.648 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 8 Base level 3, inputs: [5296696(64MB) 5296697(64MB)], [5296628(64MB) 5296629(4365KB)]
2021-11-23 14:44:44.648 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649884649278, "job": 7, "event": "compaction_started", "files_L3": [5296696, 5296697], "files_L4": [5296628, 5296629], "score": 1.31432, "input_data_size": 206845377}
2021-11-23 14:44:45.001 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 7] Generated table #5296703: 722 keys, 67452555 bytes
2021-11-23 14:44:45.001 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885002301, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 5296703, "file_size": 67452555, "table_properties": {"data_size": 67284883, "index_size": 28766, "filter_size": 137915, "raw_key_size": 18772, "raw_average_key_size": 26, "raw_value_size": 67253115, "raw_average_value_size": 93148, "num_data_blocks": 722, "num_entries": 722, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:45.165 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885166708, "job": 0, "event": "table_file_deletion", "file_number": 5296685}
2021-11-23 14:44:45.169 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885171220, "job": 0, "event": "table_file_deletion", "file_number": 5296683}
2021-11-23 14:44:45.411 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885412863, "job": 0, "event": "table_file_deletion", "file_number": 5296681}
2021-11-23 14:44:45.416 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 7] Generated table #5296704: 671 keys, 67459376 bytes
2021-11-23 14:44:45.416 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885417569, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 5296704, "file_size": 67459376, "table_properties": {"data_size": 67294175, "index_size": 26734, "filter_size": 137476, "raw_key_size": 17446, "raw_average_key_size": 26, "raw_value_size": 67264651, "raw_average_value_size": 100245, "num_data_blocks": 671, "num_entries": 671, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:45.419 7fa54d00f700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 7] Compacted 2@3 + 2@4 files to L4 => 134911931 bytes
2021-11-23 14:44:45.419 7fa54d00f700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:45.425 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:45.426659) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 2 max bytes base 268435456 files[0 0 2 4 6 0 0] max score 0.81, MB/sec: 268.2 rd, 174.9 wr, level 4, files in(2, 2) out(2) MB in(128.7, 68.6) out(128.7), read-write-amplify(2.5) write-amplify(1.0) OK, records in: 2150, records dropped: 757 output_compression: NoCompression
2021-11-23 14:44:45.425 7fa54d00f700 4 rocksdb: (Original Log Time 2021/11/23-14:44:45.426681) EVENT_LOG_v1 {"time_micros": 1637649885426671, "job": 7, "event": "compaction_finished", "compaction_time_micros": 771234, "output_level": 4, "num_output_files": 2, "total_output_size": 134911931, "num_input_records": 2150, "num_output_records": 1393, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 2, 4, 6, 0, 0]}
2021-11-23 14:44:45.428 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885429692, "job": 0, "event": "table_file_deletion", "file_number": 5296679}
2021-11-23 14:44:45.436 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885438205, "job": 0, "event": "table_file_deletion", "file_number": 5296677}
2021-11-23 14:44:45.438 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885439618, "job": 7, "event": "table_file_deletion", "file_number": 5296697}
2021-11-23 14:44:45.451 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885452444, "job": 0, "event": "table_file_deletion", "file_number": 5296676}
2021-11-23 14:44:45.451 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885453087, "job": 7, "event": "table_file_deletion", "file_number": 5296696}
2021-11-23 14:44:45.457 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885458881, "job": 7, "event": "table_file_deletion", "file_number": 5296629}
2021-11-23 14:44:45.470 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885472015, "job": 0, "event": "table_file_deletion", "file_number": 5296667}
2021-11-23 14:44:45.474 7fa54d00f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885475975, "job": 7, "event": "table_file_deletion", "file_number": 5296628}
2021-11-23 14:44:45.487 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885489091, "job": 0, "event": "table_file_deletion", "file_number": 5296665}
2021-11-23 14:44:45.499 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885500418, "job": 0, "event": "table_file_deletion", "file_number": 5296663}
2021-11-23 14:44:45.516 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885517570, "job": 0, "event": "table_file_deletion", "file_number": 5296662}
2021-11-23 14:44:45.526 7fa548806700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649885528055, "job": 0, "event": "table_file_deletion", "file_number": 5296615}
2021-11-23 14:44:49.055 7fa548806700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #5296705. Immutable memtables: 0.
2021-11-23 14:44:49.059 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.057196) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
2021-11-23 14:44:49.059 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:300] [default] [JOB 8] Flushing memtable with next log file: 5296705
2021-11-23 14:44:49.059 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649889060489, "job": 8, "event": "flush_started", "num_memtables": 1, "num_entries": 22687, "num_deletes": 15046, "memory_usage": 32815816, "flush_reason": "Write Buffer Full"}
2021-11-23 14:44:49.059 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:330] [default] [JOB 8] Level-0 flush table #5296706: started
2021-11-23 14:44:49.232 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649889233993, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 5296706, "file_size": 31705506, "table_properties": {"data_size": 31536661, "index_size": 57227, "filter_size": 110621, "raw_key_size": 353382, "raw_average_key_size": 20, "raw_value_size": 31272292, "raw_average_value_size": 1843, "num_data_blocks": 1659, "num_entries": 16961, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "9320", "kMergeOperands": "0"}}
2021-11-23 14:44:49.232 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:370] [default] [JOB 8] Level-0 flush table #5296706: 31705506 bytes OK
2021-11-23 14:44:49.233 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:49.234 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.234524) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:374] [default] Level-0 commit table #5296706 started
2021-11-23 14:44:49.234 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.235261) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #5296706: memtable #1 done
2021-11-23 14:44:49.234 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.235278) EVENT_LOG_v1 {"time_micros": 1637649889235269, "job": 8, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 2, 4, 6, 0, 0], "immutable_memtables": 0}
2021-11-23 14:44:49.234 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.235297) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 2 max bytes base 268435456 files[1 0 2 4 6 0 0] max score 0.81
2021-11-23 14:44:49.234 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_files.cc:461] [JOB 8] Try to delete WAL files size 31593920, prev total WAL file size 35051002, number of live WAL files 2.
2021-11-23 14:44:49.234 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:49.763 7fa548806700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #5296707. Immutable memtables: 0.
2021-11-23 14:44:49.767 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.764954) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
2021-11-23 14:44:49.767 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:300] [default] [JOB 9] Flushing memtable with next log file: 5296707
2021-11-23 14:44:49.767 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649889768666, "job": 9, "event": "flush_started", "num_memtables": 1, "num_entries": 446, "num_deletes": 0, "memory_usage": 33231688, "flush_reason": "Write Buffer Full"}
2021-11-23 14:44:49.767 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:330] [default] [JOB 9] Level-0 flush table #5296708: started
2021-11-23 14:44:49.956 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649889958065, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 5296708, "file_size": 32834583, "table_properties": {"data_size": 32757670, "index_size": 9504, "filter_size": 66416, "raw_key_size": 10214, "raw_average_key_size": 22, "raw_value_size": 32744520, "raw_average_value_size": 73418, "num_data_blocks": 250, "num_entries": 446, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:49.956 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:370] [default] [JOB 9] Level-0 flush table #5296708: 32834583 bytes OK
2021-11-23 14:44:49.957 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:49.958 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.958604) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:374] [default] Level-0 commit table #5296708 started
2021-11-23 14:44:49.958 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.959313) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #5296708: memtable #1 done
2021-11-23 14:44:49.958 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.959325) EVENT_LOG_v1 {"time_micros": 1637649889959319, "job": 9, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 2, 4, 6, 0, 0], "immutable_memtables": 0}
2021-11-23 14:44:49.958 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:49.959343) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 2 max bytes base 268435456 files[2 0 2 4 6 0 0] max score 0.81
2021-11-23 14:44:49.958 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_files.cc:461] [JOB 9] Try to delete WAL files size 32753597, prev total WAL file size 35793065, number of live WAL files 2.
2021-11-23 14:44:49.958 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:50.417 7fa548806700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #5296709. Immutable memtables: 0.
2021-11-23 14:44:50.421 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:50.418911) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
2021-11-23 14:44:50.421 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:300] [default] [JOB 10] Flushing memtable with next log file: 5296709
2021-11-23 14:44:50.421 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649890422524, "job": 10, "event": "flush_started", "num_memtables": 1, "num_entries": 435, "num_deletes": 0, "memory_usage": 32830552, "flush_reason": "Write Buffer Full"}
2021-11-23 14:44:50.421 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:330] [default] [JOB 10] Level-0 flush table #5296710: started
2021-11-23 14:44:50.590 7fa54c80e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637649890591601, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 5296710, "file_size": 32548317, "table_properties": {"data_size": 32463142, "index_size": 16875, "filter_size": 67306, "raw_key_size": 11016, "raw_average_key_size": 25, "raw_value_size": 32444396, "raw_average_value_size": 74584, "num_data_blocks": 433, "num_entries": 435, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2021-11-23 14:44:50.590 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/flush_job.cc:370] [default] [JOB 10] Level-0 flush table #5296710: 32548317 bytes OK
2021-11-23 14:44:50.590 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:50.591 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:50.592073) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:374] [default] Level-0 commit table #5296710 started
2021-11-23 14:44:50.591 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:50.592736) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #5296710: memtable #1 done
2021-11-23 14:44:50.591 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:50.592748) EVENT_LOG_v1 {"time_micros": 1637649890592742, "job": 10, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 0, 2, 4, 6, 0, 0], "immutable_memtables": 0}
2021-11-23 14:44:50.591 7fa54c80e700 4 rocksdb: (Original Log Time 2021/11/23-14:44:50.592766) [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 2 max bytes base 268435456 files[3 0 2 4 6 0 0] max score 0.82
2021-11-23 14:44:50.591 7fa54c80e700 4 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/db_impl_files.cc:461] [JOB 10] Try to delete WAL files size 32454424, prev total WAL file size 35137646, number of live WAL files 2.
2021-11-23 14:44:50.591 7fa54c80e700 3 rocksdb: [/root/rpmbuild/BUILD/ucsm-13.2.5-68-g53a67b5/src/rocksdb/db/version_set.cc:2501] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2021-11-23 14:44:50.825 7fa548806700 0 mon.host15@4(synchronizing).mds e1065 new map
2021-11-23 14:44:50.825 7fa548806700 0 mon.host15@4(synchronizing).mds e1065 print_map
e1065
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 1045
flags c
created 2019-03-08 18:38:50.945451
modified 2021-03-12 10:58:03.138480
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 112760
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=3282386930}
failed
damaged
stopped
data_pools [2]
metadata_pool 3
inline_data disabled
balancer
standby_count_wanted 1
3282386930: 10.1.222.13:6800/2879194895 'host13' mds.0.1042 up:active seq 9
Standby daemons:
3683837653: 10.1.222.15:6800/3608600498 'host15' mds.-1.0 up:standby seq 213084
3748578774: 10.1.222.18:6800/759636281 'host18' mds.-1.0 up:standby seq 2
2021-11-23 14:44:50.832 7fa548806700 -1 mon.host15@4(synchronizing).mgr e5166 Failed to load mgr commands: (2) No such file or directory
2021-11-23 14:44:50.833 7fa548806700 -1 mon.host15@4(probing) e1 handle_probe_reply existing cluster has not completed a full luminous scrub to purge legacy snapdir objects; please scrub before upgrading beyond luminous.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
