Here's a log dump from one that is marked as down on a node that seems to
no longer communicate with the rest of the cluster:
At the end it just keeps saying start_boot:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_compaction_bytes: 1677721600
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.arena_block_size: 33554432
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.disable_auto_compactions: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_pri: kMinOverlappingRatio
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.table_properties_collectors:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_support: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_num_locks: 10000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_whole_key_filtering: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_huge_page_size: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bloom_locality: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_successive_merges: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.optimize_filters_for_hits: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.paranoid_file_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.force_consistency_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.report_bg_io_stats: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.ttl: 2592000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.periodic_compaction_seconds: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
[column_family.cc:552] --------------- Options for column family [O-0]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.merge_operator: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter_factory: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_factory: SkipListFactory
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.table_factory: BlockBasedTable
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x560b4972f7c8)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks_with_high_priority: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_l0_filter_and_index_blocks_in_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_top_level_index_and_filter: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_shortening:
1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_hash_table_util_ratio: 0.750000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
hash_index_allow_collision: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: checksum: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: no_block_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache:
0x560b49774910
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache_name:
BinnedLRUCache
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_options:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: capacity :
536870912
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: num_shard_bits
: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
strict_capacity_limit : 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
high_pri_pool_ratio: 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_compressed: (nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: persistent_cache:
(nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_size_deviation: 10
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_restart_interval: 16
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
index_block_restart_interval: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
metadata_block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
partition_filters: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
use_delta_encoding: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: filter_policy:
rocksdb.BuiltinBloomFilter
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
whole_key_filtering: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
verify_compression: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
read_amp_bytes_per_bit: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: format_version: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
enable_index_compression: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_align: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.write_buffer_size: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression: NoCompression
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression: Disabled
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.num_levels: 7
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_size_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.level0_stop_writes_trigger: 36
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_base: 67108864
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_multiplier: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_compaction_bytes: 1677721600
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.arena_block_size: 33554432
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.disable_auto_compactions: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_pri: kMinOverlappingRatio
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.table_properties_collectors:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_support: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_num_locks: 10000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_whole_key_filtering: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_huge_page_size: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.bloom_locality: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.max_successive_merges: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.optimize_filters_for_hits: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.paranoid_file_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.force_consistency_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.report_bg_io_stats: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.ttl: 2592000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.562+0000 7fa6a95f8080 4 rocksdb:
Options.periodic_compaction_seconds: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
[column_family.cc:552] --------------- Options for column family [O-1]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.merge_operator: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter_factory: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_factory: SkipListFactory
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.table_factory: BlockBasedTable
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x560b4972f7c8)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks_with_high_priority: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_l0_filter_and_index_blocks_in_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_top_level_index_and_filter: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_shortening:
1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_hash_table_util_ratio: 0.750000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
hash_index_allow_collision: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: checksum: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: no_block_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache:
0x560b49774910
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache_name:
BinnedLRUCache
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_options:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: capacity :
536870912
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: num_shard_bits
: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
strict_capacity_limit : 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
high_pri_pool_ratio: 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_compressed: (nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: persistent_cache:
(nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_size_deviation: 10
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_restart_interval: 16
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
index_block_restart_interval: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
metadata_block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
partition_filters: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
use_delta_encoding: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: filter_policy:
rocksdb.BuiltinBloomFilter
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
whole_key_filtering: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
verify_compression: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
read_amp_bytes_per_bit: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: format_version: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
enable_index_compression: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_align: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.write_buffer_size: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression: NoCompression
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression: Disabled
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.num_levels: 7
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_size_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_stop_writes_trigger: 36
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_base: 67108864
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_multiplier: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_compaction_bytes: 1677721600
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.arena_block_size: 33554432
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.disable_auto_compactions: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_pri: kMinOverlappingRatio
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.table_properties_collectors:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_support: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_num_locks: 10000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_whole_key_filtering: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_huge_page_size: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bloom_locality: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_successive_merges: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.optimize_filters_for_hits: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.paranoid_file_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.force_consistency_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.report_bg_io_stats: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.ttl: 2592000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.periodic_compaction_seconds: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
[column_family.cc:552] --------------- Options for column family [O-2]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.merge_operator: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_filter_factory: None
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_factory: SkipListFactory
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.table_factory: BlockBasedTable
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x560b4972f7c8)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
cache_index_and_filter_blocks_with_high_priority: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_l0_filter_and_index_blocks_in_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
pin_top_level_index_and_filter: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_index_type: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: index_shortening:
1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
data_block_hash_table_util_ratio: 0.750000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
hash_index_allow_collision: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: checksum: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: no_block_cache: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache:
0x560b49774910
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_cache_name:
BinnedLRUCache
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_options:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: capacity :
536870912
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: num_shard_bits
: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
strict_capacity_limit : 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
high_pri_pool_ratio: 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_cache_compressed: (nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: persistent_cache:
(nil)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_size_deviation: 10
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
block_restart_interval: 16
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
index_block_restart_interval: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
metadata_block_size: 4096
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
partition_filters: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
use_delta_encoding: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: filter_policy:
rocksdb.BuiltinBloomFilter
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
whole_key_filtering: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
verify_compression: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
read_amp_bytes_per_bit: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: format_version: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
enable_index_compression: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: block_align: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.write_buffer_size: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression: NoCompression
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression: Disabled
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.num_levels: 7
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_write_buffer_size_to_maintain: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bottommost_compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.window_bits: -14
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.level: 32767
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.strategy: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.zstd_max_train_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compression_opts.enabled: false
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level0_stop_writes_trigger: 36
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_base: 67108864
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.target_file_size_multiplier: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_compaction_bytes: 1677721600
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.arena_block_size: 33554432
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.disable_auto_compactions: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_pri: kMinOverlappingRatio
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.table_properties_collectors:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_support: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.inplace_update_num_locks: 10000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_whole_key_filtering: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.memtable_huge_page_size: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.bloom_locality: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.max_successive_merges: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.optimize_filters_for_hits: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.paranoid_file_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.force_consistency_checks: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.report_bg_io_stats: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.ttl: 2592000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
Options.periodic_compaction_seconds: 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
[column_family.cc:555] (skipping printing options)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.563+0000 7fa6a95f8080 4 rocksdb:
[column_family.cc:555] (skipping printing options)
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4568]
Recovered from manifest file:db/MANIFEST-086679
succeeded,manifest_file_number is 86679, next_file_number is 86681,
last_sequence is 692690878, log_number is 86668,prev_log_number is
0,max_column_family is 11,min_log_number_to_keep is 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [default] (ID 0), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [m-0] (ID 1), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [m-1] (ID 2), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [m-2] (ID 3), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [p-0] (ID 4), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [p-1] (ID 5), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [p-2] (ID 6), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [O-0] (ID 7), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [O-1] (ID 8), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [O-2] (ID 9), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [L] (ID 10), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:4577]
Column family [P] (ID 11), log number is 86668
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653570837, "job": 1, "event": "recovery_started",
"log_files": [86680]}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 4 rocksdb:
[db_impl/db_impl_open.cc:760] Recovering log #86680 mode 2
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.569+0000 7fa6a95f8080 3 rocksdb:
[le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high
(20) bits/key. Dramatic filter space and/or accuracy improvement is
available with format_version>=5.
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.644+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653645421, "cf_name": "default", "job": 1, "event":
"table_file_creation", "file_number": 86681, "file_size": 1010,
"table_properties": {"data_size": 70, "index_size": 35, "index_partitions":
0, "top_level_index_size": 0, "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 37,
"raw_average_key_size": 18, "raw_value_size": 16, "raw_average_value_size":
8, "num_data_blocks": 1, "num_entries": 2, "num_deletions": 0,
"num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0,
"fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
"column_family_name": "default", "column_family_id": 0, "comparator":
"leveldb.BytewiseComparator", "merge_operator":
".T:int64_array.b:bitwise_xor", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
1658785653, "oldest_key_time": 3, "file_creation_time": 0}}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.694+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653695396, "cf_name": "O-0", "job": 1, "event":
"table_file_creation", "file_number": 86682, "file_size": 1556,
"table_properties": {"data_size": 612, "index_size": 60,
"index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size":
369, "raw_average_key_size": 41, "raw_value_size": 272,
"raw_average_value_size": 30, "num_data_blocks": 1, "num_entries": 9,
"num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0,
"format_version": 0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "O-0",
"column_family_id": 7, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
1658785653, "oldest_key_time": 3, "file_creation_time": 0}}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.744+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653745435, "cf_name": "O-1", "job": 1, "event":
"table_file_creation", "file_number": 86683, "file_size": 1627,
"table_properties": {"data_size": 683, "index_size": 60,
"index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size":
410, "raw_average_key_size": 41, "raw_value_size": 306,
"raw_average_value_size": 30, "num_data_blocks": 1, "num_entries": 10,
"num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0,
"format_version": 0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "O-1",
"column_family_id": 8, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
1658785653, "oldest_key_time": 3, "file_creation_time": 0}}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.794+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653795435, "cf_name": "O-2", "job": 1, "event":
"table_file_creation", "file_number": 86684, "file_size": 2092,
"table_properties": {"data_size": 1148, "index_size": 60,
"index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size":
697, "raw_average_key_size": 41, "raw_value_size": 515,
"raw_average_value_size": 30, "num_data_blocks": 1, "num_entries": 17,
"num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0,
"format_version": 0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "O-2",
"column_family_id": 9, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
1658785653, "oldest_key_time": 3, "file_creation_time": 0}}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.794+0000 7fa6a95f8080 4 rocksdb: [version_set.cc:3826]
Creating manifest 86685
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.919+0000 7fa6a95f8080 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1658785653920521, "job": 1, "event": "recovery_finished"}
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa6a95f8080 4 rocksdb: DB pointer
0x560b49839000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa699d12700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.996562) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa6a95f8080 1
bluestore(/var/lib/ceph/osd/ceph-34) _open_db opened rocksdb path db
options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa6a95f8080 1
bluestore(/var/lib/ceph/osd/ceph-34) _upgrade_super from 4, latest 4
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa6a95f8080 1
bluestore(/var/lib/ceph/osd/ceph-34) _upgrade_super done
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.995+0000 7fa699d12700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.996691) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699d12700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997391) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa692af5700 4 rocksdb:
[db_impl/db_impl.cc:850] ------- DUMPING STATS -------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa692af5700 4 rocksdb:
[db_impl/db_impl.cc:851]
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** DB Stats **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative writes:
0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest:
0.00 GB, 0.00 MB/s
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative WAL: 0
writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative stall:
00:00:0.000 H:M:S, 0.0 percent
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval writes: 0
writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00
MB, 0.00 MB/s
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval WAL: 0
writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval stall:
00:00:0.000 H:M:S, 0.0 percent
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
1.97 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.07 0.00 1 0.075 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 2/0
111.88 MB 0.4 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
111.88 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.07 0.00 1 0.075
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.07 0.00 1 0.075 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.07 0.00 1 0.075 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 4/0
140.78 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
140.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
2.60 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 3/0
133.79 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 5/0
133.79 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 3/0
134.61 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 3/0
134.61 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
3.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
628.05 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
636.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
3.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
628.16 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
636.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
4.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
626.04 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
634.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 3/0
147.61 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
166.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
314.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 1/0
32.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
51.98 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 2/0
52.01 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.4 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
1.97 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.07 0.00 1 0.075 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 2/0
111.88 MB 0.4 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
111.88 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.07 0.00 1 0.075
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.07 0.00 1 0.075 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [default] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [m-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 4/0
140.78 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
140.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
2.60 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 3/0
133.79 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 5/0
133.79 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 3/0
134.61 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 3/0
134.61 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [p-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
3.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
628.05 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
636.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-0] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
3.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
628.16 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
636.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.1 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-1] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 2/0
4.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
8.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L2 10/0
626.04 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 13/0
634.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1.0 0.0 0.0 0.05 0.00 1 0.050
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: User 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.05 0.00 1 0.050 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [O-2] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 3/0
147.61 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
166.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 4/0
314.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.00 0.00 0 0.000
0 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [L] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Level Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L0 1/0
32.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: L1 1/0
51.98 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Sum 2/0
52.01 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Int 0/0
0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.00 0.00 0 0.000 0
0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** Compaction Stats
[P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Priority Files
Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp
Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn
KeyDrop
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Uptime(secs): 0.4
total, 0.0 interval
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Flush(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(GB):
cumulative 0.000, interval 0.000
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Total
Files): cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(L0 Files):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: AddFile(Keys):
cumulative 0, interval 0
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Cumulative
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Interval
compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read,
0.0 seconds
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: Stalls(count): 0
level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: ** File Read
Latency Histogram By Level [P] **
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]:
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699d12700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997410) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699d12700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997449) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699511700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997481) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699511700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997548) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699511700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997567) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:33 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:33.996+0000 7fa699511700 4 rocksdb: (Original Log Time
2022/07/25-21:47:33.997582) [db_impl/db_impl_compaction_flush.cc:2611]
Compaction nothing to do
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.275+0000 7fa6a95f8080 0 <cls>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/cls/cephfs/cls_cephfs.cc:201:
loading cephfs
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.276+0000 7fa6a95f8080 0 <cls>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/cls/hello/cls_hello.cc:316:
loading cls_hello
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.276+0000 7fa6a95f8080 0 _get_class not permitted to
load kvs
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.276+0000 7fa6a95f8080 0 _get_class not permitted to
load lua
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.280+0000 7fa6a95f8080 0 _get_class not permitted to
load sdk
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.280+0000 7fa6a95f8080 0 osd.34 30689 crush map has
features 432629239337189376, adjusting msgr requires for clients
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.280+0000 7fa6a95f8080 0 osd.34 30689 crush map has
features 432629239337189376 was 8705, adjusting msgr requires for mons
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.280+0000 7fa6a95f8080 0 osd.34 30689 crush map has
features 3314933000854323200, adjusting msgr requires for osds
Jul 25 14:47:34 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:34.280+0000 7fa6a95f8080 1 osd.34 30689
check_osdmap_features require_osd_release unknown -> pacific
Jul 25 14:47:35 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:35.901+0000 7fa6a95f8080 0 osd.34 30689 load_pgs
Jul 25 14:47:39 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:39.711+0000 7fa6a95f8080 0 osd.34 30689 load_pgs opened
36 pgs
Jul 25 14:47:39 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:47:39.711+0000 7fa6a95f8080 -1 osd.34 30689 log_to_monitors
{default=true}
Jul 25 14:48:13 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:48:13.105+0000 7fa6a95f8080 0 osd.34 30689 done with init,
starting boot process
Jul 25 14:48:13 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:48:13.105+0000 7fa6a95f8080 1 osd.34 30689 start_boot
Jul 25 14:48:43 cn06.ceph.la1.clx.corp
ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[6928]: debug
2022-07-25T21:48:43.221+0000 7fa691cb9700 1 osd.34 30689 start_boot
On Mon, Jul 25, 2022 at 2:41 PM Adam King <adking@xxxxxxxxxx> wrote:
> Do the journal logs for any of the OSDs that are marked down give any
> useful info on why they're failing to start back up? If the host level ip
> issues have gone away I think that would be the next place to check.
>
> On Mon, Jul 25, 2022 at 5:03 PM Jeremy Hansen <
> farnsworth.mcfadden@xxxxxxxxx> wrote:
>
>> I noticed this on the initial run of ceph health, but I no longer see it.
>> When you say "don't use ceph adm", can you explain why this is bad?
>>
>> This is ceph health outside of cephadm shell:
>>
>> HEALTH_WARN 1 filesystem is degraded; 2 MDSs report slow metadata IOs; 2/5
>> mons down, quorum cn02,cn03,cn01; 10 osds down; 3 hosts (17 osds) down;
>> Reduced data ava
>> ilability: 13 pgs inactive, 9 pgs down; Degraded data redundancy:
>> 8515690/30862245 objects degraded (27.593%), 326 pgs degraded, 447 pgs
>> undersized
>> [WRN] FS_DEGRADED: 1 filesystem is degraded
>> fs coldlogix is degraded
>> [WRN] MDS_SLOW_METADATA_IO: 2 MDSs report slow metadata IOs
>> mds.coldlogix.cn01.uriofo(mds.0): 2 slow metadata IOs are blocked > 30
>> secs, oldest blocked for 3701 secs
>> mds.btc.cn02.ouvaus(mds.0): 1 slow metadata IOs are blocked > 30 secs,
>> oldest blocked for 382 secs
>> [WRN] MON_DOWN: 2/5 mons down, quorum cn02,cn03,cn01
>> mon.cn05 (rank 0) addr [v2:
>> 192.168.30.15:3300/0,v1:192.168.30.15:6789/0]
>> is down (out of quorum)
>> mon.cn04 (rank 3) addr [v2:
>> 192.168.30.14:3300/0,v1:192.168.30.14:6789/0]
>> is down (out of quorum)
>> [WRN] OSD_DOWN: 10 osds down
>> osd.0 (root=default,host=cn05) is down
>> osd.1 (root=default,host=cn06) is down
>> osd.7 (root=default,host=cn04) is down
>> osd.13 (root=default,host=cn06) is down
>> osd.15 (root=default,host=cn05) is down
>> osd.18 (root=default,host=cn04) is down
>> osd.20 (root=default,host=cn04) is down
>> osd.33 (root=default,host=cn06) is down
>> osd.34 (root=default,host=cn06) is down
>> osd.36 (root=default,host=cn05) is down
>> [WRN] OSD_HOST_DOWN: 3 hosts (17 osds) down
>> host cn04 (root=default) (6 osds) is down
>> host cn05 (root=default) (5 osds) is down
>> host cn06 (root=default) (6 osds) is down
>> [WRN] PG_AVAILABILITY: Reduced data availability: 13 pgs inactive, 9 pgs
>> down
>> pg 9.3a is down, acting [8]
>> pg 9.7a is down, acting [8]
>> pg 9.ba is down, acting [8]
>> pg 9.fa is down, acting [8]
>> pg 11.3 is stuck inactive for 39h, current state
>> undersized+degraded+peered, last acting [11]
>> pg 11.11 is down, acting [19,9]
>> pg 11.1f is stuck inactive for 13h, current state
>> undersized+degraded+peered, last acting [10]
>> pg 12.36 is down, acting [21,16]
>> pg 12.59 is down, acting [26,5]
>> pg 12.66 is down, acting [5]
>> pg 19.4 is stuck inactive for 39h, current state
>> undersized+degraded+peered, last acting [6]
>> pg 19.1c is down, acting [21,16,11]
>> pg 21.1 is stuck inactive for 2m, current state unknown, last acting
>> []
>> [WRN] PG_DEGRADED: Degraded data redundancy: 8515690/30862245 objects
>> degraded (27.593%), 326 pgs degraded, 447 pgs undersized
>> pg 9.75 is stuck undersized for 61m, current state
>> active+undersized+remapped, last acting [4,8,35]
>> pg 9.76 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [35,10,21]
>> pg 9.77 is stuck undersized for 61m, current state
>> active+undersized+remapped, last acting [32,35,4]
>> pg 9.78 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [14,10]
>> pg 9.79 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [21,32]
>> pg 9.7b is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,12,5]
>> pg 9.7c is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [4,35,10]
>> pg 9.7d is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [5,19,10]
>> pg 9.7e is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [21,10,17]
>> pg 9.80 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,4,17]
>> pg 9.81 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [14,26]
>> pg 9.82 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [26,16]
>> pg 9.83 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,4]
>> pg 9.84 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [4,35,6]
>> pg 9.85 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [32,12,9]
>> pg 9.86 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [35,5,8]
>> pg 9.87 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [9,12]
>> pg 9.88 is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [19,32,35]
>> pg 9.89 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [10,14,4]
>> pg 9.8a is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [21,19]
>> pg 9.8b is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,35]
>> pg 9.8c is stuck undersized for 58m, current state
>> active+undersized+remapped, last acting [10,19,5]
>> pg 9.8d is stuck undersized for 61m, current state
>> active+undersized+remapped, last acting [9,6]
>> pg 9.8f is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [19,26,17]
>> pg 9.90 is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [35,26]
>> pg 9.91 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [17,5]
>> pg 9.92 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [21,26]
>> pg 9.93 is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [19,26,5]
>> pg 9.94 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [21,11]
>> pg 9.95 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,19]
>> pg 9.96 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [17,6]
>> pg 9.97 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [8,9,16]
>> pg 9.98 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [6,21]
>> pg 9.99 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [10,9]
>> pg 9.9a is stuck undersized for 61m, current state
>> active+undersized+remapped, last acting [4,16,10]
>> pg 9.9b is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [12,4,11]
>> pg 9.9c is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [9,16]
>> pg 9.9d is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [26,35]
>> pg 9.9f is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [9,17,26]
>> pg 12.70 is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [21,35]
>> pg 12.71 is active+undersized+degraded, acting [6,12]
>> pg 12.72 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [10,14,4]
>> pg 12.73 is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [5,17,11]
>> pg 12.78 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [5,8,35]
>> pg 12.79 is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [4,17]
>> pg 12.7a is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [10,21]
>> pg 12.7b is stuck undersized for 62m, current state
>> active+undersized+remapped, last acting [17,21,11]
>> pg 12.7c is stuck undersized for 62m, current state
>> active+undersized+degraded, last acting [32,21,16]
>> pg 12.7d is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [35,6,9]
>> pg 12.7e is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [26,4]
>> pg 12.7f is stuck undersized for 61m, current state
>> active+undersized+degraded, last acting [9,14]
>>
>> It's no longer giving me the ssh key issues but hasn't done anything to
>> improve my situation. When the machines came up with a different IP, did
>> this somehow throw off some kind of ssh known hosts file or pub key
>> exchange? It's all very strange why a momentary bad IP could wreak so
>> much
>> havoc.
>>
>> Thank you
>> -jeremy
>>
>>
>> On Mon, Jul 25, 2022 at 1:44 PM Frank Schilder <frans@xxxxxx> wrote:
>>
>> > I don't use ceph-adm and I also don't know how you got the "some more
>> > info". However, I did notice that it contains instructions, starting at
>> > "Please make sure that the host is reachable ...". How about starting to
>> > follow those?
>> >
>> > Best regards,
>> > =================
>> > Frank Schilder
>> > AIT Risø Campus
>> > Bygning 109, rum S14
>> >
>> > ________________________________________
>> > From: Jeremy Hansen <farnsworth.mcfadden@xxxxxxxxx>
>> > Sent: 25 July 2022 22:32:32
>> > To: ceph-users@xxxxxxx
>> > Subject: [Warning Possible spam] Re: Issues after a
>> shutdown
>> >
>> > Here's some more info:
>> >
>> > HEALTH_WARN 2 failed cephadm daemon(s); 3 hosts fail cephadm check; 2
>> > filesystems are degraded; 1 MDSs report slow metadata IOs; 2/5 mons
>> down,
>> > quorum cn02,cn03,cn01; 10 osds down; 3 hosts (17 osds) down; Reduced
>> data
>> > availability: 13 pgs inactive, 9 pgs down; Degraded data redundancy:
>> > 8515690/30862245 objects degraded (27.593%), 326 pgs degraded, 447 pgs
>> > undersized
>> > [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
>> > daemon osd.3 on cn01.ceph is in error state
>> > daemon osd.2 on cn01.ceph is in error state
>> > [WRN] CEPHADM_HOST_CHECK_FAILED: 3 hosts fail cephadm check
>> > host cn04.ceph (192.168.30.14) failed check: Failed to connect to
>> > cn04.ceph (192.168.30.14).
>> > Please make sure that the host is reachable and accepts connections
>> using
>> > the cephadm SSH key
>> >
>> > To add the cephadm SSH key to the host:
>> > > ceph cephadm get-pub-key > ~/ceph.pub
>> > > ssh-copy-id -f -i ~/ceph.pub root@192.168.30.14
>> >
>> > To check that the host is reachable open a new shell with the --no-hosts
>> > flag:
>> > > cephadm shell --no-hosts
>> >
>> > Then run the following:
>> > > ceph cephadm get-ssh-config > ssh_config
>> > > ceph config-key get mgr/cephadm/ssh_identity_key >
>> ~/cephadm_private_key
>> > > chmod 0600 ~/cephadm_private_key
>> > > ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.14
>> > host cn06.ceph (192.168.30.16) failed check: Failed to connect to
>> > cn06.ceph (192.168.30.16).
>> > Please make sure that the host is reachable and accepts connections
>> using
>> > the cephadm SSH key
>> >
>> > To add the cephadm SSH key to the host:
>> > > ceph cephadm get-pub-key > ~/ceph.pub
>> > > ssh-copy-id -f -i ~/ceph.pub root@192.168.30.16
>> >
>> > To check that the host is reachable open a new shell with the --no-hosts
>> > flag:
>> > > cephadm shell --no-hosts
>> >
>> > Then run the following:
>> > > ceph cephadm get-ssh-config > ssh_config
>> > > ceph config-key get mgr/cephadm/ssh_identity_key >
>> ~/cephadm_private_key
>> > > chmod 0600 ~/cephadm_private_key
>> > > ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.16
>> > host cn05.ceph (192.168.30.15) failed check: Failed to connect to
>> > cn05.ceph (192.168.30.15).
>> > Please make sure that the host is reachable and accepts connections
>> using
>> > the cephadm SSH key
>> >
>> > To add the cephadm SSH key to the host:
>> > > ceph cephadm get-pub-key > ~/ceph.pub
>> > > ssh-copy-id -f -i ~/ceph.pub root@192.168.30.15
>> >
>> > To check that the host is reachable open a new shell with the --no-hosts
>> > flag:
>> > > cephadm shell --no-hosts
>> >
>> > Then run the following:
>> > > ceph cephadm get-ssh-config > ssh_config
>> > > ceph config-key get mgr/cephadm/ssh_identity_key >
>> ~/cephadm_private_key
>> > > chmod 0600 ~/cephadm_private_key
>> > > ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.15
>> > [WRN] FS_DEGRADED: 2 filesystems are degraded
>> > fs coldlogix is degraded
>> > fs btc is degraded
>> > [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
>> > mds.coldlogix.cn01.uriofo(mds.0): 2 slow metadata IOs are blocked >
>> 30
>> > secs, oldest blocked for 2096 secs
>> > [WRN] MON_DOWN: 2/5 mons down, quorum cn02,cn03,cn01
>> > mon.cn05 (rank 0) addr [v2:
>> > 192.168.30.15:3300/0,v1:192.168.30.15:6789/0]
>> > is down (out of quorum)
>> > mon.cn04 (rank 3) addr [v2:
>> > 192.168.30.14:3300/0,v1:192.168.30.14:6789/0]
>> > is down (out of quorum)
>> > [WRN] OSD_DOWN: 10 osds down
>> > osd.0 (root=default,host=cn05) is down
>> > osd.1 (root=default,host=cn06) is down
>> > osd.7 (root=default,host=cn04) is down
>> > osd.13 (root=default,host=cn06) is down
>> > osd.15 (root=default,host=cn05) is down
>> > osd.18 (root=default,host=cn04) is down
>> > osd.20 (root=default,host=cn04) is down
>> > osd.33 (root=default,host=cn06) is down
>> > osd.34 (root=default,host=cn06) is down
>> > osd.36 (root=default,host=cn05) is down
>> > [WRN] OSD_HOST_DOWN: 3 hosts (17 osds) down
>> > host cn04 (root=default) (6 osds) is down
>> > host cn05 (root=default) (5 osds) is down
>> > host cn06 (root=default) (6 osds) is down
>> > [WRN] PG_AVAILABILITY: Reduced data availability: 13 pgs inactive, 9 pgs
>> > down
>> > pg 9.3a is down, acting [8]
>> > pg 9.7a is down, acting [8]
>> > pg 9.ba is down, acting [8]
>> > pg 9.fa is down, acting [8]
>> > pg 11.3 is stuck inactive for 39h, current state
>> > undersized+degraded+peered, last acting [11]
>> > pg 11.11 is down, acting [19,9]
>> > pg 11.1f is stuck inactive for 13h, current state
>> > undersized+degraded+peered, last acting [10]
>> > pg 12.36 is down, acting [21,16]
>> > pg 12.59 is down, acting [26,5]
>> > pg 12.66 is down, acting [5]
>> > pg 19.4 is stuck inactive for 39h, current state
>> > undersized+degraded+peered, last acting [6]
>> > pg 19.1c is down, acting [21,16,11]
>> > pg 21.1 is stuck inactive for 36m, current state unknown, last
>> acting
>> > []
>> > [WRN] PG_DEGRADED: Degraded data redundancy: 8515690/30862245 objects
>> > degraded (27.593%), 326 pgs degraded, 447 pgs undersized
>> > pg 9.75 is stuck undersized for 34m, current state
>> > active+undersized+remapped, last acting [4,8,35]
>> > pg 9.76 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [35,10,21]
>> > pg 9.77 is stuck undersized for 34m, current state
>> > active+undersized+remapped, last acting [32,35,4]
>> > pg 9.78 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [14,10]
>> > pg 9.79 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [21,32]
>> > pg 9.7b is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,12,5]
>> > pg 9.7c is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [4,35,10]
>> > pg 9.7d is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [5,19,10]
>> > pg 9.7e is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [21,10,17]
>> > pg 9.80 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,4,17]
>> > pg 9.81 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [14,26]
>> > pg 9.82 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [26,16]
>> > pg 9.83 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,4]
>> > pg 9.84 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [4,35,6]
>> > pg 9.85 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [32,12,9]
>> > pg 9.86 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [35,5,8]
>> > pg 9.87 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [9,12]
>> > pg 9.88 is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [19,32,35]
>> > pg 9.89 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [10,14,4]
>> > pg 9.8a is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [21,19]
>> > pg 9.8b is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,35]
>> > pg 9.8c is stuck undersized for 31m, current state
>> > active+undersized+remapped, last acting [10,19,5]
>> > pg 9.8d is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [9,6]
>> > pg 9.8f is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [19,26,17]
>> > pg 9.90 is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [35,26]
>> > pg 9.91 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [17,5]
>> > pg 9.92 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [21,26]
>> > pg 9.93 is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [19,26,5]
>> > pg 9.94 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [21,11]
>> > pg 9.95 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,19]
>> > pg 9.96 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [17,6]
>> > pg 9.97 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [8,9,16]
>> > pg 9.98 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [6,21]
>> > pg 9.99 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [10,9]
>> > pg 9.9a is stuck undersized for 34m, current state
>> > active+undersized+remapped, last acting [4,16,10]
>> > pg 9.9b is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [12,4,11]
>> > pg 9.9c is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [9,16]
>> > pg 9.9d is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [26,35]
>> > pg 9.9f is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [9,17,26]
>> > pg 12.70 is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [21,35]
>> > pg 12.71 is active+undersized+degraded, acting [6,12]
>> > pg 12.72 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [10,14,4]
>> > pg 12.73 is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [5,17,11]
>> > pg 12.78 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [5,8,35]
>> > pg 12.79 is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [4,17]
>> > pg 12.7a is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [10,21]
>> > pg 12.7b is stuck undersized for 35m, current state
>> > active+undersized+remapped, last acting [17,21,11]
>> > pg 12.7c is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [32,21,16]
>> > pg 12.7d is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [35,6,9]
>> > pg 12.7e is stuck undersized for 34m, current state
>> > active+undersized+degraded, last acting [26,4]
>> > pg 12.7f is stuck undersized for 35m, current state
>> > active+undersized+degraded, last acting [9,14]
>> >
>> > On Mon, Jul 25, 2022 at 12:43 PM Jeremy Hansen <
>> > farnsworth.mcfadden@xxxxxxxxx> wrote:
>> >
>> > > Pretty desperate here. Can someone suggest what I might be able to
>> do to
>> > > get these OSDs back up. It looks like my recovery had stalled.
>> > >
>> > >
>> > > On Mon, Jul 25, 2022 at 7:26 AM Anthony D'Atri <
>> anthony.datri@xxxxxxxxx>
>> > > wrote:
>> > >
>> > >> Do your values for public and cluster network include the new
>> addresses
>> > >> on all nodes?
>> > >>
>> > >
>> > > This cluster only has one network. There is no separation between
>> > > public and cluster. Three of the nodes momentarily came up using a
>> > > different IP address.
>> > >
>> > > I've also noticed on one of the nodes that did not move or have any IP
>> > > issue, on a single node, from the dashboard, it names the same device
>> for
>> > > two different osd's:
>> > >
>> > > 2 cn01 out destroyed hdd TOSHIBA_MG04SCA40EE_21M0A0CKFWZB Unknown sdb
>> > osd.2
>> > >
>> > > 3 cn01 out destroyed ssd SAMSUNG_MZILT3T8HBLS/007_S5G0NE0R200159
>> Unknown
>> > > sdb osd.3
>> > >
>> > >
>> > > [ceph: root@cn01 /]# ceph-volume inventory
>> > >
>> > > Device Path Size rotates available Model name
>> > > /dev/sda 3.64 TB True False MG04SCA40EE
>> > > /dev/sdb 3.49 TB False False
>> MZILT3T8HBLS/007
>> > > /dev/sdc 3.64 TB True False MG04SCA40EE
>> > > /dev/sdd 3.64 TB True False MG04SCA40EE
>> > > /dev/sde 3.49 TB False False
>> MZILT3T8HBLS/007
>> > > /dev/sdf 3.64 TB True False MG04SCA40EE
>> > > /dev/sdg 698.64 GB True False SEAGATE
>> ST375064
>> > >
>> > > [ceph: root@cn01 /]# ceph osd info
>> > > osd.0 down out weight 0 up_from 30231 up_thru 30564 down_at 30688
>> > > last_clean_interval [25500,30228) [v2:
>> > > 192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421] [v2:
>> > > 192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421]
>> > > autoout,exists d14cf503-a303-4fa4-a713-9530b67d613a
>> > > osd.1 down out weight 0 up_from 30393 up_thru 30688 down_at 30697
>> > > last_clean_interval [25518,30321) [v2:
>> > > 192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831] [v2:
>> > > 192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831]
>> > > autoout,exists 0d521411-c835-4fa3-beca-3631b4ff6bf7
>> > > osd.2 down out weight 0 up_from 31316 up_thru 31293 down_at 31317
>> > > last_clean_interval [31218,31296) [v2:
>> > > 192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880] [v2:
>> > > 192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880]
>> > > destroyed,exists
>> > > osd.3 down out weight 0 up_from 31265 up_thru 31266 down_at 31268
>> > > last_clean_interval [31254,31256) [v2:
>> > > 192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535] [v2:
>> > > 192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535]
>> > > destroyed,exists
>> > > osd.4 up in weight 1 up_from 31356 up_thru 31581 down_at 31339
>> > > last_clean_interval [31320,31338) [v2:
>> > > 192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179] [v2:
>> > > 192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179]
>> > exists,up
>> > > 3afd06db-b91d-44fe-9305-5eb95f7a59b9
>> > > osd.5 up in weight 1 up_from 31347 up_thru 31699 down_at 31339
>> > > last_clean_interval [31311,31338) [v2:
>> > > 192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540] [v2:
>> > > 192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540]
>> > exists,up
>> > > 063c2ccf-02ce-4f5e-8252-dddfbb258a95
>> > > osd.6 up in weight 1 up_from 31218 up_thru 31711 down_at 31217
>> > > last_clean_interval [30978,31195) [v2:
>> > > 192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160] [v2:
>> > > 192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160]
>> > exists,up
>> > > 94250ea2-f12e-4dc6-9135-b626086ccffd
>> > > osd.7 down out weight 0 up_from 30353 up_thru 30558 down_at 30688
>> > > last_clean_interval [25533,30349) [v2:
>> > > 192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061] [v2:
>> > > 192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061]
>> > > autoout,exists de351aec-b91e-4c22-a0bf-85369bc14579
>> > > osd.8 up in weight 1 up_from 31226 up_thru 31668 down_at 31225
>> > > last_clean_interval [30983,31195) [v2:
>> > > 192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329] [v2:
>> > > 192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329]
>> > exists,up
>> > > 51f665b4-fa5b-4b17-8390-ed130145ef04
>> > > osd.9 up in weight 1 up_from 31351 up_thru 31673 down_at 31340
>> > > last_clean_interval [31315,31338) [v2:
>> > > 192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877] [v2:
>> > > 192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877]
>> > exists,up
>> > > 985f1127-d126-4629-b8cd-03cf2d914d99
>> > > osd.10 up in weight 1 up_from 31219 up_thru 31639 down_at 31218
>> > > last_clean_interval [30980,31195) [v2:
>> > > 192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953] [v2:
>> > > 192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953]
>> > exists,up
>> > > c7fca03e-4bd5-4485-a090-658ca967d5f6
>> > > osd.11 up in weight 1 up_from 31234 up_thru 31659 down_at 31223
>> > > last_clean_interval [30978,31195) [v2:
>> > > 192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742] [v2:
>> > > 192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742]
>> > exists,up
>> > > 81074bd7-ad9f-4e56-8885-cca4745f6c95
>> > > osd.12 up in weight 1 up_from 31230 up_thru 31717 down_at 31223
>> > > last_clean_interval [30975,31195) [v2:
>> > > 192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910] [v2:
>> > > 192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910]
>> > exists,up
>> > > af1b55dd-c110-4861-aed9-c0737cef8be1
>> > > osd.13 down out weight 0 up_from 30389 up_thru 30688 down_at 30695
>> > > last_clean_interval [25513,30317) [v2:
>> > > 192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727] [v2:
>> > > 192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727]
>> > > autoout,exists 737a3234-0f1f-4286-80e9-e89b581aae41
>> > > osd.14 up in weight 1 up_from 31218 up_thru 31709 down_at 31217
>> > > last_clean_interval [30979,31195) [v2:
>> > > 192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817] [v2:
>> > > 192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817]
>> > exists,up
>> > > 97cd6ac7-aca0-42fd-a049-d27289f83183
>> > > osd.15 down out weight 0 up_from 29463 up_thru 30531 down_at 30688
>> > > last_clean_interval [25493,29462) [v2:
>> > > 192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991] [v2:
>> > > 192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991]
>> > > autoout,exists 61aea8f4-5905-4be3-ae32-5eacf75a514e
>> > > osd.16 up in weight 1 up_from 31226 up_thru 31647 down_at 31223
>> > > last_clean_interval [30970,31195) [v2:
>> > > 192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081] [v2:
>> > > 192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081]
>> > exists,up
>> > > 791a7542-87cd-403d-a37e-8f00506b2eb6
>> > > osd.17 up in weight 1 up_from 31219 up_thru 31703 down_at 31218
>> > > last_clean_interval [30975,31195) [v2:
>> > > 192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397] [v2:
>> > > 192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397]
>> > exists,up
>> > > 4a915645-412f-49e6-8477-1577469905da
>> > > osd.18 down out weight 0 up_from 30334 up_thru 30566 down_at 30688
>> > > last_clean_interval [25543,30327) [v2:
>> > > 192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137] [v2:
>> > > 192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137]
>> > > autoout,exists 85f59d83-710c-4896-9200-bda4894fc3e8
>> > > osd.19 down in weight 1 up_from 31303 up_thru 31321 down_at 31323
>> > > last_clean_interval [31292,31296) [v2:
>> > > 192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427] [v2:
>> > > 192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427] exists
>> > > 7d09b51a-bd6d-40f8-a009-78ab9937795d
>> > > osd.20 down out weight 0 up_from 30445 up_thru 30531 down_at 30688
>> > > last_clean_interval [28947,30444) [v2:
>> > > 192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162] [v2:
>> > > 192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162]
>> > > autoout,exists 7ef6cc1a-4755-4a14-b9df-f1f538d903ce
>> > > osd.21 up in weight 1 up_from 31345 up_thru 31567 down_at 31341
>> > > last_clean_interval [31307,31340) [v2:
>> > > 192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298] [v2:
>> > > 192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298]
>> > exists,up
>> > > 5ef2e39a-a353-4cb8-a49e-093fe39b94ef
>> > > osd.22 down in weight 1 up_from 30383 up_thru 30528 down_at 30688
>> > > last_clean_interval [25549,30317) [v2:
>> > > 192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629] [v2:
>> > > 192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629]
>> exists
>> > > c9befe11-a035-449c-8d17-42aaf191923d
>> > > osd.23 down in weight 1 up_from 30334 up_thru 30576 down_at 30688
>> > > last_clean_interval [30263,30332) [v2:
>> > > 192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490] [v2:
>> > > 192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490]
>> exists
>> > > 2081147b-065d-4da7-89d9-747e1ae02b8d
>> > > osd.24 down in weight 1 up_from 29455 up_thru 30570 down_at 30688
>> > > last_clean_interval [25487,29453) [v2:
>> > > 192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583] [v2:
>> > > 192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583]
>> exists
>> > > 39d78380-261c-4689-b53d-90713e6ffcca
>> > > osd.26 up in weight 1 up_from 31208 up_thru 31643 down_at 31207
>> > > last_clean_interval [30967,31195) [v2:
>> > > 192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947] [v2:
>> > > 192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947]
>> > exists,up
>> > > 046622c8-c09c-4254-8c15-3dc05a2f01ed
>> > > osd.28 down in weight 1 up_from 30389 up_thru 30574 down_at 30691
>> > > last_clean_interval [25513,30312) [v2:
>> > > 192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570] [v2:
>> > > 192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570]
>> exists
>> > > 10578b97-e3c4-4553-a8d0-6dcc46af5db1
>> > > osd.29 down in weight 1 up_from 30378 up_thru 30554 down_at 30688
>> > > last_clean_interval [28595,30376) [v2:
>> > > 192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672] [v2:
>> > > 192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672]
>> exists
>> > > 9698e936-8edf-4adf-92c9-a0b5202ed01a
>> > > osd.30 down in weight 1 up_from 30449 up_thru 30531 down_at 30688
>> > > last_clean_interval [25502,30446) [v2:
>> > > 192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296] [v2:
>> > > 192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296]
>> exists
>> > > e14d2a0f-a98a-44d4-8c06-4d893f673629
>> > > osd.31 down in weight 1 up_from 30364 up_thru 30688 down_at 30700
>> > > last_clean_interval [25514,30361) [v2:
>> > > 192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708] [v2:
>> > > 192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708]
>> exists
>> > > 541bca38-e704-483a-8cb8-39e5f69007d1
>> > > osd.32 up in weight 1 up_from 31209 up_thru 31627 down_at 31208
>> > > last_clean_interval [30974,31195) [v2:
>> > > 192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997] [v2:
>> > > 192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997]
>> > exists,up
>> > > 9200a57e-2845-43ff-9787-8f1f3158fe90
>> > > osd.33 down in weight 1 up_from 30354 up_thru 30688 down_at 30693
>> > > last_clean_interval [25521,30350) [v2:
>> > > 192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666] [v2:
>> > > 192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666]
>> exists
>> > > 20c55d85-cf9a-4133-a189-7fdad2318f58
>> > > osd.34 down in weight 1 up_from 30390 up_thru 30688 down_at 30691
>> > > last_clean_interval [25516,30314) [v2:
>> > > 192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870] [v2:
>> > > 192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870]
>> exists
>> > > 77e0ef8f-c047-4f84-afb2-a8ad054e562f
>> > > osd.35 up in weight 1 up_from 31204 up_thru 31657 down_at 31203
>> > > last_clean_interval [30958,31195) [v2:
>> > > 192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520] [v2:
>> > > 192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520]
>> > exists,up
>> > > 2d2de0cb-6d41-4957-a473-2bbe9ce227bf
>> > > osd.36 down in weight 1 up_from 29494 up_thru 30560 down_at 30688
>> > > last_clean_interval [25491,29492) [v2:
>> > > 192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591] [v2:
>> > > 192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591]
>> exists
>> > > 26114668-68b2-458b-89c2-cbad5507ab75
>> > >
>> > >
>> > >
>> > >>
>> > >> > On Jul 25, 2022, at 3:29 AM, Jeremy Hansen <
>> > >> farnsworth.mcfadden@xxxxxxxxx> wrote:
>> > >> >
>> > >> > I transitioned some servers to a new rack and now I'm having major
>> > >> issues
>> > >> > with Ceph upon bringing things back up.
>> > >> >
>> > >> > I believe the issue may be related to the ceph nodes coming back up
>> > with
>> > >> > different IPs before VLANs were set. That's just a guess because I
>> > >> can't
>> > >> > think of any other reason this would happen.
>> > >> >
>> > >> > Current state:
>> > >> >
>> > >> > Every 2.0s: ceph -s
>> > >> > cn01.ceph.la1.clx.corp: Mon Jul 25 10:13:05 2022
>> > >> >
>> > >> > cluster:
>> > >> > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
>> > >> > health: HEALTH_WARN
>> > >> > 1 filesystem is degraded
>> > >> > 2 MDSs report slow metadata IOs
>> > >> > 2/5 mons down, quorum cn02,cn03,cn01
>> > >> > 9 osds down
>> > >> > 3 hosts (17 osds) down
>> > >> > Reduced data availability: 97 pgs inactive, 9 pgs down
>> > >> > Degraded data redundancy: 13860144/30824413 objects
>> > degraded
>> > >> > (44.965%), 411 pgs degraded, 482 pgs undersized
>> > >> >
>> > >> > services:
>> > >> > mon: 5 daemons, quorum cn02,cn03,cn01 (age 62m), out of quorum:
>> > cn05,
>> > >> > cn04
>> > >> > mgr: cn02.arszct(active, since 5m)
>> > >> > mds: 2/2 daemons up, 2 standby
>> > >> > osd: 35 osds: 15 up (since 62m), 24 in (since 58m); 222 remapped
>> > pgs
>> > >> >
>> > >> > data:
>> > >> > volumes: 1/2 healthy, 1 recovering
>> > >> > pools: 8 pools, 545 pgs
>> > >> > objects: 7.71M objects, 6.7 TiB
>> > >> > usage: 15 TiB used, 39 TiB / 54 TiB avail
>> > >> > pgs: 0.367% pgs unknown
>> > >> > 17.431% pgs not active
>> > >> > 13860144/30824413 objects degraded (44.965%)
>> > >> > 1137693/30824413 objects misplaced (3.691%)
>> > >> > 280 active+undersized+degraded
>> > >> > 67 undersized+degraded+remapped+backfilling+peered
>> > >> > 57 active+undersized+remapped
>> > >> > 45 active+clean+remapped
>> > >> > 44 active+undersized+degraded+remapped+backfilling
>> > >> > 18 undersized+degraded+peered
>> > >> > 10 active+undersized
>> > >> > 9 down
>> > >> > 7 active+clean
>> > >> > 3 active+undersized+remapped+backfilling
>> > >> > 2 active+undersized+degraded+remapped+backfill_wait
>> > >> > 2 unknown
>> > >> > 1 undersized+peered
>> > >> >
>> > >> > io:
>> > >> > client: 170 B/s rd, 0 op/s rd, 0 op/s wr
>> > >> > recovery: 168 MiB/s, 158 keys/s, 166 objects/s
>> > >> >
>> > >> > I have to disable and re-enable the dashboard just to use it. It
>> > seems
>> > >> to
>> > >> > get bogged down after a few moments.
>> > >> >
>> > >> > The three servers that were moved to the new rack Ceph has marked
>> as
>> > >> > "Down", but if I do a cephadm host-check, they all seem to pass:
>> > >> >
>> > >> > ************************ ceph ************************
>> > >> > --------- cn01.ceph.---------
>> > >> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> > --------- cn02.ceph.---------
>> > >> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> > --------- cn03.ceph.---------
>> > >> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> > --------- cn04.ceph.---------
>> > >> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> > --------- cn05.ceph.---------
>> > >> > podman|docker (/usr/bin/podman) is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> > --------- cn06.ceph.---------
>> > >> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >> > systemctl is present
>> > >> > lvcreate is present
>> > >> > Unit chronyd.service is enabled and running
>> > >> > Host looks OK
>> > >> >
>> > >> > It seems to be recovering with what it has left, but a large
>> amount of
>> > >> OSDs
>> > >> > are down. When trying to restart one of the down'd OSDs, I see a
>> huge
>> > >> dump.
>> > >> >
>> > >> > Jul 25 03:19:38 cn06.ceph
>> > >> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 0 osd.34 30689 done with
>> > >> init,
>> > >> > starting boot process
>> > >> > Jul 25 03:19:38 cn06.ceph
>> > >> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 1 osd.34 30689
>> start_boot
>> > >> > Jul 25 03:20:10 cn06.ceph
>> > >> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >> > 2022-07-25T10:20:10.655+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >> > Jul 25 03:20:41 cn06.ceph
>> > >> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >> > 2022-07-25T10:20:41.159+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >> > Jul 25 03:21:11 cn06.ceph
>> > >> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >> > 2022-07-25T10:21:11.662+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >> >
>> > >> > At this point it just keeps printing start_boot, but the dashboard
>> has
>> > >> it
>> > >> > marked as "in" but "down".
>> > >> >
>> > >> > On these three hosts that moved, there were a bunch marked as "out"
>> > and
>> > >> > "down", and some with "in" but "down".
>> > >> >
>> > >> > Not sure where to go next. I'm going to let the recovery continue
>> and
>> > >> hope
>> > >> > that my 4x replication on these pools saves me.
>> > >> >
>> > >> > Not sure where to go from here. Any help is very much appreciated.
>> > >> This
>> > >> > Ceph cluster holds all of our Cloudstack images... it would be
>> > >> terrible to
>> > >> > lose this data.
>> > >> > _______________________________________________
>> > >> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> > >>
>> > >
>> > > On Mon, Jul 25, 2022 at 10:15 AM Jeremy Hansen <
>> > > farnsworth.mcfadden@xxxxxxxxx> wrote:
>> > >
>> > >>
>> > >>
>> > >> On Mon, Jul 25, 2022 at 7:26 AM Anthony D'Atri <
>> anthony.datri@xxxxxxxxx
>> > >
>> > >> wrote:
>> > >>
>> > >>> Do your values for public and cluster network include the new
>> addresses
>> > >>> on all nodes?
>> > >>>
>> > >>
>> > >> This cluster only has one network. There is no separation between
>> > >> public and cluster. Three of the nodes momentarily came up using a
>> > >> different IP address.
>> > >>
>> > >> I've also noticed on one of the nodes that did not move or have any
>> IP
>> > >> issue, on a single node, from the dashboard, it names the same device
>> > for
>> > >> two different osd's:
>> > >>
>> > >> 2 cn01 out destroyed hdd TOSHIBA_MG04SCA40EE_21M0A0CKFWZB Unknown sdb
>> > >> osd.2
>> > >>
>> > >> 3 cn01 out destroyed ssd SAMSUNG_MZILT3T8HBLS/007_S5G0NE0R200159
>> Unknown
>> > >> sdb osd.3
>> > >>
>> > >>
>> > >> [ceph: root@cn01 /]# ceph-volume inventory
>> > >>
>> > >> Device Path Size rotates available Model name
>> > >> /dev/sda 3.64 TB True False MG04SCA40EE
>> > >> /dev/sdb 3.49 TB False False
>> > MZILT3T8HBLS/007
>> > >> /dev/sdc 3.64 TB True False MG04SCA40EE
>> > >> /dev/sdd 3.64 TB True False MG04SCA40EE
>> > >> /dev/sde 3.49 TB False False
>> > MZILT3T8HBLS/007
>> > >> /dev/sdf 3.64 TB True False MG04SCA40EE
>> > >> /dev/sdg 698.64 GB True False SEAGATE
>> > ST375064
>> > >>
>> > >> [ceph: root@cn01 /]# ceph osd info
>> > >> osd.0 down out weight 0 up_from 30231 up_thru 30564 down_at 30688
>> > >> last_clean_interval [25500,30228) [v2:
>> > >> 192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421] [v2:
>> > >> 192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421]
>> > >> autoout,exists d14cf503-a303-4fa4-a713-9530b67d613a
>> > >> osd.1 down out weight 0 up_from 30393 up_thru 30688 down_at 30697
>> > >> last_clean_interval [25518,30321) [v2:
>> > >> 192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831] [v2:
>> > >> 192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831]
>> > >> autoout,exists 0d521411-c835-4fa3-beca-3631b4ff6bf7
>> > >> osd.2 down out weight 0 up_from 31316 up_thru 31293 down_at 31317
>> > >> last_clean_interval [31218,31296) [v2:
>> > >> 192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880] [v2:
>> > >> 192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880]
>> > >> destroyed,exists
>> > >> osd.3 down out weight 0 up_from 31265 up_thru 31266 down_at 31268
>> > >> last_clean_interval [31254,31256) [v2:
>> > >> 192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535] [v2:
>> > >> 192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535]
>> > >> destroyed,exists
>> > >> osd.4 up in weight 1 up_from 31356 up_thru 31581 down_at 31339
>> > >> last_clean_interval [31320,31338) [v2:
>> > >> 192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179] [v2:
>> > >> 192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179]
>> > >> exists,up 3afd06db-b91d-44fe-9305-5eb95f7a59b9
>> > >> osd.5 up in weight 1 up_from 31347 up_thru 31699 down_at 31339
>> > >> last_clean_interval [31311,31338) [v2:
>> > >> 192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540] [v2:
>> > >> 192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540]
>> > >> exists,up 063c2ccf-02ce-4f5e-8252-dddfbb258a95
>> > >> osd.6 up in weight 1 up_from 31218 up_thru 31711 down_at 31217
>> > >> last_clean_interval [30978,31195) [v2:
>> > >> 192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160] [v2:
>> > >> 192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160]
>> > >> exists,up 94250ea2-f12e-4dc6-9135-b626086ccffd
>> > >> osd.7 down out weight 0 up_from 30353 up_thru 30558 down_at 30688
>> > >> last_clean_interval [25533,30349) [v2:
>> > >> 192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061] [v2:
>> > >> 192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061]
>> > >> autoout,exists de351aec-b91e-4c22-a0bf-85369bc14579
>> > >> osd.8 up in weight 1 up_from 31226 up_thru 31668 down_at 31225
>> > >> last_clean_interval [30983,31195) [v2:
>> > >> 192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329] [v2:
>> > >> 192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329]
>> > >> exists,up 51f665b4-fa5b-4b17-8390-ed130145ef04
>> > >> osd.9 up in weight 1 up_from 31351 up_thru 31673 down_at 31340
>> > >> last_clean_interval [31315,31338) [v2:
>> > >> 192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877] [v2:
>> > >> 192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877]
>> > >> exists,up 985f1127-d126-4629-b8cd-03cf2d914d99
>> > >> osd.10 up in weight 1 up_from 31219 up_thru 31639 down_at 31218
>> > >> last_clean_interval [30980,31195) [v2:
>> > >> 192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953] [v2:
>> > >> 192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953]
>> > >> exists,up c7fca03e-4bd5-4485-a090-658ca967d5f6
>> > >> osd.11 up in weight 1 up_from 31234 up_thru 31659 down_at 31223
>> > >> last_clean_interval [30978,31195) [v2:
>> > >> 192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742] [v2:
>> > >> 192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742]
>> > >> exists,up 81074bd7-ad9f-4e56-8885-cca4745f6c95
>> > >> osd.12 up in weight 1 up_from 31230 up_thru 31717 down_at 31223
>> > >> last_clean_interval [30975,31195) [v2:
>> > >> 192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910] [v2:
>> > >> 192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910]
>> > >> exists,up af1b55dd-c110-4861-aed9-c0737cef8be1
>> > >> osd.13 down out weight 0 up_from 30389 up_thru 30688 down_at 30695
>> > >> last_clean_interval [25513,30317) [v2:
>> > >> 192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727] [v2:
>> > >> 192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727]
>> > >> autoout,exists 737a3234-0f1f-4286-80e9-e89b581aae41
>> > >> osd.14 up in weight 1 up_from 31218 up_thru 31709 down_at 31217
>> > >> last_clean_interval [30979,31195) [v2:
>> > >> 192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817] [v2:
>> > >> 192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817]
>> > >> exists,up 97cd6ac7-aca0-42fd-a049-d27289f83183
>> > >> osd.15 down out weight 0 up_from 29463 up_thru 30531 down_at 30688
>> > >> last_clean_interval [25493,29462) [v2:
>> > >> 192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991] [v2:
>> > >> 192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991]
>> > >> autoout,exists 61aea8f4-5905-4be3-ae32-5eacf75a514e
>> > >> osd.16 up in weight 1 up_from 31226 up_thru 31647 down_at 31223
>> > >> last_clean_interval [30970,31195) [v2:
>> > >> 192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081] [v2:
>> > >> 192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081]
>> > >> exists,up 791a7542-87cd-403d-a37e-8f00506b2eb6
>> > >> osd.17 up in weight 1 up_from 31219 up_thru 31703 down_at 31218
>> > >> last_clean_interval [30975,31195) [v2:
>> > >> 192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397] [v2:
>> > >> 192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397]
>> > >> exists,up 4a915645-412f-49e6-8477-1577469905da
>> > >> osd.18 down out weight 0 up_from 30334 up_thru 30566 down_at 30688
>> > >> last_clean_interval [25543,30327) [v2:
>> > >> 192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137] [v2:
>> > >> 192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137]
>> > >> autoout,exists 85f59d83-710c-4896-9200-bda4894fc3e8
>> > >> osd.19 down in weight 1 up_from 31303 up_thru 31321 down_at 31323
>> > >> last_clean_interval [31292,31296) [v2:
>> > >> 192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427] [v2:
>> > >> 192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427] exists
>> > >> 7d09b51a-bd6d-40f8-a009-78ab9937795d
>> > >> osd.20 down out weight 0 up_from 30445 up_thru 30531 down_at 30688
>> > >> last_clean_interval [28947,30444) [v2:
>> > >> 192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162] [v2:
>> > >> 192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162]
>> > >> autoout,exists 7ef6cc1a-4755-4a14-b9df-f1f538d903ce
>> > >> osd.21 up in weight 1 up_from 31345 up_thru 31567 down_at 31341
>> > >> last_clean_interval [31307,31340) [v2:
>> > >> 192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298] [v2:
>> > >> 192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298]
>> > >> exists,up 5ef2e39a-a353-4cb8-a49e-093fe39b94ef
>> > >> osd.22 down in weight 1 up_from 30383 up_thru 30528 down_at 30688
>> > >> last_clean_interval [25549,30317) [v2:
>> > >> 192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629] [v2:
>> > >> 192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629]
>> exists
>> > >> c9befe11-a035-449c-8d17-42aaf191923d
>> > >> osd.23 down in weight 1 up_from 30334 up_thru 30576 down_at 30688
>> > >> last_clean_interval [30263,30332) [v2:
>> > >> 192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490] [v2:
>> > >> 192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490]
>> exists
>> > >> 2081147b-065d-4da7-89d9-747e1ae02b8d
>> > >> osd.24 down in weight 1 up_from 29455 up_thru 30570 down_at 30688
>> > >> last_clean_interval [25487,29453) [v2:
>> > >> 192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583] [v2:
>> > >> 192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583]
>> exists
>> > >> 39d78380-261c-4689-b53d-90713e6ffcca
>> > >> osd.26 up in weight 1 up_from 31208 up_thru 31643 down_at 31207
>> > >> last_clean_interval [30967,31195) [v2:
>> > >> 192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947] [v2:
>> > >> 192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947]
>> > >> exists,up 046622c8-c09c-4254-8c15-3dc05a2f01ed
>> > >> osd.28 down in weight 1 up_from 30389 up_thru 30574 down_at 30691
>> > >> last_clean_interval [25513,30312) [v2:
>> > >> 192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570] [v2:
>> > >> 192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570]
>> exists
>> > >> 10578b97-e3c4-4553-a8d0-6dcc46af5db1
>> > >> osd.29 down in weight 1 up_from 30378 up_thru 30554 down_at 30688
>> > >> last_clean_interval [28595,30376) [v2:
>> > >> 192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672] [v2:
>> > >> 192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672]
>> exists
>> > >> 9698e936-8edf-4adf-92c9-a0b5202ed01a
>> > >> osd.30 down in weight 1 up_from 30449 up_thru 30531 down_at 30688
>> > >> last_clean_interval [25502,30446) [v2:
>> > >> 192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296] [v2:
>> > >> 192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296]
>> exists
>> > >> e14d2a0f-a98a-44d4-8c06-4d893f673629
>> > >> osd.31 down in weight 1 up_from 30364 up_thru 30688 down_at 30700
>> > >> last_clean_interval [25514,30361) [v2:
>> > >> 192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708] [v2:
>> > >> 192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708]
>> exists
>> > >> 541bca38-e704-483a-8cb8-39e5f69007d1
>> > >> osd.32 up in weight 1 up_from 31209 up_thru 31627 down_at 31208
>> > >> last_clean_interval [30974,31195) [v2:
>> > >> 192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997] [v2:
>> > >> 192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997]
>> > >> exists,up 9200a57e-2845-43ff-9787-8f1f3158fe90
>> > >> osd.33 down in weight 1 up_from 30354 up_thru 30688 down_at 30693
>> > >> last_clean_interval [25521,30350) [v2:
>> > >> 192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666] [v2:
>> > >> 192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666]
>> exists
>> > >> 20c55d85-cf9a-4133-a189-7fdad2318f58
>> > >> osd.34 down in weight 1 up_from 30390 up_thru 30688 down_at 30691
>> > >> last_clean_interval [25516,30314) [v2:
>> > >> 192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870] [v2:
>> > >> 192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870]
>> exists
>> > >> 77e0ef8f-c047-4f84-afb2-a8ad054e562f
>> > >> osd.35 up in weight 1 up_from 31204 up_thru 31657 down_at 31203
>> > >> last_clean_interval [30958,31195) [v2:
>> > >> 192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520] [v2:
>> > >> 192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520]
>> > >> exists,up 2d2de0cb-6d41-4957-a473-2bbe9ce227bf
>> > >> osd.36 down in weight 1 up_from 29494 up_thru 30560 down_at 30688
>> > >> last_clean_interval [25491,29492) [v2:
>> > >> 192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591] [v2:
>> > >> 192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591]
>> exists
>> > >> 26114668-68b2-458b-89c2-cbad5507ab75
>> > >>
>> > >>
>> > >>
>> > >>>
>> > >>> > On Jul 25, 2022, at 3:29 AM, Jeremy Hansen <
>> > >>> farnsworth.mcfadden@xxxxxxxxx> wrote:
>> > >>> >
>> > >>> > I transitioned some servers to a new rack and now I'm having major
>> > >>> issues
>> > >>> > with Ceph upon bringing things back up.
>> > >>> >
>> > >>> > I believe the issue may be related to the ceph nodes coming back
>> up
>> > >>> with
>> > >>> > different IPs before VLANs were set. That's just a guess because
>> I
>> > >>> can't
>> > >>> > think of any other reason this would happen.
>> > >>> >
>> > >>> > Current state:
>> > >>> >
>> > >>> > Every 2.0s: ceph -s
>> > >>> > cn01.ceph.la1.clx.corp: Mon Jul 25 10:13:05 2022
>> > >>> >
>> > >>> > cluster:
>> > >>> > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
>> > >>> > health: HEALTH_WARN
>> > >>> > 1 filesystem is degraded
>> > >>> > 2 MDSs report slow metadata IOs
>> > >>> > 2/5 mons down, quorum cn02,cn03,cn01
>> > >>> > 9 osds down
>> > >>> > 3 hosts (17 osds) down
>> > >>> > Reduced data availability: 97 pgs inactive, 9 pgs down
>> > >>> > Degraded data redundancy: 13860144/30824413 objects
>> > degraded
>> > >>> > (44.965%), 411 pgs degraded, 482 pgs undersized
>> > >>> >
>> > >>> > services:
>> > >>> > mon: 5 daemons, quorum cn02,cn03,cn01 (age 62m), out of quorum:
>> > >>> cn05,
>> > >>> > cn04
>> > >>> > mgr: cn02.arszct(active, since 5m)
>> > >>> > mds: 2/2 daemons up, 2 standby
>> > >>> > osd: 35 osds: 15 up (since 62m), 24 in (since 58m); 222
>> remapped
>> > pgs
>> > >>> >
>> > >>> > data:
>> > >>> > volumes: 1/2 healthy, 1 recovering
>> > >>> > pools: 8 pools, 545 pgs
>> > >>> > objects: 7.71M objects, 6.7 TiB
>> > >>> > usage: 15 TiB used, 39 TiB / 54 TiB avail
>> > >>> > pgs: 0.367% pgs unknown
>> > >>> > 17.431% pgs not active
>> > >>> > 13860144/30824413 objects degraded (44.965%)
>> > >>> > 1137693/30824413 objects misplaced (3.691%)
>> > >>> > 280 active+undersized+degraded
>> > >>> > 67 undersized+degraded+remapped+backfilling+peered
>> > >>> > 57 active+undersized+remapped
>> > >>> > 45 active+clean+remapped
>> > >>> > 44 active+undersized+degraded+remapped+backfilling
>> > >>> > 18 undersized+degraded+peered
>> > >>> > 10 active+undersized
>> > >>> > 9 down
>> > >>> > 7 active+clean
>> > >>> > 3 active+undersized+remapped+backfilling
>> > >>> > 2 active+undersized+degraded+remapped+backfill_wait
>> > >>> > 2 unknown
>> > >>> > 1 undersized+peered
>> > >>> >
>> > >>> > io:
>> > >>> > client: 170 B/s rd, 0 op/s rd, 0 op/s wr
>> > >>> > recovery: 168 MiB/s, 158 keys/s, 166 objects/s
>> > >>> >
>> > >>> > I have to disable and re-enable the dashboard just to use it. It
>> > >>> seems to
>> > >>> > get bogged down after a few moments.
>> > >>> >
>> > >>> > The three servers that were moved to the new rack Ceph has marked
>> as
>> > >>> > "Down", but if I do a cephadm host-check, they all seem to pass:
>> > >>> >
>> > >>> > ************************ ceph ************************
>> > >>> > --------- cn01.ceph.---------
>> > >>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> > --------- cn02.ceph.---------
>> > >>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> > --------- cn03.ceph.---------
>> > >>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> > --------- cn04.ceph.---------
>> > >>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> > --------- cn05.ceph.---------
>> > >>> > podman|docker (/usr/bin/podman) is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> > --------- cn06.ceph.---------
>> > >>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > >>> > systemctl is present
>> > >>> > lvcreate is present
>> > >>> > Unit chronyd.service is enabled and running
>> > >>> > Host looks OK
>> > >>> >
>> > >>> > It seems to be recovering with what it has left, but a large
>> amount
>> > of
>> > >>> OSDs
>> > >>> > are down. When trying to restart one of the down'd OSDs, I see a
>> > huge
>> > >>> dump.
>> > >>> >
>> > >>> > Jul 25 03:19:38 cn06.ceph
>> > >>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 0 osd.34 30689 done
>> with
>> > >>> init,
>> > >>> > starting boot process
>> > >>> > Jul 25 03:19:38 cn06.ceph
>> > >>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 1 osd.34 30689
>> start_boot
>> > >>> > Jul 25 03:20:10 cn06.ceph
>> > >>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >>> > 2022-07-25T10:20:10.655+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >>> > Jul 25 03:20:41 cn06.ceph
>> > >>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >>> > 2022-07-25T10:20:41.159+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >>> > Jul 25 03:21:11 cn06.ceph
>> > >>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > >>> > 2022-07-25T10:21:11.662+0000 7fcdfd12d700 1 osd.34 30689
>> start_boot
>> > >>> >
>> > >>> > At this point it just keeps printing start_boot, but the dashboard
>> > has
>> > >>> it
>> > >>> > marked as "in" but "down".
>> > >>> >
>> > >>> > On these three hosts that moved, there were a bunch marked as
>> "out"
>> > and
>> > >>> > "down", and some with "in" but "down".
>> > >>> >
>> > >>> > Not sure where to go next. I'm going to let the recovery continue
>> > and
>> > >>> hope
>> > >>> > that my 4x replication on these pools saves me.
>> > >>> >
>> > >>> > Not sure where to go from here. Any help is very much
>> appreciated.
>> > >>> This
>> > >>> > Ceph cluster holds all of our Cloudstack images... it would be
>> > >>> terrible to
>> > >>> > lose this data.
>> > >>> > _______________________________________________
>> > >>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> > >>>
>> > >>>
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
