Hi, I have 3 node ceph cluster (managed via proxmox). Got single node
fatal failure and replaced it. Os boots correctly, however monitor on
failed node did not start successfully; Other 2 monitors are OK, ceph
status is healthy:
ceph -s
cluster:
id: 845868a1-9902-4b61-aa06-0767cb09f1c2
health: HEALTH_OK
services:
mon: 2 daemons, quorum pxmx1,pxmx3 (age 2h)
mgr: pxmx1(active, since 56m), standbys: pxmx3
osd: 18 osds: 18 up (since 111m), 18 in (since 3h)
data:
pools: 1 pools, 256 pgs
objects: 2.12M objects, 8.1 TiB
usage: 24 TiB used, 21 TiB / 45 TiB avail
pgs: 256 active+clean
content of ceph.conf
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.60.10.1/24
fsid = 845868a1-9902-4b61-aa06-0767cb09f1c2
mon_allow_pool_delete = true
mon_host = 10.60.10.1 10.60.10.3 10.60.10.2
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.60.10.1/24
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
Monitor is failing (at least as I understand the problem) with following
logged error:
mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
whole mon log attached.
I have tried to scrap dead monitor and recreate it via proxmoxes gui,
shell and even have created content /var/lib/ceph/mon/ manually and
tried to run monitor from terminal. It starts, listens to connections on
port 3300 and 6789, but does not communicate properly with other
remaining mons.
thanks for info
Tomas Hodek
/usr/bin/ceph-mon -f --cluster ceph --id pxmx2 --setuser ceph --setgroup ceph -d -f
2022-03-28 03:39:34.971 7f7fc664d280 0 set uid:gid to 64045:64045 (ceph:ceph)
2022-03-28 03:39:34.971 7f7fc664d280 0 ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable), process ceph-mon, pid 132126
2022-03-28 03:39:34.971 7f7fc664d280 0 pidfile_write: ignore empty --pid-file
2022-03-28 03:39:34.983 7f7fc664d280 0 load: jerasure load: lrc load: isa
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option compression = kNoCompression
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option level_compaction_dynamic_level_bytes = true
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option write_buffer_size = 33554432
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option compression = kNoCompression
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option level_compaction_dynamic_level_bytes = true
2022-03-28 03:39:34.987 7f7fc664d280 0 set rocksdb option write_buffer_size = 33554432
2022-03-28 03:39:34.987 7f7fc664d280 1 rocksdb: do_open column families: [default]
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: RocksDB version: 6.1.2
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Compile date Apr 15 2020
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: DB SUMMARY
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: CURRENT file: CURRENT
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: IDENTITY file: IDENTITY
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: MANIFEST file: MANIFEST-000057 size: 569 Bytes
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-pxmx2/store.db dir, Total Num: 1, files: 000068.sst
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-pxmx2/store.db: 000066.log size: 7391017 ;
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.error_if_exists: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.create_if_missing: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.paranoid_checks: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.env: 0x565340c4f640
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.info_log: 0x56534191f6e0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_file_opening_threads: 16
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.statistics: (nil)
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.use_fsync: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_log_file_size: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_manifest_file_size: 1073741824
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.log_file_time_to_roll: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.keep_log_file_num: 1000
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.recycle_log_file_num: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.allow_fallocate: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.allow_mmap_reads: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.allow_mmap_writes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.use_direct_reads: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.create_missing_column_families: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.db_log_dir:
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-pxmx2/store.db
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.table_cache_numshardbits: 6
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_subcompactions: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_background_flushes: -1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.WAL_ttl_seconds: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.WAL_size_limit_MB: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.manifest_preallocation_size: 4194304
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.is_fd_close_on_exec: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.advise_random_on_open: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.db_write_buffer_size: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.write_buffer_manager: 0x5653418ffb30
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.access_hint_on_compaction_start: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.use_adaptive_mutex: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.rate_limiter: (nil)
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.wal_recovery_mode: 2
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.enable_thread_tracking: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.enable_pipelined_write: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.write_thread_max_yield_usec: 100
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.row_cache: None
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.wal_filter: None
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.avoid_flush_during_recovery: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.allow_ingest_behind: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.preserve_deletes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.two_write_queues: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.manual_wal_flush: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.atomic_flush: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_background_jobs: 2
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_background_compactions: -1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.delayed_write_rate : 16777216
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_total_wal_size: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.stats_dump_period_sec: 600
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.stats_persist_period_sec: 600
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.stats_history_buffer_size: 1048576
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_open_files: -1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bytes_per_sync: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.wal_bytes_per_sync: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_readahead_size: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Compression algorithms supported:
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kZSTDNotFinalCompression supported: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kZSTD supported: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kXpressCompression supported: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kLZ4HCCompression supported: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kLZ4Compression supported: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kBZip2Compression supported: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kZlibCompression supported: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: kSnappyCompression supported: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Fast CRC32 supported: Supported on x86
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-pxmx2/store.db/MANIFEST-000057
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]:
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.merge_operator:
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_filter: None
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_filter_factory: None
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.memtable_factory: SkipListFactory
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.table_factory: BlockBasedTable
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56534185eca8)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x565341874810
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.write_buffer_size: 33554432
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_write_buffer_number: 2
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression: NoCompression
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression: Disabled
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.prefix_extractor: nullptr
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.num_levels: 7
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.level: 32767
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bottommost_compression_opts.enabled: false
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.window_bits: -14
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.level: 32767
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.strategy: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compression_opts.enabled: false
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.level0_stop_writes_trigger: 36
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.target_file_size_base: 67108864
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.target_file_size_multiplier: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_compaction_bytes: 1677721600
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.arena_block_size: 4194304
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.disable_auto_compactions: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.table_properties_collectors:
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.inplace_update_support: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.inplace_update_num_locks: 10000
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.memtable_whole_key_filtering: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.memtable_huge_page_size: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.bloom_locality: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.max_successive_merges: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.optimize_filters_for_hits: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.paranoid_file_checks: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.force_consistency_checks: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.report_bg_io_stats: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: Options.ttl: 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-pxmx2/store.db/MANIFEST-000057 succeeded,manifest_file_number is 57, next_file_number is 70, last_sequence is 27043, log_number is 66,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 66
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431574992537, "job": 1, "event": "recovery_started", "log_files": [66]}
2022-03-28 03:39:34.987 7f7fc664d280 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #66 mode 2
2022-03-28 03:39:35.079 7f7fc664d280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431575083373, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 70, "file_size": 7312355, "table_properties": {"data_size": 7297903, "index_size": 12284, "filter_size": 1349, "raw_key_size": 11867, "raw_average_key_size": 22, "raw_value_size": 7281514, "raw_average_value_size": 14084, "num_data_blocks": 334, "num_entries": 517, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2022-03-28 03:39:35.079 7f7fc664d280 4 rocksdb: [db/version_set.cc:3036] Creating manifest 71
2022-03-28 03:39:35.095 7f7fc664d280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431575098904, "job": 1, "event": "recovery_finished"}
2022-03-28 03:39:35.115 7f7fc664d280 4 rocksdb: DB pointer 0x565342616000
2022-03-28 03:39:35.115 7f7fbb95e700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
2022-03-28 03:39:35.115 7f7fbb95e700 4 rocksdb: [db/db_impl.cc:778]
** DB Stats **
Uptime(secs): 0.1 total, 0.1 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 1/0 6.97 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.5 0.07 0.00 1 0.072 0 0
L6 1/0 33.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 2/0 40.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.5 0.07 0.00 1 0.072 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.5 0.07 0.00 1 0.072 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 97.5 0.07 0.00 1 0.072 0 0
Uptime(secs): 0.1 total, 0.1 interval
Flush(GB): cumulative 0.007, interval 0.007
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.01 GB write, 54.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds
Interval compaction: 0.01 GB write, 54.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 1/0 6.97 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.5 0.07 0.00 1 0.072 0 0
L6 1/0 33.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 2/0 40.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.5 0.07 0.00 1 0.072 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 97.5 0.07 0.00 1 0.072 0 0
Uptime(secs): 0.1 total, 0.0 interval
Flush(GB): cumulative 0.007, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.01 GB write, 54.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
2022-03-28 03:39:35.115 7f7fc664d280 0 mon.pxmx2 does not exist in monmap, will attempt to join an existing cluster
2022-03-28 03:39:35.115 7f7fc664d280 0 using public_addr v2:10.60.10.2:0/0 -> [v2:10.60.10.2:3300/0,v1:10.60.10.2:6789/0]
2022-03-28 03:39:35.115 7f7fc664d280 0 starting mon.pxmx2 rank -1 at public addrs [v2:10.60.10.2:3300/0,v1:10.60.10.2:6789/0] at bind addrs [v2:10.60.10.2:3300/0,v1:10.60.10.2:6789/0] mon_data /var/lib/ceph/mon/ceph-pxmx2 fsid 845868a1-9902-4b61-aa06-0767cb09f1c2
2022-03-28 03:39:35.119 7f7fc664d280 1 mon.pxmx2@-1(???) e4 preinit fsid 845868a1-9902-4b61-aa06-0767cb09f1c2
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).mds e91 new map
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).mds e91 print_map
e91
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: -1
No filesystems configured
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).osd e6252 crush map has features 288514051259236352, adjusting msgr requires
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).osd e6252 crush map has features 288514051259236352, adjusting msgr requires
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).osd e6252 crush map has features 3314933000852226048, adjusting msgr requires
2022-03-28 03:39:35.119 7f7fc664d280 0 mon.pxmx2@-1(???).osd e6252 crush map has features 288514051259236352, adjusting msgr requires
2022-03-28 03:39:35.119 7f7fc664d280 1 mon.pxmx2@-1(???).paxosservice(auth 30002..30195) refresh upgraded, format 0 -> 3
2022-03-28 03:39:35.255 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:36.051 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:36.143 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:37.543 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:39.243 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:39.251 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:39.387 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:39.395 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:40.579 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:42.495 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:42.651 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:42.663 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:43.255 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:44.247 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:44.255 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:45.291 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:45.943 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:47.251 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:48.243 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:48.251 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:49.079 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:49.231 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:51.243 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:51.251 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:52.279 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:52.379 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:52.391 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:52.563 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:55.247 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:55.651 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:55.843 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:55.855 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:56.039 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.215 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.247 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.263 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.267 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.947 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:58.955 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:39:59.259 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:00.275 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:01.571 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:03.251 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:03.255 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:04.715 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:07.255 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:07.875 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:07.951 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:08.247 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:08.259 7f7fbd161700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:10.639 7f7fb995a700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #75. Immutable memtables: 0.
2022-03-28 03:40:10.639 7f7fc396e700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.641008) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
2022-03-28 03:40:10.639 7f7fc396e700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 3] Flushing memtable with next log file: 75
2022-03-28 03:40:10.639 7f7fc396e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431610641105, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 561, "num_deletes": 254, "total_data_size": 2404706, "memory_usage": 2415592, "flush_reason": "Manual Compaction"}
2022-03-28 03:40:10.639 7f7fc396e700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 3] Level-0 flush table #76: started
2022-03-28 03:40:10.667 7f7fc396e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431610671133, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 76, "file_size": 2404463, "table_properties": {"data_size": 2398348, "index_size": 4071, "filter_size": 1221, "raw_key_size": 9806, "raw_average_key_size": 22, "raw_value_size": 2389269, "raw_average_value_size": 5479, "num_data_blocks": 114, "num_entries": 436, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2022-03-28 03:40:10.667 7f7fc396e700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 3] Level-0 flush table #76: 2404463 bytes OK
2022-03-28 03:40:10.683 7f7fc396e700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.678173) [db/memtable_list.cc:380] [default] Level-0 commit table #76 started
2022-03-28 03:40:10.683 7f7fc396e700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.685749) [db/memtable_list.cc:428] [default] Level-0 commit table #76: memtable #1 done
2022-03-28 03:40:10.683 7f7fc396e700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.685777) EVENT_LOG_v1 {"time_micros": 1648431610685764, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0}
2022-03-28 03:40:10.683 7f7fc396e700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.685813) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[2 0 0 0 0 0 1] max score 0.50
2022-03-28 03:40:10.683 7f7fc396e700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 3] Try to delete WAL files size 2400585, prev total WAL file size 2400667, number of live WAL files 2.
2022-03-28 03:40:10.683 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:10.683 7f7fc416f700 4 rocksdb: (Original Log Time 2022/03/28-03:40:10.686165) [db/db_impl_compaction_flush.cc:2374] [default] Manual compaction from level-0 to level-6 from 'paxos .. 'paxos; will stop at (end)
2022-03-28 03:40:10.683 7f7fc416f700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 4] Compacting 2@0 + 1@6 files to L6, score -1.00
2022-03-28 03:40:10.683 7f7fc416f700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 3 Base level 0, inputs: [76(2348KB) 70(7140KB)], [68(33MB)]
2022-03-28 03:40:10.683 7f7fc416f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431610686232, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [76, 70], "files_L6": [68], "score": -1, "input_data_size": 44621735}
2022-03-28 03:40:12.227 7f7fc416f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 4] Generated table #77: 3891 keys, 40497655 bytes
2022-03-28 03:40:12.227 7f7fc416f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431612229646, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 77, "file_size": 40497655, "table_properties": {"data_size": 40406066, "index_size": 80971, "filter_size": 9797, "raw_key_size": 84677, "raw_average_key_size": 21, "raw_value_size": 40293334, "raw_average_value_size": 10355, "num_data_blocks": 2234, "num_entries": 3891, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2022-03-28 03:40:12.235 7f7fc416f700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 4] Compacted 2@0 + 1@6 files to L6 => 40497655 bytes
2022-03-28 03:40:12.243 7f7fc416f700 4 rocksdb: (Original Log Time 2022/03/28-03:40:12.244865) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 28.9 rd, 26.2 wr, level 6, files in(2, 1) out(1) MB in(9.3, 33.3) out(38.6), read-write-amplify(8.8) write-amplify(4.2) OK, records in: 4425, records dropped: 534 output_compression: NoCompression
2022-03-28 03:40:12.243 7f7fc416f700 4 rocksdb: (Original Log Time 2022/03/28-03:40:12.244901) EVENT_LOG_v1 {"time_micros": 1648431612244884, "job": 4, "event": "compaction_finished", "compaction_time_micros": 1543480, "compaction_time_cpu_micros": 116224, "output_level": 6, "num_output_files": 1, "total_output_size": 40497655, "num_input_records": 4425, "num_output_records": 3891, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]}
2022-03-28 03:40:12.243 7f7fc416f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431612245805, "job": 4, "event": "table_file_deletion", "file_number": 76}
2022-03-28 03:40:12.243 7f7fc416f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431612247827, "job": 4, "event": "table_file_deletion", "file_number": 70}
2022-03-28 03:40:12.247 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:12.255 7f7fc416f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1648431612257588, "job": 4, "event": "table_file_deletion", "file_number": 68}
2022-03-28 03:40:12.255 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:12.255 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:12.255 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:12.255 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:12.255 7f7fb995a700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2022-03-28 03:40:12.259 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:12.275 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:13.979 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:16.259 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:17.443 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:19.075 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:19.079 7f7fc296c700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:19.883 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:20.255 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:20.263 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:20.271 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:20.623 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
2022-03-28 03:40:20.699 7f7fc216b700 1 mon.pxmx2@-1(probing) e4 handle_auth_request failed to assign global_id
^C2022-03-28 03:40:20.855 7f7fc4970700 -1 received signal: Interrupt, si_code : 128, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 0, si_uid : 0, si_addr0, si_status0
2022-03-28 03:40:20.855 7f7fc4970700 -1 mon.pxmx2@-1(probing) e4 *** Got Signal Interrupt ***
2022-03-28 03:40:20.855 7f7fc4970700 1 mon.pxmx2@-1(probing) e4 shutdown
2022-03-28 03:40:20.859 7f7fc664d280 1 rocksdb: close waiting for compaction thread to stop
2022-03-28 03:40:20.859 7f7fc664d280 1 rocksdb: close compaction thread to stopped
2022-03-28 03:40:20.859 7f7fc664d280 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work
2022-03-28 03:40:20.859 7f7fc664d280 4 rocksdb: [db/db_impl.cc:563] Shutdown complet
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
