Re: mons fail as soon as I attempt to mount

[Jeremy Hansen <jeremy@xxxxxxxxxx>]

Thank you. I was able to gather more logs from one of the failing mon processes:

cluster 2021-11-16T21:23:58.114701+0000 mgr.cn02.arszct (mgr.8444129) 66657 : cluster [DBG] pgmap v66699: 289 pgs: 289 active+clean; 4.4 TiB data, 17 TiB used, 108 TiB / 126 TiB avail; 50 KiB/s rd, 123 KiB/s wr, 74 op/s
audit 2021-11-16T21:23:59.322056+0000 mon.cn05 (mon.0) 19 : audit [INF] from='mgr.8444129 ' entity='mgr.cn02.arszct'
cluster 2021-11-16T21:24:00.116045+0000 mgr.cn02.arszct (mgr.8444129) 66658 : cluster [DBG] pgmap v66700: 289 pgs: 289 active+clean; 4.4 TiB data, 17 TiB used, 108 TiB / 126 TiB avail; 11 KiB/s rd, 130 KiB/s wr, 30 op/s
debug 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0)
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at
*** Caught signal (Aborted) **
in thread 7f1d716db700 thread_name:ms_dispatch
ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8]
8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045]
9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6]
10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329]
11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258]
12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc]
15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71]
17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a]
18: clone()
debug 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) **
in thread 7f1d716db700 thread_name:ms_dispatch

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8]
8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045]
9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6]
10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329]
11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258]
12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc]
15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71]
17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a]
18: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
debug -734> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command assert hook 0x5607840e8530
debug -733> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command abort hook 0x5607840e8530
debug -732> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command leak_some_memory hook 0x5607840e8530
debug -731> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_dump hook 0x5607840e8530
debug -730> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 1 hook 0x5607840e8530
debug -729> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf dump hook 0x5607840e8530
debug -728> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_schema hook 0x5607840e8530
debug -727> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram dump hook 0x5607840e8530
debug -726> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 2 hook 0x5607840e8530
debug -725> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf schema hook 0x5607840e8530
debug -724> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram schema hook 0x5607840e8530
debug -723> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf reset hook 0x5607840e8530
debug -722> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config show hook 0x5607840e8530
debug -721> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config help hook 0x5607840e8530
debug -720> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config set hook 0x5607840e8530
debug -719> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config unset hook 0x5607840e8530
debug -718> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config get hook 0x5607840e8530
debug -717> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff hook 0x5607840e8530
debug -716> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff get hook 0x5607840e8530
debug -715> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command injectargs hook 0x5607840e8530
debug -714> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log flush hook 0x5607840e8530
debug -713> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log dump hook 0x5607840e8530
debug -712> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log reopen hook 0x5607840e8530
debug -711> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_mempools hook 0x560784188328
debug -710> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 set uid:gid to 167:167 (ceph:ceph)
debug -709> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 8
debug -708> 2021-11-16T21:23:28.656+0000 7f1d880be700 0 pidfile_write: ignore empty --pid-file
debug -707> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) init /var/run/ceph/ceph-mon.cn03.asok
debug -706> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) bind_and_listen /var/run/ceph/ceph-mon.cn03.asok
debug -705> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command 0 hook 0x5607840ec108
debug -704> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command version hook 0x5607840ec108

debug -703> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command git_version hook 0x5607840ec108
debug -702> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command help hook 0x5607840e8230
debug -701> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command get_command_descriptions hook 0x5607840e81e0
debug -700> 2021-11-16T21:23:28.656+0000 7f1d78f67700 5 asok(0x56078419c000) entry start
debug -699> 2021-11-16T21:23:28.660+0000 7f1d880be700 0 load: jerasure load: lrc load: isa
debug -698> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -697> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432
debug -696> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression
debug -695> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -694> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432
debug -693> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression
debug -692> 2021-11-16T21:23:28.661+0000 7f1d880be700 5 rocksdb: verify_sharding column families from rocksdb: [default]
debug -691> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: RocksDB version: 6.8.1

debug -690> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@

debug -689> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compile date May 13 2021
debug -688> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: DB SUMMARY

debug -687> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: CURRENT file: CURRENT

debug -686> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: IDENTITY file: IDENTITY

debug -685> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: MANIFEST file: MANIFEST-531948 size: 302 Bytes

debug -684> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-cn03/store.db dir, Total Num: 3, files: 531944.sst 531945.sst 531947.sst

debug -683> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-cn03/store.db: 531949.log size: 214103 ;

debug -682> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.error_if_exists: 0
debug -681> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_if_missing: 0
debug -680> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_checks: 1
debug -679> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.env: 0x5607837a11c0
debug -678> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.fs: Posix File System
debug -677> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.info_log: 0x5607841d78c0
debug -676> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_file_opening_threads: 16
debug -675> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.statistics: (nil)
debug -674> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_fsync: 0
debug -673> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_log_file_size: 0
debug -672> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_manifest_file_size: 1073741824
debug -671> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_file_time_to_roll: 0
debug -670> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.keep_log_file_num: 1000
debug -669> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.recycle_log_file_num: 0
debug -668> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_fallocate: 1
debug -667> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_reads: 0
debug -666> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_writes: 0
debug -665> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_reads: 0
debug -664> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
debug -663> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_missing_column_families: 0
debug -662> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_log_dir:
debug -661> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-cn03/store.db
debug -660> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_cache_numshardbits: 6
debug -659> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_subcompactions: 1
debug -658> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_flushes: -1
debug -657> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_ttl_seconds: 0
debug -656> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_size_limit_MB: 0
debug -655> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576
debug -654> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manifest_preallocation_size: 4194304
debug -653> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.is_fd_close_on_exec: 1
debug -652> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.advise_random_on_open: 1
debug -651> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_write_buffer_size: 0
debug -650> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_manager: 0x5607841ddf80
debug -649> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.access_hint_on_compaction_start: 1
debug -648> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
debug -647> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.random_access_max_buffer_size: 1048576
debug -646> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_adaptive_mutex: 0
debug -645> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limiter: (nil)
debug -644> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
debug -643> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_recovery_mode: 2
debug -642> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_thread_tracking: 0
debug
-641> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_pipelined_write: 0
debug -640> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.unordered_write: 0
debug -639> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_concurrent_memtable_write: 1
debug -638> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
debug -637> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_max_yield_usec: 100
debug -636> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_slow_yield_usec: 3
debug -635> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.row_cache: None
debug -634> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_filter: None
debug -633> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_recovery: 0
debug -632> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_ingest_behind: 0
debug -631> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.preserve_deletes: 0
debug -630> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.two_write_queues: 0
debug -629> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manual_wal_flush: 0
debug -628> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.atomic_flush: 0
debug -627> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0
debug -626> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.persist_stats_to_disk: 0
debug -625> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_dbid_to_manifest: 0
debug -624> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_readahead_size: 0
debug -623> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_checksum_func: Unknown
debug -622> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_jobs: 2
debug -621> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_compactions: -1
debug -620> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_shutdown: 0
debug -619> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
debug -618> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delayed_write_rate : 16777216
debug -617> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_total_wal_size: 0
debug -616> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
debug -615> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_dump_period_sec: 600
debug -614> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_persist_period_sec: 600
debug -613> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_history_buffer_size: 1048576
debug -612> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_open_files: -1
debug -611> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bytes_per_sync: 0
debug -610> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_bytes_per_sync: 0
debug -609> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.strict_bytes_per_sync: 0
debug -608> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_readahead_size: 0
debug -607> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compression algorithms supported:
debug -606> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTDNotFinalCompression supported: 0
debug -605> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTD supported: 0
debug -604> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kXpressCompression supported: 0
debug -603> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4HCCompression supported: 1
debug -602> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4Compression supported: 1
debug -601> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kBZip2Compression supported: 0
debug -600> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZlibCompression supported: 1
debug -599> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kSnappyCompression supported: 1
debug -598> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Fast CRC32 supported: Supported on x86
debug -597> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948

debug -596> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]:

debug -595> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
debug -594> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.merge_operator:
debug -593> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter: None
debug -592> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter_factory: None
debug
-591> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_factory: SkipListFactory
debug -590> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_factory: BlockBasedTable
debug -589> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607840ec128)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x560784125010
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0

debug -588> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_size: 33554432
debug -587> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number: 2
debug -586> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression: NoCompression
debug -585> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression: Disabled
debug -584> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.prefix_extractor: nullptr
debug -583> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
debug -582> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.num_levels: 7
debug -581> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
debug -580> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
debug -579> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0
debug -578> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
debug -577> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.level: 32767
debug -576> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
debug -575> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
debug -574> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
debug -573> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.enabled: false
debug -572> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.window_bits: -14
debug -571> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.level: 32767
debug -570> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.strategy: 0
debug -569> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
debug -568> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
debug -567> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.enabled: false
debug -566> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
debug -565> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
debug -564> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_stop_writes_trigger: 36
debug -563> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_base: 67108864
debug -562> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_multiplier: 1
debug -561> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_base: 268435456
debug -560> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
debug -559> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
debug -558> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
debug -557> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
debug -556> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
debug -555> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
debug -554> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
debug -553> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
debug -552> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
debug -551> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
debug -550> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_compaction_bytes: 1677721600
debug -549> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.arena_block_size: 4194304
debug -548> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
debug -547> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
debug -546> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
debug -545> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.disable_auto_compactions: 0
debug
-544> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
debug -543> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
debug -542> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
debug -541> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
debug -540> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
debug -539> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
debug -538> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
debug -537> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
debug -536> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
debug -535> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
debug -534> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_properties_collectors:
debug -533> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_support: 0
debug -532> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_num_locks: 10000
debug -531> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
debug -530> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_whole_key_filtering: 0
debug -529> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_huge_page_size: 0
debug -528> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bloom_locality: 0
debug -527> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_successive_merges: 0
debug -526> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.optimize_filters_for_hits: 0
debug -525> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_file_checks: 0
debug -524> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.force_consistency_checks: 0
debug -523> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.report_bg_io_stats: 0
debug -522> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.ttl: 2592000
debug -521> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.periodic_compaction_seconds: 0
debug -520> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 succeeded,manifest_file_number is 531948, next_file_number is 531950, last_sequence is 201837822, log_number is 531943,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

debug -519> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 531943

debug -518> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808663528, "job": 1, "event": "recovery_started", "log_files": [531949]}
debug -517> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #531949 mode 2
debug -516> 2021-11-16T21:23:28.663+0000 7f1d880be700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
debug -515> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808697864, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 531950, "file_size": 151829, "table_properties": {"data_size": 150555, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": 759, "raw_average_key_size": 25, "raw_value_size": 149707, "raw_average_value_size": 4990, "num_data_blocks": 8, "num_entries": 30, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097808, "oldest_key_time": 3, "file_creation_time": 0}}
debug -514> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: [version_set.cc:3826] Creating manifest 531951

debug -513> 2021-11-16T21:23:28.755+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808756165, "job": 1, "event": "recovery_finished"}
debug -512> 2021-11-16T21:23:28.797+0000 7f1d880be700 4 rocksdb: DB pointer 0x5607841e9800
debug -511> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS -------
debug -510> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:851]
** DB Stats **
Uptime(secs): 0.1 total, 0.1 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0

** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Uptime(secs): 0.1 total, 0.1 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0

** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Uptime(secs): 0.1 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

debug -509> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -508> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -507> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -506> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: none
debug -505> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -504> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -503> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -502> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -501> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -500> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug
-499> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -498> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -497> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -496> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -495> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -494> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -493> 2021-11-16T21:23:28.797+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -492> 2021-11-16T21:23:28.798+0000 7f1d880be700 0 starting mon.cn03 rank 2 at public addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] at bind addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon_data /var/lib/ceph/mon/ceph-cn03 fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
debug -491> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -490> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -489> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -488> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: none
debug -487> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -486> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -485> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -484> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -483> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -482> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -481> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -480> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -479> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -478> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -477> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -476> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -475> 2021-11-16T21:23:28.798+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -474> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx
debug -473> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx
debug -472> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
debug -471> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
debug -470> 2021-11-16T21:23:28.799+0000 7f1d880be700 1 mon.cn03@-1(???) e69 preinit fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
debug -469> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 new map
debug -468> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 print_map
e432
enable_multiple, ever_enabled_multiple: 1,1
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 2

Filesystem 'coldlogix' (2)
fs_name coldlogix
epoch 402
flags 12
created 2021-06-27T06:32:25.086015+0000
modified 2021-11-15T08:30:22.670221+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 17247
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=8444199}
failed
damaged
stopped
data_pools [12]
metadata_pool 11
inline_data disabled
balancer
standby_count_wanted 1
[mds.coldlogix.cn01.dirzwe{0:8444199} state up:active seq 77 join_fscid=2 addr [v2:192.168.30.11:6850/3427514521,v1:192.168.30.11:6851/3427514521]]

Filesystem 'btc' (5)
fs_name btc
epoch 423
flags 12
created 2021-11-15T08:49:14.520956+0000
modified 2021-11-15T08:49:25.936407+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 17308
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=8534156}
failed
damaged
stopped
data_pools [19]
metadata_pool 18
inline_data disabled
balancer
standby_count_wanted 1
[mds.btc.cn04.pyuaxe{0:8534156} state up:active seq 5 join_fscid=5 addr [v2:192.168.30.14:6800/4253241567,v1:192.168.30.14:6801/4253241567]]

Standby daemons:

[mds.btc.cn06.hqmwsx{-1:8534166} state up:standby seq 3 join_fscid=5 addr [v2:192.168.30.16:6848/446743436,v1:192.168.30.16:6849/446743436]]
[mds.coldlogix.cn03.lyxzhp{-1:8604119} state up:standby seq 4 join_fscid=2 addr [v2:192.168.30.13:6800/2645334932,v1:192.168.30.13:6801/2645334932]]

debug -467> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 3314933000854323200, adjusting msgr requires
debug -466> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -465> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -464> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -463> 2021-11-16T21:23:28.800+0000 7f1d880be700 1 mon.cn03@-1(???).paxosservice(auth 10008..10244) refresh upgraded, format 0 -> 3
debug -462> 2021-11-16T21:23:28.800+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e0 loading version 500
debug -461> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 active server: [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106](8444129)
debug -460> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 mkfs or daemon transitioned to available, loading commands
debug -459> 2021-11-16T21:23:28.802+0000 7f1d880be700 4 set_mon_vals no callback set
debug -458> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals auth_allow_insecure_global_id_reclaim = false
debug -457> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949
debug -456> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals mon_allow_pool_delete = true
debug -455> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals rbd_move_to_trash_on_remove = true
debug -454> 2021-11-16T21:23:28.803+0000 7f1d880be700 4 set_mon_vals no callback set
debug -453> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 aut
h: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -452> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command compact hook 0x5607840e89a0
debug -451> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command smart hook 0x5607840e89a0
debug -450> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command mon_status hook 0x5607840e89a0
debug -449> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command heap hook 0x5607840e89a0
debug -448> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores dump hook 0x5607840e89a0
debug -447> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores reset hook 0x5607840e89a0
debug -446> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sync_force hook 0x5607840e89a0
debug -445> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hint hook 0x5607840e89a0
debug -444> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hintv hook 0x5607840e89a0
debug -443> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum enter hook 0x5607840e89a0
debug -442> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum exit hook 0x5607840e89a0
debug -441> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command ops hook 0x5607840e89a0
debug -440> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sessions hook 0x5607840e89a0
debug -439> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_historic_ops hook 0x5607840e89a0
debug -438> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -437> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -436> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -435> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: none
debug -434> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -433> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -432> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -431> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -430> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -429> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -428> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -427> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -426> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -425> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -424> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -423> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -422> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -421> 2021-11-16T21:23:28.804+0000 7f1d880be700 2 mon.cn03@-1(???) e69 init
debug -420> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Got map version 500
debug -419> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -418> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -417> 2021-11-16T21:23:28.805+0000 7f1d726dd700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d726dd700' had timed out after 0.000000000s
debug -416> 2021-11-16T21:23:28.806+0000 7f1d880be700 0 mon.cn03@-1(probing) e69 my rank is now 2 (was -1)
debug -415> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure stats_period=5
debug -414> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure updated stats threshold: 5
debug -413> 2021-11-16T21:23:28.807+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection
debug -412> 2021-11-16T21:23:29.213+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection
debug -411> 2021-11-16T21:23:29.214+0000 7f1d716db700 0 log_channel(cluster) log [INF] : mon.cn03 calling monitor election
debug -410> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client _send_to_mon log to self
debug -409> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
debug -408> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client will send 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election
debug -407> 2021-11-16T21:23:29.214+0000 7f1d716db700 5 paxos.2).electionLogic(0) start -- can i be leader?
debug -406> 2021-11-16T21:23:29.214+0000 7f1d716db700 1 paxos.2).electionLogic(5412) init, last seen epoch 5412
debug -405> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -404> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248762+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -403> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248771+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -402> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.1
debug -401> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 paxos.2).electionLogic(5413) defer to 1
debug -400> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3
debug -399> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 paxos.2).electionLogic(5413) no, we already acked 1
debug -398> 2021-11-16T21:23:30.619+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1
debug -397> 2021-11-16T21:23:33.077+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -396> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.4
debug -395> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 paxos.2).electionLogic(5413) ignoring old propose
debug -394> 2021-11-16T21:23:34.411+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3
debug -393> 2021-11-16T21:23:34.440+0000 7f1d716db700 5 paxos.2).electionLogic(5415) no, we already acked 1
debug -392> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -391> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5415) handle_propose from mon.0
debug -390> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 paxos.2).electionLogic(5415) ignoring old propose
debug -389> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) election period ended
debug -388> 2021-11-16T21:23:35.249+0000 7f1d73ee070
0 5 paxos.2).electionLogic(5415) start -- can i be leader?
debug -387> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 1 paxos.2).electionLogic(5415) init, last seen epoch 5415, mid-election, bumping
debug -386> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_ack from mon.3
debug -385> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) so far i have { mon.2: features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) }
debug -384> 2021-11-16T21:23:35.392+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_propose from mon.0
debug -383> 2021-11-16T21:23:35.424+0000 7f1d716db700 5 paxos.2).electionLogic(5419) defer to 0
debug -382> 2021-11-16T21:23:36.621+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1
debug -381> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256876+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -380> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256899+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -379> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256913+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -378> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256925+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -377> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257153+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -376> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257169+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -375> 2021-11-16T21:23:38.342+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5419) handle_victory from mon.0 quorum_features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging])
debug -374> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_quorum_to_compatset_features
debug -373> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_monmap_to_compatset_features
debug -372> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382930+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -371> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382942+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -370> 2021-11-16T21:23:38.450+0000 7f1d716db700 4 set_mon_vals no callback set
debug -369> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -368> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -367> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -366> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451523+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -365> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451536+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -364> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451571+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -363> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451596+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -362> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451620+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -361> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451641+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -360> 2021-11-16T21:23:38.452+0000 7f1d736df700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d736df700' had timed out after 0.000000000s
debug -359> 2021-11-16T21:23:38.538+0000 7f1d706d9700 1 mon.cn03@2(peon) e69 handle_auth_request failed to assign global_id
debug -358> 2021-11-16T21:23:38.541+0000 7f1d716db700 4 set_mon_vals no callback set
debug -357> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -356> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -355> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -354> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557605) is_readable = 1 - now=2021-11-16T21:23:38.542983+0000 lease_expire=2021-11-16T21:23:43.507566+0000 has v0 lc 19557605
debug -353> 2021-11-16T21:23:38.635+0000 7f1d716db700 4 set_mon_vals no callback set
debug -352> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -351> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -350> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -349> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557606) is_readable = 1 - now=2021-11-16T21:23:38.636621+0000 lease_expire=2021-11-16T21:23:43.603066+0000 has v0 lc 19557606
debug -348> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client handle_log_ack log(last 1) v1
debug -347> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client logged 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election
debug -346> 2021-11-16T21:23:38.725+0000 7f1d716db700 4 set_mon_vals no callback set
debug -345> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -344> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -343> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -342> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557607) is_readable = 1 - now=2021-11-16T21:23:38.726705+0000 lease_expire=2021-11-16T21:23:43.690619+0000 has v0 lc 19557607
debug -341> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783222+0000 lease_expire=1970-01-01T00:00:00.000000+0000 h
as v0 lc 19557607
debug -340> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783242+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607
debug -339> 2021-11-16T21:23:38.841+0000 7f1d716db700 4 set_mon_vals no callback set
debug -338> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -337> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -336> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -335> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.843331+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -334> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).elector(5420) begin_peer_ping against 3
debug -333> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944355+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -332> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853ebef0 0x5607858deb60 auth_reply(proto 2 0 (0) Success) v1
debug -331> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944811+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -330> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858df380 auth_reply(proto 2 0 (0) Success) v1
debug -329> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984545+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -328> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x5607858de820 auth_reply(proto 2 0 (0) Success) v1
debug -327> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984931+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -326> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851533b0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1
debug -325> 2021-11-16T21:23:39.026+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.026942+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -324> 2021-11-16T21:23:39.026+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x5607858de9c0 auth_reply(proto 2 0 (0) Success) v1
debug -323> 2021-11-16T21:23:39.027+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.027319+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -322> 2021-11-16T21:23:39.027+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x5607858deea0 auth_reply(proto 2 0 (0) Success) v1
debug -321> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067500+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -320> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x5607858df520 auth_reply(proto 2 0 (0) Success) v1
debug -319> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067791+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -318> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607858de680 auth_reply(proto 2 0 (0) Success) v1
debug -317> 2021-11-16T21:23:39.105+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.105918+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -316> 2021-11-16T21:23:39.105+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858dc9c0 auth_reply(proto 2 0 (0) Success) v1
debug -315> 2021-11-16T21:23:39.106+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.106361+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -314> 2021-11-16T21:23:39.106+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607858dc820 auth_reply(proto 2 0 (0) Success) v1
debug -313> 2021-11-16T21:23:39.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.217724+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -312> 2021-11-16T21:23:39.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e4b0 0x560785262b60 auth_reply(proto 2 0 (0) Success) v1
debug -311> 2021-11-16T21:23:39.218+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.218155+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -310> 2021-11-16T21:23:39.218+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x5607852636c0 auth_reply(proto 2 0 (0) Success) v1
debug -309> 2021-11-16T21:23:39.335+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336070+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -308> 2021-11-16T21:23:39.335+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858eb6c0 auth_reply(proto 2 0 (0) Success) v1
debug -307> 2021-11-16T21:23:39.336+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336443+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -306> 2021-11-16T21:23:39.336+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858eab60 auth_reply(proto 2 0 (0) Success) v1
debug -305> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.388745+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -304> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x560785916680 auth_reply(proto 2 0 (0) Success) v1
debug -303> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.389042+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -302> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x560785916340 auth_reply(proto 2 0 (0) Success) v1
debug -301> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415381+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -300> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb770 0x5607858f2b60 auth_reply(proto 2 0 (0) Success) v1
debug -299> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415676+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -298> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607858ec9c0 auth_reply(proto 2 0 (0) Success) v1
debug -297> 2021-11
-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441170+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -296> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785912d00 auth_reply(proto 2 0 (0) Success) v1
debug -295> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441498+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -294> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785912680 auth_reply(proto 2 0 (0) Success) v1
debug -293> 2021-11-16T21:23:39.547+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.547657+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -292> 2021-11-16T21:23:39.547+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x5607859124e0 auth_reply(proto 2 0 (0) Success) v1
debug -291> 2021-11-16T21:23:39.548+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.548146+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -290> 2021-11-16T21:23:39.548+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607852621a0 auth_reply(proto 2 0 (0) Success) v1
debug -289> 2021-11-16T21:23:39.717+0000 7f1d716db700 4 set_mon_vals no callback set
debug -288> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -287> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -286> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -285> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.718923+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -284> 2021-11-16T21:23:39.796+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.796980+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -283> 2021-11-16T21:23:39.796+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x560785263860 auth_reply(proto 2 0 (0) Success) v1
debug -282> 2021-11-16T21:23:39.797+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.797502+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -281> 2021-11-16T21:23:39.797+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858b36c0 auth_reply(proto 2 0 (0) Success) v1
debug -280> 2021-11-16T21:23:39.853+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854122+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -279> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e000 0x56078525eb60 auth_reply(proto 2 0 (0) Success) v1
debug -278> 2021-11-16T21:23:39.854+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854658+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -277> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858eb520 auth_reply(proto 2 0 (0) Success) v1
debug -276> 2021-11-16T21:23:39.999+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.999601+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -275> 2021-11-16T21:23:39.999+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x560785263520 auth_reply(proto 2 0 (0) Success) v1
debug -274> 2021-11-16T21:23:40.000+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.000151+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -273> 2021-11-16T21:23:40.000+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x5607858b3040 auth_reply(proto 2 0 (0) Success) v1
debug -272> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078366+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -271> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607852631e0 auth_reply(proto 2 0 (0) Success) v1
debug -270> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078744+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -269> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eba40 0x560785263380 auth_reply(proto 2 0 (0) Success) v1
debug -268> 2021-11-16T21:23:40.176+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177084+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -267> 2021-11-16T21:23:40.176+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x560785225520 auth_reply(proto 2 0 (0) Success) v1
debug -266> 2021-11-16T21:23:40.177+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177745+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -265> 2021-11-16T21:23:40.177+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x5607858ad520 auth_reply(proto 2 0 (0) Success) v1
debug -264> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -263> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785917380 auth_reply(proto 2 0 (0) Success) v1
debug -262> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217665+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -261> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785916ea0 auth_reply(proto 2 0 (0) Success) v1
debug -260> 2021-11-16T21:23:40.274+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.274992+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -259> 2021-11-16T21:23:40.274+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ec30 0x56078525a820 auth_reply(proto 2 0 (0) Success) v1
debug -258> 2021-11-16T21:23:40.275+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.275761+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -257> 2021-11-16T21:23:40.275+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851521e0 0x560785258d00 auth_reply(proto 2 0 (0) Success) v1
debug -256> 2021-11-16T21:23:40.315+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.315817+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -255> 2021-11-16T21:23:40.315+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x56078591a4e0 auth_reply(proto 2 0 (0) Success) v1
debug -254> 2021-1
1-16T21:23:40.316+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.316340+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -253> 2021-11-16T21:23:40.316+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785259520 auth_reply(proto 2 0 (0) Success) v1
debug -252> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -251> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859131e0 auth_reply(proto 2 0 (0) Success) v1
debug -250> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413824+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -249> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785916000 auth_reply(proto 2 0 (0) Success) v1
debug -248> 2021-11-16T21:23:40.493+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.493816+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -247> 2021-11-16T21:23:40.493+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607859161a0 auth_reply(proto 2 0 (0) Success) v1
debug -246> 2021-11-16T21:23:40.494+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.494344+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -245> 2021-11-16T21:23:40.494+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x560785913380 auth_reply(proto 2 0 (0) Success) v1
debug -244> 2021-11-16T21:23:40.571+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.571680+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -243> 2021-11-16T21:23:40.571+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x560785916340 auth_reply(proto 2 0 (0) Success) v1
debug -242> 2021-11-16T21:23:40.572+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.572418+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -241> 2021-11-16T21:23:40.572+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785913520 auth_reply(proto 2 0 (0) Success) v1
debug -240> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634431+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -239> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ef00 0x5607859164e0 auth_reply(proto 2 0 (0) Success) v1
debug -238> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634908+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -237> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eff0 0x560785913d40 auth_reply(proto 2 0 (0) Success) v1
debug -236> 2021-11-16T21:23:40.696+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.696476+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -235> 2021-11-16T21:23:40.696+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1
debug -234> 2021-11-16T21:23:40.697+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.697195+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -233> 2021-11-16T21:23:40.697+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x560785913860 auth_reply(proto 2 0 (0) Success) v1
debug -232> 2021-11-16T21:23:40.784+0000 7f1d716db700 4 set_mon_vals no callback set
debug -231> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -230> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -229> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -228> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557610) is_readable = 1 - now=2021-11-16T21:23:40.785527+0000 lease_expire=2021-11-16T21:23:45.757378+0000 has v0 lc 19557610
debug -227> 2021-11-16T21:23:40.867+0000 7f1d716db700 4 set_mon_vals no callback set
debug -226> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -225> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -224> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -223> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.868717+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -222> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910205+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -221> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607858e9ba0 auth_reply(proto 2 0 (0) Success) v1
debug -220> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910749+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -219> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858ea340 auth_reply(proto 2 0 (0) Success) v1
debug -218> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038619+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -217> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x560785910820 auth_reply(proto 2 0 (0) Success) v1
debug -216> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038981+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -215> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785911520 auth_reply(proto 2 0 (0) Success) v1
debug -214> 2021-11-16T21:23:41.082+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.082731+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -213> 2021-11-16T21:23:41.082+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859111e0 auth_reply(proto 2 0 (0) Success) v1
debug -212> 2021-11-16T21:23:41.083+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.083295+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -211> 2021-11-16T21:23:41.083+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785911380 auth_reply(proto 2 0 (0) Success) v1
debug -210> 2021-11-16T21:23:41.167+0000 7f1d716db700 5 mon.cn03@2(peon).pax
os(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168120+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -209> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607859109c0 auth_reply(proto 2 0 (0) Success) v1
debug -208> 2021-11-16T21:23:41.168+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168780+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -207> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x560785911040 auth_reply(proto 2 0 (0) Success) v1
debug -206> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264181+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -205> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e0f0 0x560785910b60 auth_reply(proto 2 0 (0) Success) v1
debug -204> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264798+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -203> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785910d00 auth_reply(proto 2 0 (0) Success) v1
debug -202> 2021-11-16T21:23:41.360+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.360828+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -201> 2021-11-16T21:23:41.360+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153680 0x560785910ea0 auth_reply(proto 2 0 (0) Success) v1
debug -200> 2021-11-16T21:23:41.361+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.361219+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -199> 2021-11-16T21:23:41.361+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x560785910000 auth_reply(proto 2 0 (0) Success) v1
debug -198> 2021-11-16T21:23:41.475+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.475815+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -197> 2021-11-16T21:23:41.475+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x56078590f6c0 auth_reply(proto 2 0 (0) Success) v1
debug -196> 2021-11-16T21:23:41.476+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.476381+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -195> 2021-11-16T21:23:41.476+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x56078590fd40 auth_reply(proto 2 0 (0) Success) v1
debug -194> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.531532+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -193> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x56078590f860 auth_reply(proto 2 0 (0) Success) v1
debug -192> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.532037+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -191> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x56078590fa00 auth_reply(proto 2 0 (0) Success) v1
debug -190> 2021-11-16T21:23:42.200+0000 7f1d716db700 4 set_mon_vals no callback set
debug -189> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -188> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -187> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -186> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557612) is_readable = 1 - now=2021-11-16T21:23:42.202649+0000 lease_expire=2021-11-16T21:23:47.173895+0000 has v0 lc 19557612
debug -185> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 set_mon_vals no callback set
debug -184> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -183> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -182> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -181> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557613) is_readable = 1 - now=2021-11-16T21:23:42.294055+0000 lease_expire=2021-11-16T21:23:47.265226+0000 has v0 lc 19557613
debug -180> 2021-11-16T21:23:43.283+0000 7f1d716db700 4 set_mon_vals no callback set
debug -179> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -178> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -177> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -176> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557614) is_readable = 1 - now=2021-11-16T21:23:43.286367+0000 lease_expire=2021-11-16T21:23:48.257095+0000 has v0 lc 19557614
debug -175> 2021-11-16T21:23:43.384+0000 7f1d716db700 4 set_mon_vals no callback set
debug -174> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -173> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -172> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -171> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557615) is_readable = 1 - now=2021-11-16T21:23:43.386353+0000 lease_expire=2021-11-16T21:23:48.357251+0000 has v0 lc 19557615
debug -170> 2021-11-16T21:23:44.467+0000 7f1d716db700 4 set_mon_vals no callback set
debug -169> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -168> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -167> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -166> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557616) is_readable = 1 - now=2021-11-16T21:23:44.469822+0000 lease_expire=2021-11-16T21:23:49.440271+0000 has v0 lc 19557616
debug -165> 2021-11-16T21:23:44.567+0000 7f1d716db700 4 set_mon_vals no callback set
debug -164> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -163> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -162> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -161> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557617) is_readable = 1 - now=2021-11-16T21:23:44.569810+0000 lease_expire=2021-11-16T21:23:49.531775+0000 has
v0 lc 19557617
debug -160> 2021-11-16T21:23:45.684+0000 7f1d716db700 4 set_mon_vals no callback set
debug -159> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -158> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -157> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -156> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557618) is_readable = 1 - now=2021-11-16T21:23:45.686877+0000 lease_expire=2021-11-16T21:23:50.648710+0000 has v0 lc 19557618
debug -155> 2021-11-16T21:23:46.776+0000 7f1d716db700 4 set_mon_vals no callback set
debug -154> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -153> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -152> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -151> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557619) is_readable = 1 - now=2021-11-16T21:23:46.778635+0000 lease_expire=2021-11-16T21:23:51.740053+0000 has v0 lc 19557619
debug -150> 2021-11-16T21:23:46.876+0000 7f1d716db700 4 set_mon_vals no callback set
debug -149> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -148> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -147> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -146> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557620) is_readable = 1 - now=2021-11-16T21:23:46.878541+0000 lease_expire=2021-11-16T21:23:51.839977+0000 has v0 lc 19557620
debug -145> 2021-11-16T21:23:47.243+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #531955. Immutable memtables: 0.

debug -144> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.244227) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
debug -143> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 531955

debug -142> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827244296, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 683, "num_deletes": 506, "total_data_size": 1462899, "memory_usage": 1475944, "flush_reason": "Manual Compaction"}
debug -141> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #531956: started
debug -140> 2021-11-16T21:23:47.243+0000 7f1d716db700 4 set_mon_vals no callback set
debug -139> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -138> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -137> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -136> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557621) is_readable = 1 - now=2021-11-16T21:23:47.245909+0000 lease_expire=2021-11-16T21:23:52.207163+0000 has v0 lc 19557621
debug -135> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827293833, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 531956, "file_size": 966005, "table_properties": {"data_size": 962689, "index_size": 997, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1477, "raw_key_size": 12966, "raw_average_key_size": 22, "raw_value_size": 954089, "raw_average_value_size": 1667, "num_data_blocks": 29, "num_entries": 572, "num_deletions": 506, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097809, "oldest_key_time": 1637097809, "file_creation_time": 1637097827}}
debug -134> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:401] [default] [JOB 3] Level-0 flush table #531956: 966005 bytes OK
debug -133> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.293881) [memtable_list.cc:447] [default] Level-0 commit table #531956 started
debug -132> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327047) [memtable_list.cc:503] [default] Level-0 commit table #531956: memtable #1 done
debug -131> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327058) EVENT_LOG_v1 {"time_micros": 1637097827327054, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0}
debug -130> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327073) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[3 0 0 0 0 0 2] max score 0.75

debug -129> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 3] Try to delete WAL files size 1458128, prev total WAL file size 1458455, number of live WAL files 2.

debug -128> 2021-11-16T21:23:47.326+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -127> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327469) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'logm .. 'logm; will stop at (end)

debug -126> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 4] Compacting 3@0 + 2@6 files to L6, score -1.00
debug -125> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 3 Base level 0, inputs: [531956(943KB) 531950(148KB) 531947(7726KB)], [531944(64MB) 531945(36MB)]

debug -124> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827327529, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [531956, 531950, 531947], "files_L6": [531944, 531945], "score": -1, "input_data_size": 114546999}
debug -123> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531957: 3454 keys, 67198385 bytes
debug -122> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827785630, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531957, "file_size": 67198385, "table_properties": {"data_size": 67136484, "index_size": 52412, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is
_delta_encoded": 0, "filter_size": 8645, "raw_key_size": 76781, "raw_average_key_size": 22, "raw_value_size": 67047935, "raw_average_value_size": 19411, "num_data_blocks": 1555, "num_entries": 3454, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}}
debug -121> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531958: 1281 keys, 32015375 bytes
debug -120> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828044065, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531958, "file_size": 32015375, "table_properties": {"data_size": 31968485, "index_size": 42777, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 3269, "raw_key_size": 29915, "raw_average_key_size": 23, "raw_value_size": 31919151, "raw_average_value_size": 24917, "num_data_blocks": 1155, "num_entries": 1281, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}}
debug -119> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 4] Compacted 3@0 + 2@6 files to L6 => 99213760 bytes
debug -118> 2021-11-16T21:23:48.066+0000 7f1d716db700 4 set_mon_vals no callback set
debug -117> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -116> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -115> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -114> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557622) is_readable = 1 - now=2021-11-16T21:23:48.069418+0000 lease_expire=2021-11-16T21:23:52.960557+0000 has v0 lc 19557622
debug -113> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085683) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 159.9 rd, 138.5 wr, level 6, files in(3, 2) out(2) MB in(8.6, 100.6) out(94.6), read-write-amplify(23.7) write-amplify(11.0) OK, records in: 5811, records dropped: 1076 output_compression: NoCompression

debug -112> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085701) EVENT_LOG_v1 {"time_micros": 1637097828085695, "job": 4, "event": "compaction_finished", "compaction_time_micros": 716586, "compaction_time_cpu_micros": 83813, "output_level": 6, "num_output_files": 2, "total_output_size": 99213760, "num_input_records": 5811, "num_output_records": 4735, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]}
debug -111> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085923, "job": 4, "event": "table_file_deletion", "file_number": 531956}
debug -110> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085969, "job": 4, "event": "table_file_deletion", "file_number": 531950}
debug -109> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828086487, "job": 4, "event": "table_file_deletion", "file_number": 531947}
debug -108> 2021-11-16T21:23:48.088+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828089419, "job": 4, "event": "table_file_deletion", "file_number": 531945}
debug -107> 2021-11-16T21:23:48.093+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828094233, "job": 4, "event": "table_file_deletion", "file_number": 531944}
debug -106> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -105> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -104> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -103> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -102> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -101> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -100> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -99> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -98> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -97> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -96> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -95> 2021-11-16T21:23:49.260+0000 7f1d716db700 4 set_mon_vals no callback set
debug -94> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -93> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -92> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -91> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557623) is_readable = 1 - now=2021-11-16T21:23:49.262917+0000 lease_expire=2021-11-16T21:23:54.223482+0000 has v0 lc 19557623
debug -90> 2021-11-16T21:23:50.002+0000 7f1d716db700 4 set_mon_vals no callback set
debug -89> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -88> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -87> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -86> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557624) is_readable = 1 - now=2021-11-16T21:23:50.004764+0000 lease_expire=2021-11-16T21:23:54.973508+0000 has v0 lc 19557624
debug -85> 2021-11-16T21:23:51.102+0000 7f1d716db700 4 set_mon_vals no callback set
debug -84> 2021-11-16T21:23:51.103+0000
7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -83> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -82> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -81> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557625) is_readable = 1 - now=2021-11-16T21:23:51.104896+0000 lease_expire=2021-11-16T21:23:56.064982+0000 has v0 lc 19557625
debug -80> 2021-11-16T21:23:51.202+0000 7f1d716db700 4 set_mon_vals no callback set
debug -79> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -78> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -77> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -76> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557626) is_readable = 1 - now=2021-11-16T21:23:51.204980+0000 lease_expire=2021-11-16T21:23:56.164769+0000 has v0 lc 19557626
debug -75> 2021-11-16T21:23:52.261+0000 7f1d716db700 4 set_mon_vals no callback set
debug -74> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -73> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -72> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -71> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557627) is_readable = 1 - now=2021-11-16T21:23:52.263508+0000 lease_expire=2021-11-16T21:23:57.231263+0000 has v0 lc 19557627
debug -70> 2021-11-16T21:23:52.352+0000 7f1d716db700 4 set_mon_vals no callback set
debug -69> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -68> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -67> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -66> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557628) is_readable = 1 - now=2021-11-16T21:23:52.355232+0000 lease_expire=2021-11-16T21:23:57.322723+0000 has v0 lc 19557628
debug -65> 2021-11-16T21:23:53.436+0000 7f1d716db700 4 set_mon_vals no callback set
debug -64> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -63> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -62> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -61> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557629) is_readable = 1 - now=2021-11-16T21:23:53.438872+0000 lease_expire=2021-11-16T21:23:58.406301+0000 has v0 lc 19557629
debug -60> 2021-11-16T21:23:54.519+0000 7f1d716db700 4 set_mon_vals no callback set
debug -59> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -58> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -57> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -56> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557630) is_readable = 1 - now=2021-11-16T21:23:54.522319+0000 lease_expire=2021-11-16T21:23:59.489316+0000 has v0 lc 19557630
debug -55> 2021-11-16T21:23:55.595+0000 7f1d716db700 4 set_mon_vals no callback set
debug -54> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -53> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -52> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -51> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557631) is_readable = 1 - now=2021-11-16T21:23:55.597582+0000 lease_expire=2021-11-16T21:24:00.564365+0000 has v0 lc 19557631
debug -50> 2021-11-16T21:23:56.670+0000 7f1d716db700 4 set_mon_vals no callback set
debug -49> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -48> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -47> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -46> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557632) is_readable = 1 - now=2021-11-16T21:23:56.672809+0000 lease_expire=2021-11-16T21:24:01.638988+0000 has v0 lc 19557632
debug -45> 2021-11-16T21:23:57.270+0000 7f1d716db700 4 set_mon_vals no callback set
debug -44> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -43> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -42> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -41> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557633) is_readable = 1 - now=2021-11-16T21:23:57.272872+0000 lease_expire=2021-11-16T21:24:02.238966+0000 has v0 lc 19557633
debug -40> 2021-11-16T21:23:57.753+0000 7f1d716db700 4 set_mon_vals no callback set
debug -39> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -38> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -37> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -36> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557634) is_readable = 1 - now=2021-11-16T21:23:57.756400+0000 lease_expire=2021-11-16T21:24:02.722461+0000 has v0 lc 19557634
debug -35> 2021-11-16T21:23:58.829+0000 7f1d716db700 4 set_mon_vals no callback set
debug -34> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -33> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -32> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -31> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557635) is_readable = 1 - now=2021-11-16T21:23:58.831535+0000 lease_expire=2021-11-16T21:24:03.797240+0000 has v0 lc 19557635
debug -30> 2021-11-16T21:23:58.920+0000 7f1d716db700 4 set_mon_vals no callback set
debug -29> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -28> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6
849/4203565106]
debug -27> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -26> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557636) is_readable = 1 - now=2021-11-16T21:23:58.922888+0000 lease_expire=2021-11-16T21:24:03.888665+0000 has v0 lc 19557636
debug -25> 2021-11-16T21:23:59.354+0000 7f1d716db700 4 set_mon_vals no callback set
debug -24> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -23> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -22> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -21> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557637) is_readable = 1 - now=2021-11-16T21:23:59.356709+0000 lease_expire=2021-11-16T21:24:04.321986+0000 has v0 lc 19557637
debug -20> 2021-11-16T21:24:00.462+0000 7f1d716db700 4 set_mon_vals no callback set
debug -19> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -18> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -17> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -16> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557638) is_readable = 1 - now=2021-11-16T21:24:00.465271+0000 lease_expire=2021-11-16T21:24:05.430188+0000 has v0 lc 19557638
debug -15> 2021-11-16T21:24:00.554+0000 7f1d716db700 4 set_mon_vals no callback set
debug -14> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -13> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -12> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -11> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557639) is_readable = 1 - now=2021-11-16T21:24:00.556869+0000 lease_expire=2021-11-16T21:24:05.521766+0000 has v0 lc 19557639
debug -10> 2021-11-16T21:24:01.629+0000 7f1d716db700 4 set_mon_vals no callback set
debug -9> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -8> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -7> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -6> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.632342+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -5> 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0)
debug -4> 2021-11-16T21:24:01.873+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875099+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -3> 2021-11-16T21:24:01.873+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x56078525b380 auth_reply(proto 2 0 (0) Success) v1
debug -2> 2021-11-16T21:24:01.874+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875450+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -1> 2021-11-16T21:24:01.874+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eb40 0x560785259d40 auth_reply(proto 2 0 (0) Success) v1
debug 0> 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) **
in thread 7f1d716db700 thread_name:ms_dispatch

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8]
8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045]
9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6]
10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329]
11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258]
12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc]
15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71]
17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a]
18: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
139764358260480 / rstore_compact
139764375045888 / ms_dispatch
139764391831296 / rocksdb:dump_st
139764417009408 / msgr-worker-0
139764433794816 / ms_dispatch
139764450580224 / cpu_tp
139764467365632 / cpu_tp
139764475758336 / safe_timer
139764526114560 / rocksdb:high0
139764534507264 / rocksdb:low0
139764560197376 / admin_socket
139764813260544 / ceph-mon
max_recent 10000
max_new 10000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
debug -734> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command assert hook 0x5607840e8530
debug -733> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command abort hook 0x5607840e8530
debug -732> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command leak_some_memory hook 0x5607840e8530
debug -731> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_dump hook 0x5607840e8530
debug -730> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 1 hook 0x5607840e8530
debug -729> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf dump hook 0x5607840e8530
debug -728> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfc
ounters_schema hook 0x5607840e8530
debug -727> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram dump hook 0x5607840e8530
debug -726> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 2 hook 0x5607840e8530
debug -725> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf schema hook 0x5607840e8530
debug -724> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram schema hook 0x5607840e8530
debug -723> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf reset hook 0x5607840e8530
debug -722> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config show hook 0x5607840e8530
debug -721> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config help hook 0x5607840e8530
debug -720> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config set hook 0x5607840e8530
debug -719> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config unset hook 0x5607840e8530
debug -718> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config get hook 0x5607840e8530
debug -717> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff hook 0x5607840e8530
debug -716> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff get hook 0x5607840e8530
debug -715> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command injectargs hook 0x5607840e8530
debug -714> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log flush hook 0x5607840e8530
debug -713> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log dump hook 0x5607840e8530
debug -712> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log reopen hook 0x5607840e8530
debug -711> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_mempools hook 0x560784188328
debug -710> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 set uid:gid to 167:167 (ceph:ceph)
debug -709> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 8
debug -708> 2021-11-16T21:23:28.656+0000 7f1d880be700 0 pidfile_write: ignore empty --pid-file
debug -707> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) init /var/run/ceph/ceph-mon.cn03.asok
debug -706> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) bind_and_listen /var/run/ceph/ceph-mon.cn03.asok
debug -705> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command 0 hook 0x5607840ec108
debug -704> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command version hook 0x5607840ec108
debug -703> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command git_version hook 0x5607840ec108
debug -702> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command help hook 0x5607840e8230
debug -701> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command get_command_descriptions hook 0x5607840e81e0
debug -700> 2021-11-16T21:23:28.656+0000 7f1d78f67700 5 asok(0x56078419c000) entry start
debug -699> 2021-11-16T21:23:28.660+0000 7f1d880be700 0 load: jerasure load: lrc load: isa
debug -698> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -697> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432
debug -696> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression
debug -695> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -694> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432
debug -693> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression
debug -692> 2021-11-16T21:23:28.661+0000 7f1d880be700 5 rocksdb: verify_sharding column families from rocksdb: [default]
debug -691> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: RocksDB version: 6.8.1

debug -690> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
debug -689> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compile date May 13 2021
debug -688> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: DB SUMMARY

debug -687> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: CURRENT file: CURRENT

debug -686> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: IDENTITY file: IDENTITY

debug -685> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: MANIFEST file: MANIFEST-531948 size: 302 Bytes

debug -684> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-cn03/store.db dir, Total Num: 3, files: 531944.sst 531945.sst 531947.sst

debug -683> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-cn03/store.db: 531949.log size: 214103 ;

debug -682> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.error_if_exists: 0
debug -681> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_if_missing: 0
debug -680> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_checks: 1
debug -679> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.env: 0x5607837a11c0
debug -678> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.fs: Posix File System
debug -677> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.info_log: 0x5607841d78c0
debug -676> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_file_opening_threads: 16
debug -675> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.statistics: (nil)
debug -674> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_fsync: 0
debug -673> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_log_file_size: 0
debug -672> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_manifest_file_size: 1073741824
debug -671> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_file_time_to_roll: 0
debug -670> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.keep_log_file_num: 1000
debug -669> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.recycle_log_file_num: 0
debug -668> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_fallocate: 1
debug -667> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_reads: 0
debug -666> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_writes: 0
debug -665> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_reads: 0
debug -664> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
debug -663> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_missing_column_families: 0
debug -662> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_log_dir:
debug -661> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-cn03
/store.db
debug -660> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_cache_numshardbits: 6
debug -659> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_subcompactions: 1
debug -658> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_flushes: -1
debug -657> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_ttl_seconds: 0
debug -656> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_size_limit_MB: 0
debug -655> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576
debug -654> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manifest_preallocation_size: 4194304
debug -653> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.is_fd_close_on_exec: 1
debug -652> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.advise_random_on_open: 1
debug -651> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_write_buffer_size: 0
debug -650> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_manager: 0x5607841ddf80
debug -649> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.access_hint_on_compaction_start: 1
debug -648> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
debug -647> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.random_access_max_buffer_size: 1048576
debug -646> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_adaptive_mutex: 0
debug -645> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limiter: (nil)
debug -644> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
debug -643> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_recovery_mode: 2
debug -642> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_thread_tracking: 0
debug -641> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_pipelined_write: 0
debug -640> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.unordered_write: 0
debug -639> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_concurrent_memtable_write: 1
debug -638> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
debug -637> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_max_yield_usec: 100
debug -636> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_slow_yield_usec: 3
debug -635> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.row_cache: None
debug -634> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_filter: None
debug -633> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_recovery: 0
debug -632> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_ingest_behind: 0
debug -631> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.preserve_deletes: 0
debug -630> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.two_write_queues: 0
debug -629> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manual_wal_flush: 0
debug -628> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.atomic_flush: 0
debug -627> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0
debug -626> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.persist_stats_to_disk: 0
debug -625> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_dbid_to_manifest: 0
debug -624> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_readahead_size: 0
debug -623> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_checksum_func: Unknown
debug -622> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_jobs: 2
debug -621> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_compactions: -1
debug -620> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_shutdown: 0
debug -619> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
debug -618> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delayed_write_rate : 16777216
debug -617> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_total_wal_size: 0
debug -616> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
debug -615> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_dump_period_sec: 600
debug -614> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_persist_period_sec: 600
debug -613> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_history_buffer_size: 1048576
debug -612> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_open_files: -1
debug -611> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bytes_per_sync: 0
debug -610> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_bytes_per_sync: 0
debug -609> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.strict_bytes_per_sync: 0
debug -608> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_readahead_size: 0
debug -607> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compression algorithms supported:
debug -606> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTDNotFinalCompression supported: 0
debug -605> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTD supported: 0
debug -604> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kXpressCompression supported: 0
debug -603> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4HCCompression supported: 1
debug -602> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4Compression supported: 1
debug -601> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kBZip2Compression supported: 0
debug -600> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZlibCompression supported: 1
debug -599> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kSnappyCompression supported: 1
debug -598> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Fast CRC32 supported: Supported on x86
debug -597> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948

debug -596> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]:

debug -595> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
debug -594> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.merge_operator:
debug -593> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter: None
debug -592> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter_factory: None
debug -591> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtab
le_factory: SkipListFactory
debug -590> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_factory: BlockBasedTable
debug -589> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607840ec128)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x560784125010
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0

debug -588> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_size: 33554432
debug -587> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number: 2
debug -586> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression: NoCompression
debug -585> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression: Disabled
debug -584> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.prefix_extractor: nullptr
debug -583> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
debug -582> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.num_levels: 7
debug -581> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
debug -580> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
debug -579> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0
debug -578> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
debug -577> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.level: 32767
debug -576> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
debug -575> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
debug -574> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
debug -573> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.enabled: false
debug -572> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.window_bits: -14
debug -571> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.level: 32767
debug -570> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.strategy: 0
debug -569> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
debug -568> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
debug -567> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.enabled: false
debug -566> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
debug -565> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
debug -564> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_stop_writes_trigger: 36
debug -563> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_base: 67108864
debug -562> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_multiplier: 1
debug -561> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_base: 268435456
debug -560> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
debug -559> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
debug -558> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
debug -557> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
debug -556> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
debug -555> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
debug -554> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
debug -553> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
debug -552> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
debug -551> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
debug -550> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_compaction_bytes: 1677721600
debug -549> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.arena_block_size: 4194304
debug -548> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
debug -547> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
debug -546> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
debug -545> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.disable_auto_compactions: 0
debug -544> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
debug -543> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
debug -542> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
debug -541> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
debug -540> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
debug -539> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
debug -538> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
debug -537> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
debug -536> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
debug -535> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
debug -534> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_properties_collectors:
debug -533> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Opt
ions.inplace_update_support: 0
debug -532> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_num_locks: 10000
debug -531> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
debug -530> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_whole_key_filtering: 0
debug -529> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_huge_page_size: 0
debug -528> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bloom_locality: 0
debug -527> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_successive_merges: 0
debug -526> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.optimize_filters_for_hits: 0
debug -525> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_file_checks: 0
debug -524> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.force_consistency_checks: 0
debug -523> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.report_bg_io_stats: 0
debug -522> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.ttl: 2592000
debug -521> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.periodic_compaction_seconds: 0
debug -520> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 succeeded,manifest_file_number is 531948, next_file_number is 531950, last_sequence is 201837822, log_number is 531943,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

debug -519> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 531943

debug -518> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808663528, "job": 1, "event": "recovery_started", "log_files": [531949]}
debug -517> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #531949 mode 2
debug -516> 2021-11-16T21:23:28.663+0000 7f1d880be700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
debug -515> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808697864, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 531950, "file_size": 151829, "table_properties": {"data_size": 150555, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": 759, "raw_average_key_size": 25, "raw_value_size": 149707, "raw_average_value_size": 4990, "num_data_blocks": 8, "num_entries": 30, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097808, "oldest_key_time": 3, "file_creation_time": 0}}
debug -514> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: [version_set.cc:3826] Creating manifest 531951

debug -513> 2021-11-16T21:23:28.755+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808756165, "job": 1, "event": "recovery_finished"}
debug -512> 2021-11-16T21:23:28.797+0000 7f1d880be700 4 rocksdb: DB pointer 0x5607841e9800
debug -511> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS -------
debug -510> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:851]
** DB Stats **
Uptime(secs): 0.1 total, 0.1 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0

** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Uptime(secs): 0.1 total, 0.1 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00
0.00 0 0.000 0 0

** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0
Uptime(secs): 0.1 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

debug -509> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -508> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -507> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx
debug -506> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: none
debug -505> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -504> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -503> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -502> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -501> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -500> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -499> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -498> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -497> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -496> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -495> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc
debug -494> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure
debug -493> 2021-11-16T21:23:28.797+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -492> 2021-11-16T21:23:28.798+0000 7f1d880be700 0 starting mon.cn03 rank 2 at public addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] at bind addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon_data /var/lib/ceph/mon/ceph-cn03 fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
debug -491> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -490> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -489> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx
debug -488> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: none
debug -487> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -486> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -485> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -484> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -483> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -482> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -481> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -480> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -479> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -478> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -477> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc
debug -476> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure
debug -475> 2021-11-16T21:23:28.798+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -474> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx
debug -473> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx
debug -472> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
debug -471> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
debug -470> 2021-11-16T21:23:28.799+0000 7f1d880be700 1 mon.cn03@-1(???) e69 preinit fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
debug -469> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 new map
debug -468> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 print_map
e432
enable_multiple, ever_enabled_multiple: 1,1
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 2

Filesystem 'coldlogix' (2)
fs_name coldlogix
epoch 402
flags 12
created 2021-06-27T06:32:25.086015+0000
modified 2021-11-15T08:30:22.670221+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 17247
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0
up {0=8444199}
failed
damaged
stopped
data_pools [12]
metadata_pool 11
inline_data disabled
balancer
standby_count_wanted 1
[mds.coldlogix.cn01.dirzwe{0:8444199} state up:active seq 77 join_fscid=2 addr [v2:192.168.30.11:6850/3427514521,v1:192.168.30.11:6851/3427514521]]

Filesystem 'btc' (5)
fs_name btc
epoch 423
flags 12
created 2021-11-15T08:49:14.520956+0000
modified 2021-11-15T08:49:25.936407+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 17308
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
i
n 0
up {0=8534156}
failed
damaged
stopped
data_pools [19]
metadata_pool 18
inline_data disabled
balancer
standby_count_wanted 1
[mds.btc.cn04.pyuaxe{0:8534156} state up:active seq 5 join_fscid=5 addr [v2:192.168.30.14:6800/4253241567,v1:192.168.30.14:6801/4253241567]]

Standby daemons:

[mds.btc.cn06.hqmwsx{-1:8534166} state up:standby seq 3 join_fscid=5 addr [v2:192.168.30.16:6848/446743436,v1:192.168.30.16:6849/446743436]]
[mds.coldlogix.cn03.lyxzhp{-1:8604119} state up:standby seq 4 join_fscid=2 addr [v2:192.168.30.13:6800/2645334932,v1:192.168.30.13:6801/2645334932]]

debug -467> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 3314933000854323200, adjusting msgr requires
debug -466> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -465> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -464> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires
debug -463> 2021-11-16T21:23:28.800+0000 7f1d880be700 1 mon.cn03@-1(???).paxosservice(auth 10008..10244) refresh upgraded, format 0 -> 3
debug -462> 2021-11-16T21:23:28.800+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e0 loading version 500
debug -461> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 active server: [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106](8444129)
debug -460> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 mkfs or daemon transitioned to available, loading commands
debug -459> 2021-11-16T21:23:28.802+0000 7f1d880be700 4 set_mon_vals no callback set
debug -458> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals auth_allow_insecure_global_id_reclaim = false
debug -457> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949
debug -456> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals mon_allow_pool_delete = true
debug -455> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals rbd_move_to_trash_on_remove = true
debug -454> 2021-11-16T21:23:28.803+0000 7f1d880be700 4 set_mon_vals no callback set
debug -453> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -452> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command compact hook 0x5607840e89a0
debug -451> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command smart hook 0x5607840e89a0
debug -450> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command mon_status hook 0x5607840e89a0
debug -449> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command heap hook 0x5607840e89a0
debug -448> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores dump hook 0x5607840e89a0
debug -447> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores reset hook 0x5607840e89a0
debug -446> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sync_force hook 0x5607840e89a0
debug -445> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hint hook 0x5607840e89a0
debug -444> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hintv hook 0x5607840e89a0
debug -443> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum enter hook 0x5607840e89a0
debug -442> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum exit hook 0x5607840e89a0
debug -441> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command ops hook 0x5607840e89a0
debug -440> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sessions hook 0x5607840e89a0
debug -439> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_historic_ops hook 0x5607840e89a0
debug -438> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -437> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -436> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx
debug -435> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: none
debug -434> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -433> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -432> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -431> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -430> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -429> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -428> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -427> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -426> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -425> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -424> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc
debug -423> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure
debug -422> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring
debug -421> 2021-11-16T21:23:28.804+0000 7f1d880be700 2 mon.cn03@-1(???) e69 init
debug -420> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Got map version 500
debug -419> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -418> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -417> 2021-11-16T21:23:28.805+0000 7f1d726dd700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d726dd700' had timed out after 0.000000000s
debug -416> 2021-11-16T21:23:28.806+0000 7f1d880be700 0 mon.cn03@-1(probing) e69 my rank is now 2 (was -1)
debug -415> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure stats_period=5
debug -414> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure updated stats threshold: 5
debug -413> 2021-11-16T21:23:28.807+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection
debug -412> 2021-11-16T21:23:29.213+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection
debug -411> 2021-11-16T21:23:29.214+0000 7f1d716db700 0 log_channel(cluster) log [INF] : mon.cn03 calling monitor election
debug -410> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client _send_to_mon log to self
debug -409> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
debug -408> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client will send 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election
debug -407> 2021-11-16T21:23:29.214+0000 7f1d716db
700 5 paxos.2).electionLogic(0) start -- can i be leader?
debug -406> 2021-11-16T21:23:29.214+0000 7f1d716db700 1 paxos.2).electionLogic(5412) init, last seen epoch 5412
debug -405> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -404> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248762+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -403> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248771+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -402> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.1
debug -401> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 paxos.2).electionLogic(5413) defer to 1
debug -400> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3
debug -399> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 paxos.2).electionLogic(5413) no, we already acked 1
debug -398> 2021-11-16T21:23:30.619+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1
debug -397> 2021-11-16T21:23:33.077+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -396> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.4
debug -395> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 paxos.2).electionLogic(5413) ignoring old propose
debug -394> 2021-11-16T21:23:34.411+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3
debug -393> 2021-11-16T21:23:34.440+0000 7f1d716db700 5 paxos.2).electionLogic(5415) no, we already acked 1
debug -392> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection
debug -391> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5415) handle_propose from mon.0
debug -390> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 paxos.2).electionLogic(5415) ignoring old propose
debug -389> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) election period ended
debug -388> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) start -- can i be leader?
debug -387> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 1 paxos.2).electionLogic(5415) init, last seen epoch 5415, mid-election, bumping
debug -386> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_ack from mon.3
debug -385> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) so far i have { mon.2: features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) }
debug -384> 2021-11-16T21:23:35.392+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_propose from mon.0
debug -383> 2021-11-16T21:23:35.424+0000 7f1d716db700 5 paxos.2).electionLogic(5419) defer to 0
debug -382> 2021-11-16T21:23:36.621+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1
debug -381> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256876+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -380> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256899+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -379> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256913+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -378> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256925+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -377> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257153+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -376> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257169+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -375> 2021-11-16T21:23:38.342+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5419) handle_victory from mon.0 quorum_features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging])
debug -374> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_quorum_to_compatset_features
debug -373> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_monmap_to_compatset_features
debug -372> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382930+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -371> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382942+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603
debug -370> 2021-11-16T21:23:38.450+0000 7f1d716db700 4 set_mon_vals no callback set
debug -369> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -368> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -367> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -366> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451523+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -365> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451536+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -364> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451571+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -363> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451596+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -362> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451620+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -361> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451641+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604
debug -360> 2021-11-16T21:23:38.452+0000 7f1d736df700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d736df700' had timed out after 0.000000000s
debug -359> 2021-11-16T21:23:38.538+0000 7f1d706d9700 1 mon.cn03@2(peon) e69 handle_auth_request failed to assign global_id
debug -358> 2021-11-16T21:23:38.541+0000 7f1d716db700 4 set_mon_vals no callback set
debug -357> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr
_map Got map version 500
debug -356> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -355> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -354> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557605) is_readable = 1 - now=2021-11-16T21:23:38.542983+0000 lease_expire=2021-11-16T21:23:43.507566+0000 has v0 lc 19557605
debug -353> 2021-11-16T21:23:38.635+0000 7f1d716db700 4 set_mon_vals no callback set
debug -352> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -351> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -350> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -349> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557606) is_readable = 1 - now=2021-11-16T21:23:38.636621+0000 lease_expire=2021-11-16T21:23:43.603066+0000 has v0 lc 19557606
debug -348> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client handle_log_ack log(last 1) v1
debug -347> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client logged 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election
debug -346> 2021-11-16T21:23:38.725+0000 7f1d716db700 4 set_mon_vals no callback set
debug -345> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -344> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -343> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -342> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557607) is_readable = 1 - now=2021-11-16T21:23:38.726705+0000 lease_expire=2021-11-16T21:23:43.690619+0000 has v0 lc 19557607
debug -341> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783222+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607
debug -340> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783242+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607
debug -339> 2021-11-16T21:23:38.841+0000 7f1d716db700 4 set_mon_vals no callback set
debug -338> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -337> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -336> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -335> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.843331+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -334> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).elector(5420) begin_peer_ping against 3
debug -333> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944355+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -332> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853ebef0 0x5607858deb60 auth_reply(proto 2 0 (0) Success) v1
debug -331> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944811+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -330> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858df380 auth_reply(proto 2 0 (0) Success) v1
debug -329> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984545+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -328> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x5607858de820 auth_reply(proto 2 0 (0) Success) v1
debug -327> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984931+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -326> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851533b0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1
debug -325> 2021-11-16T21:23:39.026+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.026942+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -324> 2021-11-16T21:23:39.026+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x5607858de9c0 auth_reply(proto 2 0 (0) Success) v1
debug -323> 2021-11-16T21:23:39.027+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.027319+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -322> 2021-11-16T21:23:39.027+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x5607858deea0 auth_reply(proto 2 0 (0) Success) v1
debug -321> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067500+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -320> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x5607858df520 auth_reply(proto 2 0 (0) Success) v1
debug -319> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067791+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -318> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607858de680 auth_reply(proto 2 0 (0) Success) v1
debug -317> 2021-11-16T21:23:39.105+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.105918+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -316> 2021-11-16T21:23:39.105+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858dc9c0 auth_reply(proto 2 0 (0) Success) v1
debug -315> 2021-11-16T21:23:39.106+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.106361+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -314> 2021-11-16T21:23:39.106+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607858dc820 auth_reply(proto 2 0 (0) Success) v1
debug -313> 2021-11-16T21:23:39.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.217724+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -312> 2021-11-16T21:23:39.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e4b0 0x560785262b60 auth_reply(proto 2 0 (0) Success) v1
debug -311> 2021-11-16T21:23:39.218+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.218155+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -310> 2021-11-1
6T21:23:39.218+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x5607852636c0 auth_reply(proto 2 0 (0) Success) v1
debug -309> 2021-11-16T21:23:39.335+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336070+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -308> 2021-11-16T21:23:39.335+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858eb6c0 auth_reply(proto 2 0 (0) Success) v1
debug -307> 2021-11-16T21:23:39.336+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336443+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -306> 2021-11-16T21:23:39.336+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858eab60 auth_reply(proto 2 0 (0) Success) v1
debug -305> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.388745+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -304> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x560785916680 auth_reply(proto 2 0 (0) Success) v1
debug -303> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.389042+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -302> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x560785916340 auth_reply(proto 2 0 (0) Success) v1
debug -301> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415381+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -300> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb770 0x5607858f2b60 auth_reply(proto 2 0 (0) Success) v1
debug -299> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415676+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -298> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607858ec9c0 auth_reply(proto 2 0 (0) Success) v1
debug -297> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441170+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -296> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785912d00 auth_reply(proto 2 0 (0) Success) v1
debug -295> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441498+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -294> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785912680 auth_reply(proto 2 0 (0) Success) v1
debug -293> 2021-11-16T21:23:39.547+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.547657+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -292> 2021-11-16T21:23:39.547+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x5607859124e0 auth_reply(proto 2 0 (0) Success) v1
debug -291> 2021-11-16T21:23:39.548+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.548146+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608
debug -290> 2021-11-16T21:23:39.548+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607852621a0 auth_reply(proto 2 0 (0) Success) v1
debug -289> 2021-11-16T21:23:39.717+0000 7f1d716db700 4 set_mon_vals no callback set
debug -288> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -287> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -286> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -285> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.718923+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -284> 2021-11-16T21:23:39.796+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.796980+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -283> 2021-11-16T21:23:39.796+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x560785263860 auth_reply(proto 2 0 (0) Success) v1
debug -282> 2021-11-16T21:23:39.797+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.797502+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -281> 2021-11-16T21:23:39.797+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858b36c0 auth_reply(proto 2 0 (0) Success) v1
debug -280> 2021-11-16T21:23:39.853+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854122+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -279> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e000 0x56078525eb60 auth_reply(proto 2 0 (0) Success) v1
debug -278> 2021-11-16T21:23:39.854+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854658+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -277> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858eb520 auth_reply(proto 2 0 (0) Success) v1
debug -276> 2021-11-16T21:23:39.999+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.999601+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -275> 2021-11-16T21:23:39.999+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x560785263520 auth_reply(proto 2 0 (0) Success) v1
debug -274> 2021-11-16T21:23:40.000+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.000151+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -273> 2021-11-16T21:23:40.000+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x5607858b3040 auth_reply(proto 2 0 (0) Success) v1
debug -272> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078366+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -271> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607852631e0 auth_reply(proto 2 0 (0) Success) v1
debug -270> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078744+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -269> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eba40 0x560785263380 auth_reply(proto 2 0 (0) Success) v1
debug -268> 2021-11-16T21:23:40.176+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177084+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -267> 2021-11-
16T21:23:40.176+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x560785225520 auth_reply(proto 2 0 (0) Success) v1
debug -266> 2021-11-16T21:23:40.177+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177745+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -265> 2021-11-16T21:23:40.177+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x5607858ad520 auth_reply(proto 2 0 (0) Success) v1
debug -264> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -263> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785917380 auth_reply(proto 2 0 (0) Success) v1
debug -262> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217665+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -261> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785916ea0 auth_reply(proto 2 0 (0) Success) v1
debug -260> 2021-11-16T21:23:40.274+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.274992+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -259> 2021-11-16T21:23:40.274+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ec30 0x56078525a820 auth_reply(proto 2 0 (0) Success) v1
debug -258> 2021-11-16T21:23:40.275+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.275761+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -257> 2021-11-16T21:23:40.275+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851521e0 0x560785258d00 auth_reply(proto 2 0 (0) Success) v1
debug -256> 2021-11-16T21:23:40.315+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.315817+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -255> 2021-11-16T21:23:40.315+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x56078591a4e0 auth_reply(proto 2 0 (0) Success) v1
debug -254> 2021-11-16T21:23:40.316+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.316340+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -253> 2021-11-16T21:23:40.316+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785259520 auth_reply(proto 2 0 (0) Success) v1
debug -252> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -251> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859131e0 auth_reply(proto 2 0 (0) Success) v1
debug -250> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413824+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -249> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785916000 auth_reply(proto 2 0 (0) Success) v1
debug -248> 2021-11-16T21:23:40.493+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.493816+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -247> 2021-11-16T21:23:40.493+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607859161a0 auth_reply(proto 2 0 (0) Success) v1
debug -246> 2021-11-16T21:23:40.494+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.494344+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -245> 2021-11-16T21:23:40.494+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x560785913380 auth_reply(proto 2 0 (0) Success) v1
debug -244> 2021-11-16T21:23:40.571+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.571680+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -243> 2021-11-16T21:23:40.571+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x560785916340 auth_reply(proto 2 0 (0) Success) v1
debug -242> 2021-11-16T21:23:40.572+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.572418+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -241> 2021-11-16T21:23:40.572+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785913520 auth_reply(proto 2 0 (0) Success) v1
debug -240> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634431+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -239> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ef00 0x5607859164e0 auth_reply(proto 2 0 (0) Success) v1
debug -238> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634908+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -237> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eff0 0x560785913d40 auth_reply(proto 2 0 (0) Success) v1
debug -236> 2021-11-16T21:23:40.696+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.696476+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -235> 2021-11-16T21:23:40.696+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1
debug -234> 2021-11-16T21:23:40.697+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.697195+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609
debug -233> 2021-11-16T21:23:40.697+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x560785913860 auth_reply(proto 2 0 (0) Success) v1
debug -232> 2021-11-16T21:23:40.784+0000 7f1d716db700 4 set_mon_vals no callback set
debug -231> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -230> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -229> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -228> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557610) is_readable = 1 - now=2021-11-16T21:23:40.785527+0000 lease_expire=2021-11-16T21:23:45.757378+0000 has v0 lc 19557610
debug -227> 2021-11-16T21:23:40.867+0000 7f1d716db700 4 set_mon_vals no callback set
debug -226> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -225> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -224> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -223> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40
.868717+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -222> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910205+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -221> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607858e9ba0 auth_reply(proto 2 0 (0) Success) v1
debug -220> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910749+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -219> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858ea340 auth_reply(proto 2 0 (0) Success) v1
debug -218> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038619+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -217> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x560785910820 auth_reply(proto 2 0 (0) Success) v1
debug -216> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038981+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -215> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785911520 auth_reply(proto 2 0 (0) Success) v1
debug -214> 2021-11-16T21:23:41.082+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.082731+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -213> 2021-11-16T21:23:41.082+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859111e0 auth_reply(proto 2 0 (0) Success) v1
debug -212> 2021-11-16T21:23:41.083+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.083295+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -211> 2021-11-16T21:23:41.083+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785911380 auth_reply(proto 2 0 (0) Success) v1
debug -210> 2021-11-16T21:23:41.167+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168120+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -209> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607859109c0 auth_reply(proto 2 0 (0) Success) v1
debug -208> 2021-11-16T21:23:41.168+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168780+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -207> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x560785911040 auth_reply(proto 2 0 (0) Success) v1
debug -206> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264181+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -205> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e0f0 0x560785910b60 auth_reply(proto 2 0 (0) Success) v1
debug -204> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264798+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -203> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785910d00 auth_reply(proto 2 0 (0) Success) v1
debug -202> 2021-11-16T21:23:41.360+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.360828+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -201> 2021-11-16T21:23:41.360+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153680 0x560785910ea0 auth_reply(proto 2 0 (0) Success) v1
debug -200> 2021-11-16T21:23:41.361+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.361219+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -199> 2021-11-16T21:23:41.361+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x560785910000 auth_reply(proto 2 0 (0) Success) v1
debug -198> 2021-11-16T21:23:41.475+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.475815+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -197> 2021-11-16T21:23:41.475+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x56078590f6c0 auth_reply(proto 2 0 (0) Success) v1
debug -196> 2021-11-16T21:23:41.476+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.476381+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -195> 2021-11-16T21:23:41.476+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x56078590fd40 auth_reply(proto 2 0 (0) Success) v1
debug -194> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.531532+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -193> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x56078590f860 auth_reply(proto 2 0 (0) Success) v1
debug -192> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.532037+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611
debug -191> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x56078590fa00 auth_reply(proto 2 0 (0) Success) v1
debug -190> 2021-11-16T21:23:42.200+0000 7f1d716db700 4 set_mon_vals no callback set
debug -189> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -188> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -187> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -186> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557612) is_readable = 1 - now=2021-11-16T21:23:42.202649+0000 lease_expire=2021-11-16T21:23:47.173895+0000 has v0 lc 19557612
debug -185> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 set_mon_vals no callback set
debug -184> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -183> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -182> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -181> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557613) is_readable = 1 - now=2021-11-16T21:23:42.294055+0000 lease_expire=2021-11-16T21:23:47.265226+0000 has v0 lc 19557613
debug -180> 2021-11-16T21:23:43.283+0000 7f1d716db700 4 set_mon_vals no callback set
debug -179> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -178> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -177> 2021-11-1
6T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -176> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557614) is_readable = 1 - now=2021-11-16T21:23:43.286367+0000 lease_expire=2021-11-16T21:23:48.257095+0000 has v0 lc 19557614
debug -175> 2021-11-16T21:23:43.384+0000 7f1d716db700 4 set_mon_vals no callback set
debug -174> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -173> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -172> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -171> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557615) is_readable = 1 - now=2021-11-16T21:23:43.386353+0000 lease_expire=2021-11-16T21:23:48.357251+0000 has v0 lc 19557615
debug -170> 2021-11-16T21:23:44.467+0000 7f1d716db700 4 set_mon_vals no callback set
debug -169> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -168> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -167> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -166> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557616) is_readable = 1 - now=2021-11-16T21:23:44.469822+0000 lease_expire=2021-11-16T21:23:49.440271+0000 has v0 lc 19557616
debug -165> 2021-11-16T21:23:44.567+0000 7f1d716db700 4 set_mon_vals no callback set
debug -164> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -163> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -162> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -161> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557617) is_readable = 1 - now=2021-11-16T21:23:44.569810+0000 lease_expire=2021-11-16T21:23:49.531775+0000 has v0 lc 19557617
debug -160> 2021-11-16T21:23:45.684+0000 7f1d716db700 4 set_mon_vals no callback set
debug -159> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -158> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -157> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -156> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557618) is_readable = 1 - now=2021-11-16T21:23:45.686877+0000 lease_expire=2021-11-16T21:23:50.648710+0000 has v0 lc 19557618
debug -155> 2021-11-16T21:23:46.776+0000 7f1d716db700 4 set_mon_vals no callback set
debug -154> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -153> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -152> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -151> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557619) is_readable = 1 - now=2021-11-16T21:23:46.778635+0000 lease_expire=2021-11-16T21:23:51.740053+0000 has v0 lc 19557619
debug -150> 2021-11-16T21:23:46.876+0000 7f1d716db700 4 set_mon_vals no callback set
debug -149> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -148> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -147> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -146> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557620) is_readable = 1 - now=2021-11-16T21:23:46.878541+0000 lease_expire=2021-11-16T21:23:51.839977+0000 has v0 lc 19557620
debug -145> 2021-11-16T21:23:47.243+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #531955. Immutable memtables: 0.

debug -144> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.244227) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
debug -143> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 531955

debug -142> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827244296, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 683, "num_deletes": 506, "total_data_size": 1462899, "memory_usage": 1475944, "flush_reason": "Manual Compaction"}
debug -141> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #531956: started
debug -140> 2021-11-16T21:23:47.243+0000 7f1d716db700 4 set_mon_vals no callback set
debug -139> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -138> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -137> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -136> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557621) is_readable = 1 - now=2021-11-16T21:23:47.245909+0000 lease_expire=2021-11-16T21:23:52.207163+0000 has v0 lc 19557621
debug -135> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827293833, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 531956, "file_size": 966005, "table_properties": {"data_size": 962689, "index_size": 997, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1477, "raw_key_size": 12966, "raw_average_key_size": 22, "raw_value_size": 954089, "raw_average_value_size": 1667, "num_data_blocks": 29, "num_entries": 572, "num_deletions": 506, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097809, "oldest_key_time": 1637097809, "file_creation_time": 1637097827}}
debug -134> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:401] [default] [JOB 3] Level-0 flush table #531956: 966005 bytes OK
debug -133> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.293881) [memtable_list.cc:447] [default] Level-0 commit table #531956 started
debug -132> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327047) [memtable_list.cc:503] [default] Leve
l-0 commit table #531956: memtable #1 done
debug -131> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327058) EVENT_LOG_v1 {"time_micros": 1637097827327054, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0}
debug -130> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327073) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[3 0 0 0 0 0 2] max score 0.75

debug -129> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 3] Try to delete WAL files size 1458128, prev total WAL file size 1458455, number of live WAL files 2.

debug -128> 2021-11-16T21:23:47.326+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -127> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327469) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'logm .. 'logm; will stop at (end)

debug -126> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 4] Compacting 3@0 + 2@6 files to L6, score -1.00
debug -125> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 3 Base level 0, inputs: [531956(943KB) 531950(148KB) 531947(7726KB)], [531944(64MB) 531945(36MB)]

debug -124> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827327529, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [531956, 531950, 531947], "files_L6": [531944, 531945], "score": -1, "input_data_size": 114546999}
debug -123> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531957: 3454 keys, 67198385 bytes
debug -122> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827785630, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531957, "file_size": 67198385, "table_properties": {"data_size": 67136484, "index_size": 52412, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 8645, "raw_key_size": 76781, "raw_average_key_size": 22, "raw_value_size": 67047935, "raw_average_value_size": 19411, "num_data_blocks": 1555, "num_entries": 3454, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}}
debug -121> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531958: 1281 keys, 32015375 bytes
debug -120> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828044065, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531958, "file_size": 32015375, "table_properties": {"data_size": 31968485, "index_size": 42777, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 3269, "raw_key_size": 29915, "raw_average_key_size": 23, "raw_value_size": 31919151, "raw_average_value_size": 24917, "num_data_blocks": 1155, "num_entries": 1281, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}}
debug -119> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 4] Compacted 3@0 + 2@6 files to L6 => 99213760 bytes
debug -118> 2021-11-16T21:23:48.066+0000 7f1d716db700 4 set_mon_vals no callback set
debug -117> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -116> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -115> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -114> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557622) is_readable = 1 - now=2021-11-16T21:23:48.069418+0000 lease_expire=2021-11-16T21:23:52.960557+0000 has v0 lc 19557622
debug -113> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085683) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 159.9 rd, 138.5 wr, level 6, files in(3, 2) out(2) MB in(8.6, 100.6) out(94.6), read-write-amplify(23.7) write-amplify(11.0) OK, records in: 5811, records dropped: 1076 output_compression: NoCompression

debug -112> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085701) EVENT_LOG_v1 {"time_micros": 1637097828085695, "job": 4, "event": "compaction_finished", "compaction_time_micros": 716586, "compaction_time_cpu_micros": 83813, "output_level": 6, "num_output_files": 2, "total_output_size": 99213760, "num_input_records": 5811, "num_output_records": 4735, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]}
debug -111> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085923, "job": 4, "event": "table_file_deletion", "file_number": 531956}
debug -110> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085969, "job": 4, "event": "table_file_deletion", "file_number": 531950}
debug -109> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828086487, "job": 4, "event": "table_file_deletion", "file_number": 531947}
debug -108> 2021-11-16T21:23:48.088+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828089419, "job": 4, "event": "table_file_deletion", "file_number": 531945}
debug -107> 2021-11-16T21:23:48.093+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828094233, "job": 4, "event": "table_file_deletion", "file_number": 531944}
debug -106> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -105> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -104> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -103> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -102> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc
:1443] [default] Manual compaction starting
debug -101> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -100> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -99> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -98> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -97> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -96> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting
debug -95> 2021-11-16T21:23:49.260+0000 7f1d716db700 4 set_mon_vals no callback set
debug -94> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -93> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -92> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -91> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557623) is_readable = 1 - now=2021-11-16T21:23:49.262917+0000 lease_expire=2021-11-16T21:23:54.223482+0000 has v0 lc 19557623
debug -90> 2021-11-16T21:23:50.002+0000 7f1d716db700 4 set_mon_vals no callback set
debug -89> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -88> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -87> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -86> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557624) is_readable = 1 - now=2021-11-16T21:23:50.004764+0000 lease_expire=2021-11-16T21:23:54.973508+0000 has v0 lc 19557624
debug -85> 2021-11-16T21:23:51.102+0000 7f1d716db700 4 set_mon_vals no callback set
debug -84> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -83> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -82> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -81> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557625) is_readable = 1 - now=2021-11-16T21:23:51.104896+0000 lease_expire=2021-11-16T21:23:56.064982+0000 has v0 lc 19557625
debug -80> 2021-11-16T21:23:51.202+0000 7f1d716db700 4 set_mon_vals no callback set
debug -79> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -78> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -77> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -76> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557626) is_readable = 1 - now=2021-11-16T21:23:51.204980+0000 lease_expire=2021-11-16T21:23:56.164769+0000 has v0 lc 19557626
debug -75> 2021-11-16T21:23:52.261+0000 7f1d716db700 4 set_mon_vals no callback set
debug -74> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -73> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -72> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -71> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557627) is_readable = 1 - now=2021-11-16T21:23:52.263508+0000 lease_expire=2021-11-16T21:23:57.231263+0000 has v0 lc 19557627
debug -70> 2021-11-16T21:23:52.352+0000 7f1d716db700 4 set_mon_vals no callback set
debug -69> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -68> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -67> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -66> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557628) is_readable = 1 - now=2021-11-16T21:23:52.355232+0000 lease_expire=2021-11-16T21:23:57.322723+0000 has v0 lc 19557628
debug -65> 2021-11-16T21:23:53.436+0000 7f1d716db700 4 set_mon_vals no callback set
debug -64> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -63> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -62> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -61> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557629) is_readable = 1 - now=2021-11-16T21:23:53.438872+0000 lease_expire=2021-11-16T21:23:58.406301+0000 has v0 lc 19557629
debug -60> 2021-11-16T21:23:54.519+0000 7f1d716db700 4 set_mon_vals no callback set
debug -59> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -58> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -57> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -56> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557630) is_readable = 1 - now=2021-11-16T21:23:54.522319+0000 lease_expire=2021-11-16T21:23:59.489316+0000 has v0 lc 19557630
debug -55> 2021-11-16T21:23:55.595+0000 7f1d716db700 4 set_mon_vals no callback set
debug -54> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -53> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -52> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -51> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557631) is_readable = 1 - now=2021-11-16T21:23:55.597582+0000 lease_expire=2021-11-16T21:24:00.564365+0000 has v0 lc 19557631
debug -50> 2021-11-16T21:23:56.670+0000 7f1d716db700 4 set_mon_vals no callback set
debug -49> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -48> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -47> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -46> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557632) is_readable = 1 - now=2021-11-16T21:23:56.672809+0000 lease_expire=2021-11-16T21:24:01.638988+0000 has v0 lc 19557632
debug -45> 2021-11-16T21:23:57.270+0000 7f1d716db700 4 set_mon_vals no callback set
debug -44> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -43> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -42> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -41> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557633) is_readable = 1 - now=2021-11-16T21:23:57.272872+0000 lease_expire=2021-11-16T21:24:02.238966+0000 has v0 lc 19557633
debug -40> 2021-11-16T21:23:57.753+0000 7f1d716db700 4 set_mon_vals no callback set
debug -39> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -38> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -37> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -36> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557634) is_readable = 1 - now=2021-11-16T21:23:57.756400+0000 lease_expire=2021-11-16T21:24:02.722461+0000 has v0 lc 19557634
debug -35> 2021-11-16T21:23:58.829+0000 7f1d716db700 4 set_mon_vals no callback set
debug -34> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -33> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -32> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -31> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557635) is_readable = 1 - now=2021-11-16T21:23:58.831535+0000 lease_expire=2021-11-16T21:24:03.797240+0000 has v0 lc 19557635
debug -30> 2021-11-16T21:23:58.920+0000 7f1d716db700 4 set_mon_vals no callback set
debug -29> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -28> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -27> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -26> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557636) is_readable = 1 - now=2021-11-16T21:23:58.922888+0000 lease_expire=2021-11-16T21:24:03.888665+0000 has v0 lc 19557636
debug -25> 2021-11-16T21:23:59.354+0000 7f1d716db700 4 set_mon_vals no callback set
debug -24> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -23> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -22> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -21> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557637) is_readable = 1 - now=2021-11-16T21:23:59.356709+0000 lease_expire=2021-11-16T21:24:04.321986+0000 has v0 lc 19557637
debug -20> 2021-11-16T21:24:00.462+0000 7f1d716db700 4 set_mon_vals no callback set
debug -19> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -18> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -17> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -16> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557638) is_readable = 1 - now=2021-11-16T21:24:00.465271+0000 lease_expire=2021-11-16T21:24:05.430188+0000 has v0 lc 19557638
debug -15> 2021-11-16T21:24:00.554+0000 7f1d716db700 4 set_mon_vals no callback set
debug -14> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -13> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -12> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -11> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557639) is_readable = 1 - now=2021-11-16T21:24:00.556869+0000 lease_expire=2021-11-16T21:24:05.521766+0000 has v0 lc 19557639
debug -10> 2021-11-16T21:24:01.629+0000 7f1d716db700 4 set_mon_vals no callback set
debug -9> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500
debug -8> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106]
debug -7> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable
debug -6> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.632342+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -5> 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0)
debug -4> 2021-11-16T21:24:01.873+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875099+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -3> 2021-11-16T21:24:01.873+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x56078525b380 auth_reply(proto 2 0 (0) Success) v1
debug -2> 2021-11-16T21:24:01.874+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875450+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640
debug -1> 2021-11-16T21:24:01.874+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eb40 0x560785259d40 auth_reply(proto 2 0 (0) Success) v1
debug 0> 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) **
in thread 7f1d716db700 thread_name:ms_dispatch

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8]
8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045]
9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6]
10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329]
11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258]
12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc]
15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71]
17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a]
18: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---

0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
139764358260480 / rstore_compact
139764375045888 / ms_dispatch
139764391831296 / rocksdb:dump_st
139764417009408 / msgr-worker-0
139764433794816 / ms_dispatch
139764450580224 / cpu_tp
139764467365632 / cpu_tp
139764475758336 / safe_timer
139764526114560 / rocksdb:high0
139764534507264 / rocksdb:low0
139764560197376 / admin_socket
139764813260544 / ceph-mon
max_recent 10000
max_new 10000
log_file /var/lib/ceph/crash/2021-11-16T21:24:01.877284Z_ec6af590-8752-4032-b581-ebb6ced84e72/log
--- end dump of recent events ---

meta file from the crash dump:

[ceph: root@cn03 2021-11-16T21:24:41.375317Z_4cc0b19c-e1b1-4f7c-8ed4-780a7e64dd46]# cat meta
{
"crash_id": "2021-11-16T21:24:41.375317Z_4cc0b19c-e1b1-4f7c-8ed4-780a7e64dd46",
"timestamp": "2021-11-16T21:24:41.375317Z",
"process_name": "ceph-mon",
"entity_name": "mon.cn03",
"ceph_version": "16.2.4",
"utsname_hostname": "cn03.ceph.la1.clx.corp",
"utsname_sysname": "Linux",
"utsname_release": "5.15.1-1.el8.elrepo.x86_64",
"utsname_version": "#1 SMP Thu Nov 4 19:14:37 EDT 2021",
"utsname_machine": "x86_64",
"os_name": "CentOS Linux",
"os_id": "centos",
"os_version_id": "8",
"os_version": "8",
"backtrace": [
"/lib64/libpthread.so.0(+0x12b20) [0x7f9fea30db20]",
"gsignal()",
"abort()",
"/lib64/libstdc++.so.6(+0x9009b) [0x7f9fe992b09b]",
"/lib64/libstdc++.so.6(+0x9653c) [0x7f9fe993153c]",
"/lib64/libstdc++.so.6(+0x96597) [0x7f9fe9931597]",
"/lib64/libstdc++.so.6(+0x967f8) [0x7f9fe99317f8]",
"/lib64/libstdc++.so.6(+0x92045) [0x7f9fe992d045]",
"/usr/bin/ceph-mon(+0x4d8da6) [0x5610db0d8da6]",
"(MDSMonitor::check_sub(Subscription*)+0x819) [0x5610db0cf329]",
"(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x5610daec1258]",
"(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x5610daee72ed]",
"(Monitor::_ms_dispatch(Message*)+0x670) [0x5610daee8910]",
"(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5610daf16fdc]",
"(DispatchQueue::entry()+0x126a) [0x7f9feca4db1a]",
"(DispatchQueue::DispatchThread::entry()+0x11) [0x7f9fecafdb71]",
"/lib64/libpthread.so.0(+0x814a) [0x7f9fea30314a]",
"clone()"
]
}

Thank you
-jeremy

> On Tuesday, Nov 16, 2021 at 5:58 AM, 胡 玮文 <huww98@xxxxxxxxxxx (mailto:huww98@xxxxxxxxxxx)> wrote:
>
> Hi Jeremy. Since you say the mons fail, could you share the logs of the failing mons? It is hard to diagnostic with little information.
>
>
>
>
>
> 发件人: Jeremy Hansen (mailto:jeremy@xxxxxxxxxx)
> 发送时间: 2021年11月16日 19:27
> 收件人: ceph-users (mailto:ceph-users@xxxxxxx)
> 主题: [ceph-users] Re: mons fail as soon as I attempt to mount
>
>
>
>
>
>
> No ideas on this? It has me stumped. I’m going to revert my kernel upgrades if there isn’t a more logical explanation.
>
> Thanks
>
> > On Monday, Nov 15, 2021 at 2:33 AM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote:
> > This post references a kernel issue:
> >
> > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/6JDAPD5IR46JI6R6YGWQORDJTZ5Z2FIU/
> >
> > I recently updated my ceph nodes to 5.15.1. Could this be my issue?
> >
> > -jeremy
> >
> >
> >
> >
> > > On Sunday, Nov 14, 2021 at 4:37 PM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote:
> > > I’m trying to mount a cephfs volume from a new machine. For some reason, it looks like all the mons fail when attempting to mount:
> > >
> > > [root@btc04 ~]# mount -t ceph :/ /mnt/ceph -o name=btc
> > > mount error: no mds server is up or the cluster is laggy
> > > [root@btc04 ~]# rpm -qa | grep ceph
> > > python3-cephfs-16.2.4-0.el8.x86_64
> > > ceph-common-16.2.4-0.el8.x86_64
> > > cephadm-16.2.4-0.el8.noarch
> > > python3-ceph-argparse-16.2.4-0.el8.x86_64
> > > libcephfs2-16.2.4-0.el8.x86_64
> > > python3-ceph-common-16.2.4-0.el8.x86_64
> > >
> > >
> > > [ 51.105212] libceph: loaded (mon/osd proto 15/24)
> > > [ 51.145564] ceph: loaded (mds proto 32)
> > > [ 51.164266] libceph: mon3 (1)192.168.30.14:6789 session established
> > > [ 70.199453] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state OPEN)
> > > [ 70.199464] libceph: mon3 (1)192.168.30.14:6789 session lost, hunting for new mon
> > > [ 70.204400] libceph: mon0 (1)192.168.30.11:6789 session established
> > > [ 70.771652] libceph: mon0 (1)192.168.30.11:6789 socket closed (con state OPEN)
> > > [ 70.771670] libceph: mon0 (1)192.168.30.11:6789 session lost, hunting for new mon
> > > [ 70.774588] libceph: mon4 (1)192.168.30.15:6789 session established
> > > [ 71.234037] libceph: mon4 (1)192.168.30.15:6789 socket closed (con state OPEN)
> > > [ 71.234055] libceph: mon4 (1)192.168.30.15:6789 session lost, hunting for new mon
> > > [ 77.904722] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > > [ 78.160614] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > > [ 78.664602] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > > [ 79.824787] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > > [ 81.808526] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > > [ 85.840430] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER)
> > >
> > >
> > > Not really sure why…
> > >
> > > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.data all
> > > size: 4
> > > min_size: 2
> > > pg_num: 32
> > > pgp_num: 32
> > > crush_rule: replicated_rule
> > > hashpspool: true
> > > nodelete: false
> > > nopgchange: false
> > > nosizechange: false
> > > write_fadvise_dontneed: false
> > > noscrub: false
> > > nodeep-scrub: false
> > > use_gmt_hitset: 1
> > > fast_read: 0
> > > pg_autoscale_mode: on
> > > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.meta all
> > > size: 4
> > > min_size: 2
> > > pg_num: 32
> > > pgp_num: 32
> > > crush_rule: replicated_rule
> > > hashpspool: true
> > > nodelete: false
> > > nopgchange: false
> > > nosizechange: false
> > > write_fadvise_dontneed: false
> > > noscrub: false
> > > nodeep-scrub: false
> > > use_gmt_hitset: 1
> > > fast_read: 0
> > > recovery_priority: 5
> > > pg_autoscale_mode: on
> > > pg_num_min: 16
> > > pg_autoscale_bias: 4
> > >
> > >
> > > The cluster becomes unhealthy but then clears shortly after the client times out.
> > >
> > > [ceph: root@cn01 /]# ceph -s
> > > cluster:
> > > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
> > > health: HEALTH_OK
> > >
> > > services:
> > > mon: 5 daemons, quorum cn05,cn02,cn03,cn04,cn01 (age 6m)
> > > mgr: cn05.vpuwau(active, since 6d), standbys: cn02.arszct
> > > mds: 2/2 daemons up, 4 standby
> > > osd: 35 osds: 35 up (since 2d), 35 in (since 6d)
> > >
> > > data:
> > > volumes: 2/2 healthy
> > > pools: 6 pools, 289 pgs
> > > objects: 6.73M objects, 4.3 TiB
> > > usage: 17 TiB used, 108 TiB / 126 TiB avail
> > > pgs: 289 active+clean
> > >
> > > io:
> > > client: 0 B/s rd, 105 KiB/s wr, 2 op/s rd, 13 op/s wr
> > >
> > >
> > >
> > > -jeremy
> > >
> > >
> > >
>
>
>
>
>
>
>

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx