A quick search gives https://tracker.ceph.com/issues/51077. So this is a known bug, and have been fixed in the latest version. Please refer to the tracker for details and workaround. 在 2021年11月17日，05:31，Jeremy Hansen <jeremy@xxxxxxxxxx> 写道： Thank you. I was able to gather more logs from one of the failing mon processes: cluster 2021-11-16T21:23:58.114701+0000 mgr.cn02.arszct (mgr.8444129) 66657 : cluster [DBG] pgmap v66699: 289 pgs: 289 active+clean; 4.4 TiB data, 17 TiB used, 108 TiB / 126 TiB avail; 50 KiB/s rd, 123 KiB/s wr, 74 op/s audit 2021-11-16T21:23:59.322056+0000 mon.cn05 (mon.0) 19 : audit [INF] from='mgr.8444129 ' entity='mgr.cn02.arszct' cluster 2021-11-16T21:24:00.116045+0000 mgr.cn02.arszct (mgr.8444129) 66658 : cluster [DBG] pgmap v66700: 289 pgs: 289 active+clean; 4.4 TiB data, 17 TiB used, 108 TiB / 126 TiB avail; 11 KiB/s rd, 130 KiB/s wr, 30 op/s debug 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0) terminate called after throwing an instance of 'std::out_of_range' what(): map::at *** Caught signal (Aborted) ** in thread 7f1d716db700 thread_name:ms_dispatch ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8] 8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045] 9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6] 10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329] 11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258] 12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed] 13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910] 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc] 15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71] 17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a] 18: clone() debug 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) ** in thread 7f1d716db700 thread_name:ms_dispatch ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8] 8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045] 9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6] 10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329] 11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258] 12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed] 13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910] 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc] 15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71] 17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a] 18: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- debug -734> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command assert hook 0x5607840e8530 debug -733> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command abort hook 0x5607840e8530 debug -732> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command leak_some_memory hook 0x5607840e8530 debug -731> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_dump hook 0x5607840e8530 debug -730> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 1 hook 0x5607840e8530 debug -729> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf dump hook 0x5607840e8530 debug -728> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_schema hook 0x5607840e8530 debug -727> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram dump hook 0x5607840e8530 debug -726> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 2 hook 0x5607840e8530 debug -725> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf schema hook 0x5607840e8530 debug -724> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram schema hook 0x5607840e8530 debug -723> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf reset hook 0x5607840e8530 debug -722> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config show hook 0x5607840e8530 debug -721> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config help hook 0x5607840e8530 debug -720> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config set hook 0x5607840e8530 debug -719> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config unset hook 0x5607840e8530 debug -718> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config get hook 0x5607840e8530 debug -717> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff hook 0x5607840e8530 debug -716> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff get hook 0x5607840e8530 debug -715> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command injectargs hook 0x5607840e8530 debug -714> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log flush hook 0x5607840e8530 debug -713> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log dump hook 0x5607840e8530 debug -712> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log reopen hook 0x5607840e8530 debug -711> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_mempools hook 0x560784188328 debug -710> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 set uid:gid to 167:167 (ceph:ceph) debug -709> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 8 debug -708> 2021-11-16T21:23:28.656+0000 7f1d880be700 0 pidfile_write: ignore empty --pid-file debug -707> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) init /var/run/ceph/ceph-mon.cn03.asok debug -706> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) bind_and_listen /var/run/ceph/ceph-mon.cn03.asok debug -705> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command 0 hook 0x5607840ec108 debug -704> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command version hook 0x5607840ec108 debug -703> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command git_version hook 0x5607840ec108 debug -702> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command help hook 0x5607840e8230 debug -701> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command get_command_descriptions hook 0x5607840e81e0 debug -700> 2021-11-16T21:23:28.656+0000 7f1d78f67700 5 asok(0x56078419c000) entry start debug -699> 2021-11-16T21:23:28.660+0000 7f1d880be700 0 load: jerasure load: lrc load: isa debug -698> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true debug -697> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432 debug -696> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression debug -695> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true debug -694> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432 debug -693> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression debug -692> 2021-11-16T21:23:28.661+0000 7f1d880be700 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -691> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: RocksDB version: 6.8.1 debug -690> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -689> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compile date May 13 2021 debug -688> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: DB SUMMARY debug -687> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: CURRENT file: CURRENT debug -686> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: IDENTITY file: IDENTITY debug -685> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: MANIFEST file: MANIFEST-531948 size: 302 Bytes debug -684> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-cn03/store.db dir, Total Num: 3, files: 531944.sst 531945.sst 531947.sst debug -683> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-cn03/store.db: 531949.log size: 214103 ; debug -682> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.error_if_exists: 0 debug -681> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_if_missing: 0 debug -680> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_checks: 1 debug -679> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.env: 0x5607837a11c0 debug -678> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.fs: Posix File System debug -677> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.info_log: 0x5607841d78c0 debug -676> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_file_opening_threads: 16 debug -675> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.statistics: (nil) debug -674> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_fsync: 0 debug -673> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_log_file_size: 0 debug -672> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -671> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_file_time_to_roll: 0 debug -670> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.keep_log_file_num: 1000 debug -669> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.recycle_log_file_num: 0 debug -668> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_fallocate: 1 debug -667> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_reads: 0 debug -666> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_writes: 0 debug -665> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_reads: 0 debug -664> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -663> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_missing_column_families: 0 debug -662> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_log_dir: debug -661> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-cn03/store.db debug -660> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_cache_numshardbits: 6 debug -659> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_subcompactions: 1 debug -658> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_flushes: -1 debug -657> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -656> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -655> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -654> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -653> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -652> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.advise_random_on_open: 1 debug -651> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_write_buffer_size: 0 debug -650> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_manager: 0x5607841ddf80 debug -649> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -648> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 debug -647> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -646> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_adaptive_mutex: 0 debug -645> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limiter: (nil) debug -644> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -643> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_recovery_mode: 2 debug -642> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_thread_tracking: 0 debug -641> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_pipelined_write: 0 debug -640> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.unordered_write: 0 debug -639> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -638> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -637> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -636> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -635> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.row_cache: None debug -634> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_filter: None debug -633> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -632> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_ingest_behind: 0 debug -631> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.preserve_deletes: 0 debug -630> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.two_write_queues: 0 debug -629> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manual_wal_flush: 0 debug -628> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.atomic_flush: 0 debug -627> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -626> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.persist_stats_to_disk: 0 debug -625> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -624> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_readahead_size: 0 debug -623> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -622> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_jobs: 2 debug -621> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_compactions: -1 debug -620> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -619> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 debug -618> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delayed_write_rate : 16777216 debug -617> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_total_wal_size: 0 debug -616> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -615> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_dump_period_sec: 600 debug -614> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_persist_period_sec: 600 debug -613> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -612> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_open_files: -1 debug -611> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bytes_per_sync: 0 debug -610> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -609> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -608> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_readahead_size: 0 debug -607> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compression algorithms supported: debug -606> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -605> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTD supported: 0 debug -604> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kXpressCompression supported: 0 debug -603> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4HCCompression supported: 1 debug -602> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4Compression supported: 1 debug -601> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kBZip2Compression supported: 0 debug -600> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZlibCompression supported: 1 debug -599> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kSnappyCompression supported: 1 debug -598> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -597> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 debug -596> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -595> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -594> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.merge_operator: debug -593> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter: None debug -592> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter_factory: None debug -591> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_factory: SkipListFactory debug -590> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_factory: BlockBasedTable debug -589> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607840ec128) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x560784125010 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -588> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_size: 33554432 debug -587> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number: 2 debug -586> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression: NoCompression debug -585> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression: Disabled debug -584> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.prefix_extractor: nullptr debug -583> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -582> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.num_levels: 7 debug -581> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -580> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -579> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -578> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -577> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -576> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -575> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -574> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -573> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -572> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.window_bits: -14 debug -571> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.level: 32767 debug -570> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.strategy: 0 debug -569> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -568> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -567> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.enabled: false debug -566> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -565> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -564> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -563> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_base: 67108864 debug -562> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_multiplier: 1 debug -561> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -560> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 debug -559> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -558> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -557> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -556> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -555> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -554> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -553> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -552> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -551> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -550> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -549> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.arena_block_size: 4194304 debug -548> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -547> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -546> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -545> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.disable_auto_compactions: 0 debug -544> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -543> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -542> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -541> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -540> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -539> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -538> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -537> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -536> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -535> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -534> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_properties_collectors: debug -533> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_support: 0 debug -532> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -531> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -530> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -529> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_huge_page_size: 0 debug -528> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bloom_locality: 0 debug -527> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_successive_merges: 0 debug -526> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -525> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_file_checks: 0 debug -524> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.force_consistency_checks: 0 debug -523> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.report_bg_io_stats: 0 debug -522> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.ttl: 2592000 debug -521> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -520> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 succeeded,manifest_file_number is 531948, next_file_number is 531950, last_sequence is 201837822, log_number is 531943,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -519> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 531943 debug -518> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808663528, "job": 1, "event": "recovery_started", "log_files": [531949]} debug -517> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #531949 mode 2 debug -516> 2021-11-16T21:23:28.663+0000 7f1d880be700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. debug -515> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808697864, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 531950, "file_size": 151829, "table_properties": {"data_size": 150555, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": 759, "raw_average_key_size": 25, "raw_value_size": 149707, "raw_average_value_size": 4990, "num_data_blocks": 8, "num_entries": 30, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097808, "oldest_key_time": 3, "file_creation_time": 0}} debug -514> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: [version_set.cc:3826] Creating manifest 531951 debug -513> 2021-11-16T21:23:28.755+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808756165, "job": 1, "event": "recovery_finished"} debug -512> 2021-11-16T21:23:28.797+0000 7f1d880be700 4 rocksdb: DB pointer 0x5607841e9800 debug -511> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- debug -510> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:851] ** DB Stats ** Uptime(secs): 0.1 total, 0.1 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Uptime(secs): 0.1 total, 0.1 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Uptime(secs): 0.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** debug -509> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -508> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -507> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -506> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: none debug -505> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -504> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -503> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -502> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -501> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -500> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -499> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -498> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -497> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -496> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -495> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -494> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -493> 2021-11-16T21:23:28.797+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -492> 2021-11-16T21:23:28.798+0000 7f1d880be700 0 starting mon.cn03 rank 2 at public addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] at bind addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon_data /var/lib/ceph/mon/ceph-cn03 fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d debug -491> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -490> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -489> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -488> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: none debug -487> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -486> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -485> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -484> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -483> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -482> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -481> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -480> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -479> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -478> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -477> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -476> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -475> 2021-11-16T21:23:28.798+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -474> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx debug -473> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx debug -472> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) debug -471> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) debug -470> 2021-11-16T21:23:28.799+0000 7f1d880be700 1 mon.cn03@-1(???) e69 preinit fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d debug -469> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 new map debug -468> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 print_map e432 enable_multiple, ever_enabled_multiple: 1,1 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 2 Filesystem 'coldlogix' (2) fs_name coldlogix epoch 402 flags 12 created 2021-06-27T06:32:25.086015+0000 modified 2021-11-15T08:30:22.670221+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 17247 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=8444199} failed damaged stopped data_pools [12] metadata_pool 11 inline_data disabled balancer standby_count_wanted 1 [mds.coldlogix.cn01.dirzwe{0:8444199} state up:active seq 77 join_fscid=2 addr [v2:192.168.30.11:6850/3427514521,v1:192.168.30.11:6851/3427514521]] Filesystem 'btc' (5) fs_name btc epoch 423 flags 12 created 2021-11-15T08:49:14.520956+0000 modified 2021-11-15T08:49:25.936407+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 17308 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=8534156} failed damaged stopped data_pools [19] metadata_pool 18 inline_data disabled balancer standby_count_wanted 1 [mds.btc.cn04.pyuaxe{0:8534156} state up:active seq 5 join_fscid=5 addr [v2:192.168.30.14:6800/4253241567,v1:192.168.30.14:6801/4253241567]] Standby daemons: [mds.btc.cn06.hqmwsx{-1:8534166} state up:standby seq 3 join_fscid=5 addr [v2:192.168.30.16:6848/446743436,v1:192.168.30.16:6849/446743436]] [mds.coldlogix.cn03.lyxzhp{-1:8604119} state up:standby seq 4 join_fscid=2 addr [v2:192.168.30.13:6800/2645334932,v1:192.168.30.13:6801/2645334932]] debug -467> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 3314933000854323200, adjusting msgr requires debug -466> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -465> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -464> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -463> 2021-11-16T21:23:28.800+0000 7f1d880be700 1 mon.cn03@-1(???).paxosservice(auth 10008..10244) refresh upgraded, format 0 -> 3 debug -462> 2021-11-16T21:23:28.800+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e0 loading version 500 debug -461> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 active server: [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106](8444129) debug -460> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 mkfs or daemon transitioned to available, loading commands debug -459> 2021-11-16T21:23:28.802+0000 7f1d880be700 4 set_mon_vals no callback set debug -458> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals auth_allow_insecure_global_id_reclaim = false debug -457> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 debug -456> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals mon_allow_pool_delete = true debug -455> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals rbd_move_to_trash_on_remove = true debug -454> 2021-11-16T21:23:28.803+0000 7f1d880be700 4 set_mon_vals no callback set debug -453> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 aut h: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -452> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command compact hook 0x5607840e89a0 debug -451> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command smart hook 0x5607840e89a0 debug -450> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command mon_status hook 0x5607840e89a0 debug -449> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command heap hook 0x5607840e89a0 debug -448> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores dump hook 0x5607840e89a0 debug -447> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores reset hook 0x5607840e89a0 debug -446> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sync_force hook 0x5607840e89a0 debug -445> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hint hook 0x5607840e89a0 debug -444> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hintv hook 0x5607840e89a0 debug -443> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum enter hook 0x5607840e89a0 debug -442> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum exit hook 0x5607840e89a0 debug -441> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command ops hook 0x5607840e89a0 debug -440> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sessions hook 0x5607840e89a0 debug -439> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_historic_ops hook 0x5607840e89a0 debug -438> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -437> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -436> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -435> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: none debug -434> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -433> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -432> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -431> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -430> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -429> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -428> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -427> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -426> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -425> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -424> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -423> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -422> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -421> 2021-11-16T21:23:28.804+0000 7f1d880be700 2 mon.cn03@-1(???) e69 init debug -420> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Got map version 500 debug -419> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -418> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -417> 2021-11-16T21:23:28.805+0000 7f1d726dd700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d726dd700' had timed out after 0.000000000s debug -416> 2021-11-16T21:23:28.806+0000 7f1d880be700 0 mon.cn03@-1(probing) e69 my rank is now 2 (was -1) debug -415> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure stats_period=5 debug -414> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure updated stats threshold: 5 debug -413> 2021-11-16T21:23:28.807+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection debug -412> 2021-11-16T21:23:29.213+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection debug -411> 2021-11-16T21:23:29.214+0000 7f1d716db700 0 log_channel(cluster) log [INF] : mon.cn03 calling monitor election debug -410> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client _send_to_mon log to self debug -409> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 debug -408> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client will send 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election debug -407> 2021-11-16T21:23:29.214+0000 7f1d716db700 5 paxos.2).electionLogic(0) start -- can i be leader? debug -406> 2021-11-16T21:23:29.214+0000 7f1d716db700 1 paxos.2).electionLogic(5412) init, last seen epoch 5412 debug -405> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -404> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248762+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -403> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248771+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -402> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.1 debug -401> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 paxos.2).electionLogic(5413) defer to 1 debug -400> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3 debug -399> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 paxos.2).electionLogic(5413) no, we already acked 1 debug -398> 2021-11-16T21:23:30.619+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1 debug -397> 2021-11-16T21:23:33.077+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -396> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.4 debug -395> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 paxos.2).electionLogic(5413) ignoring old propose debug -394> 2021-11-16T21:23:34.411+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3 debug -393> 2021-11-16T21:23:34.440+0000 7f1d716db700 5 paxos.2).electionLogic(5415) no, we already acked 1 debug -392> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -391> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5415) handle_propose from mon.0 debug -390> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 paxos.2).electionLogic(5415) ignoring old propose debug -389> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) election period ended debug -388> 2021-11-16T21:23:35.249+0000 7f1d73ee070 0 5 paxos.2).electionLogic(5415) start -- can i be leader? debug -387> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 1 paxos.2).electionLogic(5415) init, last seen epoch 5415, mid-election, bumping debug -386> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_ack from mon.3 debug -385> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) so far i have { mon.2: features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) } debug -384> 2021-11-16T21:23:35.392+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_propose from mon.0 debug -383> 2021-11-16T21:23:35.424+0000 7f1d716db700 5 paxos.2).electionLogic(5419) defer to 0 debug -382> 2021-11-16T21:23:36.621+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1 debug -381> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256876+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -380> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256899+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -379> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256913+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -378> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256925+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -377> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257153+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -376> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257169+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -375> 2021-11-16T21:23:38.342+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5419) handle_victory from mon.0 quorum_features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) debug -374> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_quorum_to_compatset_features debug -373> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_monmap_to_compatset_features debug -372> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382930+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -371> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382942+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -370> 2021-11-16T21:23:38.450+0000 7f1d716db700 4 set_mon_vals no callback set debug -369> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -368> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -367> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -366> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451523+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -365> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451536+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -364> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451571+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -363> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451596+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -362> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451620+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -361> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451641+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -360> 2021-11-16T21:23:38.452+0000 7f1d736df700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d736df700' had timed out after 0.000000000s debug -359> 2021-11-16T21:23:38.538+0000 7f1d706d9700 1 mon.cn03@2(peon) e69 handle_auth_request failed to assign global_id debug -358> 2021-11-16T21:23:38.541+0000 7f1d716db700 4 set_mon_vals no callback set debug -357> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -356> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -355> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -354> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557605) is_readable = 1 - now=2021-11-16T21:23:38.542983+0000 lease_expire=2021-11-16T21:23:43.507566+0000 has v0 lc 19557605 debug -353> 2021-11-16T21:23:38.635+0000 7f1d716db700 4 set_mon_vals no callback set debug -352> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -351> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -350> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -349> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557606) is_readable = 1 - now=2021-11-16T21:23:38.636621+0000 lease_expire=2021-11-16T21:23:43.603066+0000 has v0 lc 19557606 debug -348> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client handle_log_ack log(last 1) v1 debug -347> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client logged 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election debug -346> 2021-11-16T21:23:38.725+0000 7f1d716db700 4 set_mon_vals no callback set debug -345> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -344> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -343> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -342> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557607) is_readable = 1 - now=2021-11-16T21:23:38.726705+0000 lease_expire=2021-11-16T21:23:43.690619+0000 has v0 lc 19557607 debug -341> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783222+0000 lease_expire=1970-01-01T00:00:00.000000+0000 h as v0 lc 19557607 debug -340> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783242+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607 debug -339> 2021-11-16T21:23:38.841+0000 7f1d716db700 4 set_mon_vals no callback set debug -338> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -337> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -336> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -335> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.843331+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -334> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).elector(5420) begin_peer_ping against 3 debug -333> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944355+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -332> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853ebef0 0x5607858deb60 auth_reply(proto 2 0 (0) Success) v1 debug -331> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944811+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -330> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858df380 auth_reply(proto 2 0 (0) Success) v1 debug -329> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984545+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -328> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x5607858de820 auth_reply(proto 2 0 (0) Success) v1 debug -327> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984931+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -326> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851533b0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1 debug -325> 2021-11-16T21:23:39.026+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.026942+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -324> 2021-11-16T21:23:39.026+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x5607858de9c0 auth_reply(proto 2 0 (0) Success) v1 debug -323> 2021-11-16T21:23:39.027+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.027319+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -322> 2021-11-16T21:23:39.027+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x5607858deea0 auth_reply(proto 2 0 (0) Success) v1 debug -321> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067500+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -320> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x5607858df520 auth_reply(proto 2 0 (0) Success) v1 debug -319> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067791+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -318> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607858de680 auth_reply(proto 2 0 (0) Success) v1 debug -317> 2021-11-16T21:23:39.105+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.105918+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -316> 2021-11-16T21:23:39.105+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858dc9c0 auth_reply(proto 2 0 (0) Success) v1 debug -315> 2021-11-16T21:23:39.106+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.106361+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -314> 2021-11-16T21:23:39.106+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607858dc820 auth_reply(proto 2 0 (0) Success) v1 debug -313> 2021-11-16T21:23:39.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.217724+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -312> 2021-11-16T21:23:39.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e4b0 0x560785262b60 auth_reply(proto 2 0 (0) Success) v1 debug -311> 2021-11-16T21:23:39.218+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.218155+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -310> 2021-11-16T21:23:39.218+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x5607852636c0 auth_reply(proto 2 0 (0) Success) v1 debug -309> 2021-11-16T21:23:39.335+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336070+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -308> 2021-11-16T21:23:39.335+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858eb6c0 auth_reply(proto 2 0 (0) Success) v1 debug -307> 2021-11-16T21:23:39.336+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336443+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -306> 2021-11-16T21:23:39.336+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858eab60 auth_reply(proto 2 0 (0) Success) v1 debug -305> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.388745+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -304> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x560785916680 auth_reply(proto 2 0 (0) Success) v1 debug -303> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.389042+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -302> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x560785916340 auth_reply(proto 2 0 (0) Success) v1 debug -301> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415381+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -300> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb770 0x5607858f2b60 auth_reply(proto 2 0 (0) Success) v1 debug -299> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415676+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -298> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607858ec9c0 auth_reply(proto 2 0 (0) Success) v1 debug -297> 2021-11 -16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441170+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -296> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785912d00 auth_reply(proto 2 0 (0) Success) v1 debug -295> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441498+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -294> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785912680 auth_reply(proto 2 0 (0) Success) v1 debug -293> 2021-11-16T21:23:39.547+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.547657+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -292> 2021-11-16T21:23:39.547+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x5607859124e0 auth_reply(proto 2 0 (0) Success) v1 debug -291> 2021-11-16T21:23:39.548+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.548146+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -290> 2021-11-16T21:23:39.548+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607852621a0 auth_reply(proto 2 0 (0) Success) v1 debug -289> 2021-11-16T21:23:39.717+0000 7f1d716db700 4 set_mon_vals no callback set debug -288> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -287> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -286> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -285> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.718923+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -284> 2021-11-16T21:23:39.796+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.796980+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -283> 2021-11-16T21:23:39.796+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x560785263860 auth_reply(proto 2 0 (0) Success) v1 debug -282> 2021-11-16T21:23:39.797+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.797502+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -281> 2021-11-16T21:23:39.797+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858b36c0 auth_reply(proto 2 0 (0) Success) v1 debug -280> 2021-11-16T21:23:39.853+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854122+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -279> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e000 0x56078525eb60 auth_reply(proto 2 0 (0) Success) v1 debug -278> 2021-11-16T21:23:39.854+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854658+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -277> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858eb520 auth_reply(proto 2 0 (0) Success) v1 debug -276> 2021-11-16T21:23:39.999+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.999601+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -275> 2021-11-16T21:23:39.999+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x560785263520 auth_reply(proto 2 0 (0) Success) v1 debug -274> 2021-11-16T21:23:40.000+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.000151+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -273> 2021-11-16T21:23:40.000+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x5607858b3040 auth_reply(proto 2 0 (0) Success) v1 debug -272> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078366+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -271> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607852631e0 auth_reply(proto 2 0 (0) Success) v1 debug -270> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078744+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -269> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eba40 0x560785263380 auth_reply(proto 2 0 (0) Success) v1 debug -268> 2021-11-16T21:23:40.176+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177084+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -267> 2021-11-16T21:23:40.176+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x560785225520 auth_reply(proto 2 0 (0) Success) v1 debug -266> 2021-11-16T21:23:40.177+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177745+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -265> 2021-11-16T21:23:40.177+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x5607858ad520 auth_reply(proto 2 0 (0) Success) v1 debug -264> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -263> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785917380 auth_reply(proto 2 0 (0) Success) v1 debug -262> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217665+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -261> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785916ea0 auth_reply(proto 2 0 (0) Success) v1 debug -260> 2021-11-16T21:23:40.274+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.274992+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -259> 2021-11-16T21:23:40.274+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ec30 0x56078525a820 auth_reply(proto 2 0 (0) Success) v1 debug -258> 2021-11-16T21:23:40.275+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.275761+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -257> 2021-11-16T21:23:40.275+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851521e0 0x560785258d00 auth_reply(proto 2 0 (0) Success) v1 debug -256> 2021-11-16T21:23:40.315+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.315817+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -255> 2021-11-16T21:23:40.315+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x56078591a4e0 auth_reply(proto 2 0 (0) Success) v1 debug -254> 2021-1 1-16T21:23:40.316+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.316340+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -253> 2021-11-16T21:23:40.316+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785259520 auth_reply(proto 2 0 (0) Success) v1 debug -252> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -251> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859131e0 auth_reply(proto 2 0 (0) Success) v1 debug -250> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413824+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -249> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785916000 auth_reply(proto 2 0 (0) Success) v1 debug -248> 2021-11-16T21:23:40.493+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.493816+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -247> 2021-11-16T21:23:40.493+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607859161a0 auth_reply(proto 2 0 (0) Success) v1 debug -246> 2021-11-16T21:23:40.494+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.494344+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -245> 2021-11-16T21:23:40.494+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x560785913380 auth_reply(proto 2 0 (0) Success) v1 debug -244> 2021-11-16T21:23:40.571+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.571680+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -243> 2021-11-16T21:23:40.571+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x560785916340 auth_reply(proto 2 0 (0) Success) v1 debug -242> 2021-11-16T21:23:40.572+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.572418+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -241> 2021-11-16T21:23:40.572+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785913520 auth_reply(proto 2 0 (0) Success) v1 debug -240> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634431+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -239> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ef00 0x5607859164e0 auth_reply(proto 2 0 (0) Success) v1 debug -238> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634908+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -237> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eff0 0x560785913d40 auth_reply(proto 2 0 (0) Success) v1 debug -236> 2021-11-16T21:23:40.696+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.696476+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -235> 2021-11-16T21:23:40.696+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1 debug -234> 2021-11-16T21:23:40.697+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.697195+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -233> 2021-11-16T21:23:40.697+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x560785913860 auth_reply(proto 2 0 (0) Success) v1 debug -232> 2021-11-16T21:23:40.784+0000 7f1d716db700 4 set_mon_vals no callback set debug -231> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -230> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -229> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -228> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557610) is_readable = 1 - now=2021-11-16T21:23:40.785527+0000 lease_expire=2021-11-16T21:23:45.757378+0000 has v0 lc 19557610 debug -227> 2021-11-16T21:23:40.867+0000 7f1d716db700 4 set_mon_vals no callback set debug -226> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -225> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -224> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -223> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.868717+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -222> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910205+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -221> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607858e9ba0 auth_reply(proto 2 0 (0) Success) v1 debug -220> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910749+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -219> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858ea340 auth_reply(proto 2 0 (0) Success) v1 debug -218> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038619+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -217> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x560785910820 auth_reply(proto 2 0 (0) Success) v1 debug -216> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038981+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -215> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785911520 auth_reply(proto 2 0 (0) Success) v1 debug -214> 2021-11-16T21:23:41.082+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.082731+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -213> 2021-11-16T21:23:41.082+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859111e0 auth_reply(proto 2 0 (0) Success) v1 debug -212> 2021-11-16T21:23:41.083+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.083295+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -211> 2021-11-16T21:23:41.083+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785911380 auth_reply(proto 2 0 (0) Success) v1 debug -210> 2021-11-16T21:23:41.167+0000 7f1d716db700 5 mon.cn03@2(peon).pax os(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168120+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -209> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607859109c0 auth_reply(proto 2 0 (0) Success) v1 debug -208> 2021-11-16T21:23:41.168+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168780+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -207> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x560785911040 auth_reply(proto 2 0 (0) Success) v1 debug -206> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264181+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -205> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e0f0 0x560785910b60 auth_reply(proto 2 0 (0) Success) v1 debug -204> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264798+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -203> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785910d00 auth_reply(proto 2 0 (0) Success) v1 debug -202> 2021-11-16T21:23:41.360+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.360828+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -201> 2021-11-16T21:23:41.360+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153680 0x560785910ea0 auth_reply(proto 2 0 (0) Success) v1 debug -200> 2021-11-16T21:23:41.361+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.361219+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -199> 2021-11-16T21:23:41.361+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x560785910000 auth_reply(proto 2 0 (0) Success) v1 debug -198> 2021-11-16T21:23:41.475+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.475815+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -197> 2021-11-16T21:23:41.475+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x56078590f6c0 auth_reply(proto 2 0 (0) Success) v1 debug -196> 2021-11-16T21:23:41.476+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.476381+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -195> 2021-11-16T21:23:41.476+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x56078590fd40 auth_reply(proto 2 0 (0) Success) v1 debug -194> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.531532+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -193> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x56078590f860 auth_reply(proto 2 0 (0) Success) v1 debug -192> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.532037+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -191> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x56078590fa00 auth_reply(proto 2 0 (0) Success) v1 debug -190> 2021-11-16T21:23:42.200+0000 7f1d716db700 4 set_mon_vals no callback set debug -189> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -188> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -187> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -186> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557612) is_readable = 1 - now=2021-11-16T21:23:42.202649+0000 lease_expire=2021-11-16T21:23:47.173895+0000 has v0 lc 19557612 debug -185> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 set_mon_vals no callback set debug -184> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -183> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -182> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -181> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557613) is_readable = 1 - now=2021-11-16T21:23:42.294055+0000 lease_expire=2021-11-16T21:23:47.265226+0000 has v0 lc 19557613 debug -180> 2021-11-16T21:23:43.283+0000 7f1d716db700 4 set_mon_vals no callback set debug -179> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -178> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -177> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -176> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557614) is_readable = 1 - now=2021-11-16T21:23:43.286367+0000 lease_expire=2021-11-16T21:23:48.257095+0000 has v0 lc 19557614 debug -175> 2021-11-16T21:23:43.384+0000 7f1d716db700 4 set_mon_vals no callback set debug -174> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -173> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -172> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -171> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557615) is_readable = 1 - now=2021-11-16T21:23:43.386353+0000 lease_expire=2021-11-16T21:23:48.357251+0000 has v0 lc 19557615 debug -170> 2021-11-16T21:23:44.467+0000 7f1d716db700 4 set_mon_vals no callback set debug -169> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -168> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -167> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -166> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557616) is_readable = 1 - now=2021-11-16T21:23:44.469822+0000 lease_expire=2021-11-16T21:23:49.440271+0000 has v0 lc 19557616 debug -165> 2021-11-16T21:23:44.567+0000 7f1d716db700 4 set_mon_vals no callback set debug -164> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -163> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -162> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -161> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557617) is_readable = 1 - now=2021-11-16T21:23:44.569810+0000 lease_expire=2021-11-16T21:23:49.531775+0000 has v0 lc 19557617 debug -160> 2021-11-16T21:23:45.684+0000 7f1d716db700 4 set_mon_vals no callback set debug -159> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -158> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -157> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -156> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557618) is_readable = 1 - now=2021-11-16T21:23:45.686877+0000 lease_expire=2021-11-16T21:23:50.648710+0000 has v0 lc 19557618 debug -155> 2021-11-16T21:23:46.776+0000 7f1d716db700 4 set_mon_vals no callback set debug -154> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -153> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -152> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -151> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557619) is_readable = 1 - now=2021-11-16T21:23:46.778635+0000 lease_expire=2021-11-16T21:23:51.740053+0000 has v0 lc 19557619 debug -150> 2021-11-16T21:23:46.876+0000 7f1d716db700 4 set_mon_vals no callback set debug -149> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -148> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -147> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -146> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557620) is_readable = 1 - now=2021-11-16T21:23:46.878541+0000 lease_expire=2021-11-16T21:23:51.839977+0000 has v0 lc 19557620 debug -145> 2021-11-16T21:23:47.243+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #531955. Immutable memtables: 0. debug -144> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.244227) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 debug -143> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 531955 debug -142> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827244296, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 683, "num_deletes": 506, "total_data_size": 1462899, "memory_usage": 1475944, "flush_reason": "Manual Compaction"} debug -141> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #531956: started debug -140> 2021-11-16T21:23:47.243+0000 7f1d716db700 4 set_mon_vals no callback set debug -139> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -138> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -137> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -136> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557621) is_readable = 1 - now=2021-11-16T21:23:47.245909+0000 lease_expire=2021-11-16T21:23:52.207163+0000 has v0 lc 19557621 debug -135> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827293833, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 531956, "file_size": 966005, "table_properties": {"data_size": 962689, "index_size": 997, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1477, "raw_key_size": 12966, "raw_average_key_size": 22, "raw_value_size": 954089, "raw_average_value_size": 1667, "num_data_blocks": 29, "num_entries": 572, "num_deletions": 506, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097809, "oldest_key_time": 1637097809, "file_creation_time": 1637097827}} debug -134> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:401] [default] [JOB 3] Level-0 flush table #531956: 966005 bytes OK debug -133> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.293881) [memtable_list.cc:447] [default] Level-0 commit table #531956 started debug -132> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327047) [memtable_list.cc:503] [default] Level-0 commit table #531956: memtable #1 done debug -131> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327058) EVENT_LOG_v1 {"time_micros": 1637097827327054, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} debug -130> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327073) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[3 0 0 0 0 0 2] max score 0.75 debug -129> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 3] Try to delete WAL files size 1458128, prev total WAL file size 1458455, number of live WAL files 2. debug -128> 2021-11-16T21:23:47.326+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -127> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327469) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'logm .. 'logm; will stop at (end) debug -126> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 4] Compacting 3@0 + 2@6 files to L6, score -1.00 debug -125> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 3 Base level 0, inputs: [531956(943KB) 531950(148KB) 531947(7726KB)], [531944(64MB) 531945(36MB)] debug -124> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827327529, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [531956, 531950, 531947], "files_L6": [531944, 531945], "score": -1, "input_data_size": 114546999} debug -123> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531957: 3454 keys, 67198385 bytes debug -122> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827785630, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531957, "file_size": 67198385, "table_properties": {"data_size": 67136484, "index_size": 52412, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is _delta_encoded": 0, "filter_size": 8645, "raw_key_size": 76781, "raw_average_key_size": 22, "raw_value_size": 67047935, "raw_average_value_size": 19411, "num_data_blocks": 1555, "num_entries": 3454, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}} debug -121> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531958: 1281 keys, 32015375 bytes debug -120> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828044065, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531958, "file_size": 32015375, "table_properties": {"data_size": 31968485, "index_size": 42777, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 3269, "raw_key_size": 29915, "raw_average_key_size": 23, "raw_value_size": 31919151, "raw_average_value_size": 24917, "num_data_blocks": 1155, "num_entries": 1281, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}} debug -119> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 4] Compacted 3@0 + 2@6 files to L6 => 99213760 bytes debug -118> 2021-11-16T21:23:48.066+0000 7f1d716db700 4 set_mon_vals no callback set debug -117> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -116> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -115> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -114> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557622) is_readable = 1 - now=2021-11-16T21:23:48.069418+0000 lease_expire=2021-11-16T21:23:52.960557+0000 has v0 lc 19557622 debug -113> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085683) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 159.9 rd, 138.5 wr, level 6, files in(3, 2) out(2) MB in(8.6, 100.6) out(94.6), read-write-amplify(23.7) write-amplify(11.0) OK, records in: 5811, records dropped: 1076 output_compression: NoCompression debug -112> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085701) EVENT_LOG_v1 {"time_micros": 1637097828085695, "job": 4, "event": "compaction_finished", "compaction_time_micros": 716586, "compaction_time_cpu_micros": 83813, "output_level": 6, "num_output_files": 2, "total_output_size": 99213760, "num_input_records": 5811, "num_output_records": 4735, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]} debug -111> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085923, "job": 4, "event": "table_file_deletion", "file_number": 531956} debug -110> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085969, "job": 4, "event": "table_file_deletion", "file_number": 531950} debug -109> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828086487, "job": 4, "event": "table_file_deletion", "file_number": 531947} debug -108> 2021-11-16T21:23:48.088+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828089419, "job": 4, "event": "table_file_deletion", "file_number": 531945} debug -107> 2021-11-16T21:23:48.093+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828094233, "job": 4, "event": "table_file_deletion", "file_number": 531944} debug -106> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -105> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -104> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -103> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -102> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -101> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -100> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -99> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -98> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -97> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -96> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -95> 2021-11-16T21:23:49.260+0000 7f1d716db700 4 set_mon_vals no callback set debug -94> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -93> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -92> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -91> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557623) is_readable = 1 - now=2021-11-16T21:23:49.262917+0000 lease_expire=2021-11-16T21:23:54.223482+0000 has v0 lc 19557623 debug -90> 2021-11-16T21:23:50.002+0000 7f1d716db700 4 set_mon_vals no callback set debug -89> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -88> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -87> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -86> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557624) is_readable = 1 - now=2021-11-16T21:23:50.004764+0000 lease_expire=2021-11-16T21:23:54.973508+0000 has v0 lc 19557624 debug -85> 2021-11-16T21:23:51.102+0000 7f1d716db700 4 set_mon_vals no callback set debug -84> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -83> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -82> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -81> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557625) is_readable = 1 - now=2021-11-16T21:23:51.104896+0000 lease_expire=2021-11-16T21:23:56.064982+0000 has v0 lc 19557625 debug -80> 2021-11-16T21:23:51.202+0000 7f1d716db700 4 set_mon_vals no callback set debug -79> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -78> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -77> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -76> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557626) is_readable = 1 - now=2021-11-16T21:23:51.204980+0000 lease_expire=2021-11-16T21:23:56.164769+0000 has v0 lc 19557626 debug -75> 2021-11-16T21:23:52.261+0000 7f1d716db700 4 set_mon_vals no callback set debug -74> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -73> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -72> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -71> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557627) is_readable = 1 - now=2021-11-16T21:23:52.263508+0000 lease_expire=2021-11-16T21:23:57.231263+0000 has v0 lc 19557627 debug -70> 2021-11-16T21:23:52.352+0000 7f1d716db700 4 set_mon_vals no callback set debug -69> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -68> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -67> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -66> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557628) is_readable = 1 - now=2021-11-16T21:23:52.355232+0000 lease_expire=2021-11-16T21:23:57.322723+0000 has v0 lc 19557628 debug -65> 2021-11-16T21:23:53.436+0000 7f1d716db700 4 set_mon_vals no callback set debug -64> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -63> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -62> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -61> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557629) is_readable = 1 - now=2021-11-16T21:23:53.438872+0000 lease_expire=2021-11-16T21:23:58.406301+0000 has v0 lc 19557629 debug -60> 2021-11-16T21:23:54.519+0000 7f1d716db700 4 set_mon_vals no callback set debug -59> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -58> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -57> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -56> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557630) is_readable = 1 - now=2021-11-16T21:23:54.522319+0000 lease_expire=2021-11-16T21:23:59.489316+0000 has v0 lc 19557630 debug -55> 2021-11-16T21:23:55.595+0000 7f1d716db700 4 set_mon_vals no callback set debug -54> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -53> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -52> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -51> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557631) is_readable = 1 - now=2021-11-16T21:23:55.597582+0000 lease_expire=2021-11-16T21:24:00.564365+0000 has v0 lc 19557631 debug -50> 2021-11-16T21:23:56.670+0000 7f1d716db700 4 set_mon_vals no callback set debug -49> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -48> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -47> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -46> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557632) is_readable = 1 - now=2021-11-16T21:23:56.672809+0000 lease_expire=2021-11-16T21:24:01.638988+0000 has v0 lc 19557632 debug -45> 2021-11-16T21:23:57.270+0000 7f1d716db700 4 set_mon_vals no callback set debug -44> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -43> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -42> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -41> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557633) is_readable = 1 - now=2021-11-16T21:23:57.272872+0000 lease_expire=2021-11-16T21:24:02.238966+0000 has v0 lc 19557633 debug -40> 2021-11-16T21:23:57.753+0000 7f1d716db700 4 set_mon_vals no callback set debug -39> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -38> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -37> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -36> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557634) is_readable = 1 - now=2021-11-16T21:23:57.756400+0000 lease_expire=2021-11-16T21:24:02.722461+0000 has v0 lc 19557634 debug -35> 2021-11-16T21:23:58.829+0000 7f1d716db700 4 set_mon_vals no callback set debug -34> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -33> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -32> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -31> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557635) is_readable = 1 - now=2021-11-16T21:23:58.831535+0000 lease_expire=2021-11-16T21:24:03.797240+0000 has v0 lc 19557635 debug -30> 2021-11-16T21:23:58.920+0000 7f1d716db700 4 set_mon_vals no callback set debug -29> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -28> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6 849/4203565106] debug -27> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -26> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557636) is_readable = 1 - now=2021-11-16T21:23:58.922888+0000 lease_expire=2021-11-16T21:24:03.888665+0000 has v0 lc 19557636 debug -25> 2021-11-16T21:23:59.354+0000 7f1d716db700 4 set_mon_vals no callback set debug -24> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -23> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -22> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -21> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557637) is_readable = 1 - now=2021-11-16T21:23:59.356709+0000 lease_expire=2021-11-16T21:24:04.321986+0000 has v0 lc 19557637 debug -20> 2021-11-16T21:24:00.462+0000 7f1d716db700 4 set_mon_vals no callback set debug -19> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -18> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -17> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -16> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557638) is_readable = 1 - now=2021-11-16T21:24:00.465271+0000 lease_expire=2021-11-16T21:24:05.430188+0000 has v0 lc 19557638 debug -15> 2021-11-16T21:24:00.554+0000 7f1d716db700 4 set_mon_vals no callback set debug -14> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -13> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -12> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -11> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557639) is_readable = 1 - now=2021-11-16T21:24:00.556869+0000 lease_expire=2021-11-16T21:24:05.521766+0000 has v0 lc 19557639 debug -10> 2021-11-16T21:24:01.629+0000 7f1d716db700 4 set_mon_vals no callback set debug -9> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -8> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -7> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -6> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.632342+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -5> 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0) debug -4> 2021-11-16T21:24:01.873+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875099+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -3> 2021-11-16T21:24:01.873+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x56078525b380 auth_reply(proto 2 0 (0) Success) v1 debug -2> 2021-11-16T21:24:01.874+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875450+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -1> 2021-11-16T21:24:01.874+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eb40 0x560785259d40 auth_reply(proto 2 0 (0) Success) v1 debug 0> 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) ** in thread 7f1d716db700 thread_name:ms_dispatch ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8] 8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045] 9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6] 10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329] 11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258] 12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed] 13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910] 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc] 15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71] 17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a] 18: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 139764358260480 / rstore_compact 139764375045888 / ms_dispatch 139764391831296 / rocksdb:dump_st 139764417009408 / msgr-worker-0 139764433794816 / ms_dispatch 139764450580224 / cpu_tp 139764467365632 / cpu_tp 139764475758336 / safe_timer 139764526114560 / rocksdb:high0 139764534507264 / rocksdb:low0 139764560197376 / admin_socket 139764813260544 / ceph-mon max_recent 10000 max_new 10000 log_file --- end dump of recent events --- --- begin dump of recent events --- debug -734> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command assert hook 0x5607840e8530 debug -733> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command abort hook 0x5607840e8530 debug -732> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command leak_some_memory hook 0x5607840e8530 debug -731> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfcounters_dump hook 0x5607840e8530 debug -730> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 1 hook 0x5607840e8530 debug -729> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf dump hook 0x5607840e8530 debug -728> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perfc ounters_schema hook 0x5607840e8530 debug -727> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram dump hook 0x5607840e8530 debug -726> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command 2 hook 0x5607840e8530 debug -725> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf schema hook 0x5607840e8530 debug -724> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf histogram schema hook 0x5607840e8530 debug -723> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command perf reset hook 0x5607840e8530 debug -722> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config show hook 0x5607840e8530 debug -721> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config help hook 0x5607840e8530 debug -720> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config set hook 0x5607840e8530 debug -719> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config unset hook 0x5607840e8530 debug -718> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config get hook 0x5607840e8530 debug -717> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff hook 0x5607840e8530 debug -716> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command config diff get hook 0x5607840e8530 debug -715> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command injectargs hook 0x5607840e8530 debug -714> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log flush hook 0x5607840e8530 debug -713> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log dump hook 0x5607840e8530 debug -712> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command log reopen hook 0x5607840e8530 debug -711> 2021-11-16T21:23:28.649+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_mempools hook 0x560784188328 debug -710> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 set uid:gid to 167:167 (ceph:ceph) debug -709> 2021-11-16T21:23:28.655+0000 7f1d880be700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 8 debug -708> 2021-11-16T21:23:28.656+0000 7f1d880be700 0 pidfile_write: ignore empty --pid-file debug -707> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) init /var/run/ceph/ceph-mon.cn03.asok debug -706> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) bind_and_listen /var/run/ceph/ceph-mon.cn03.asok debug -705> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command 0 hook 0x5607840ec108 debug -704> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command version hook 0x5607840ec108 debug -703> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command git_version hook 0x5607840ec108 debug -702> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command help hook 0x5607840e8230 debug -701> 2021-11-16T21:23:28.656+0000 7f1d880be700 5 asok(0x56078419c000) register_command get_command_descriptions hook 0x5607840e81e0 debug -700> 2021-11-16T21:23:28.656+0000 7f1d78f67700 5 asok(0x56078419c000) entry start debug -699> 2021-11-16T21:23:28.660+0000 7f1d880be700 0 load: jerasure load: lrc load: isa debug -698> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true debug -697> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432 debug -696> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression debug -695> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option level_compaction_dynamic_level_bytes = true debug -694> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option write_buffer_size = 33554432 debug -693> 2021-11-16T21:23:28.660+0000 7f1d880be700 1 set rocksdb option compression = kNoCompression debug -692> 2021-11-16T21:23:28.661+0000 7f1d880be700 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -691> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: RocksDB version: 6.8.1 debug -690> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -689> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compile date May 13 2021 debug -688> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: DB SUMMARY debug -687> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: CURRENT file: CURRENT debug -686> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: IDENTITY file: IDENTITY debug -685> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: MANIFEST file: MANIFEST-531948 size: 302 Bytes debug -684> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-cn03/store.db dir, Total Num: 3, files: 531944.sst 531945.sst 531947.sst debug -683> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-cn03/store.db: 531949.log size: 214103 ; debug -682> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.error_if_exists: 0 debug -681> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_if_missing: 0 debug -680> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_checks: 1 debug -679> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.env: 0x5607837a11c0 debug -678> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.fs: Posix File System debug -677> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.info_log: 0x5607841d78c0 debug -676> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_file_opening_threads: 16 debug -675> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.statistics: (nil) debug -674> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_fsync: 0 debug -673> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_log_file_size: 0 debug -672> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -671> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_file_time_to_roll: 0 debug -670> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.keep_log_file_num: 1000 debug -669> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.recycle_log_file_num: 0 debug -668> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_fallocate: 1 debug -667> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_reads: 0 debug -666> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_mmap_writes: 0 debug -665> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_reads: 0 debug -664> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -663> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.create_missing_column_families: 0 debug -662> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_log_dir: debug -661> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-cn03 /store.db debug -660> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_cache_numshardbits: 6 debug -659> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_subcompactions: 1 debug -658> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_flushes: -1 debug -657> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -656> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -655> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -654> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -653> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -652> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.advise_random_on_open: 1 debug -651> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.db_write_buffer_size: 0 debug -650> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_manager: 0x5607841ddf80 debug -649> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -648> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 debug -647> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -646> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.use_adaptive_mutex: 0 debug -645> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limiter: (nil) debug -644> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -643> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_recovery_mode: 2 debug -642> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_thread_tracking: 0 debug -641> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_pipelined_write: 0 debug -640> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.unordered_write: 0 debug -639> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -638> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -637> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -636> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -635> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.row_cache: None debug -634> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_filter: None debug -633> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -632> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.allow_ingest_behind: 0 debug -631> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.preserve_deletes: 0 debug -630> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.two_write_queues: 0 debug -629> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.manual_wal_flush: 0 debug -628> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.atomic_flush: 0 debug -627> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -626> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.persist_stats_to_disk: 0 debug -625> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -624> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.log_readahead_size: 0 debug -623> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -622> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_jobs: 2 debug -621> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_background_compactions: -1 debug -620> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -619> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 debug -618> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delayed_write_rate : 16777216 debug -617> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_total_wal_size: 0 debug -616> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -615> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_dump_period_sec: 600 debug -614> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_persist_period_sec: 600 debug -613> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -612> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_open_files: -1 debug -611> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bytes_per_sync: 0 debug -610> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -609> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -608> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_readahead_size: 0 debug -607> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Compression algorithms supported: debug -606> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -605> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZSTD supported: 0 debug -604> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kXpressCompression supported: 0 debug -603> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4HCCompression supported: 1 debug -602> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kLZ4Compression supported: 1 debug -601> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kBZip2Compression supported: 0 debug -600> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kZlibCompression supported: 1 debug -599> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: kSnappyCompression supported: 1 debug -598> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -597> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 debug -596> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -595> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -594> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.merge_operator: debug -593> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter: None debug -592> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_filter_factory: None debug -591> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtab le_factory: SkipListFactory debug -590> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_factory: BlockBasedTable debug -589> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607840ec128) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x560784125010 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -588> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.write_buffer_size: 33554432 debug -587> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number: 2 debug -586> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression: NoCompression debug -585> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression: Disabled debug -584> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.prefix_extractor: nullptr debug -583> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -582> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.num_levels: 7 debug -581> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -580> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -579> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -578> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -577> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -576> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -575> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -574> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -573> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -572> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.window_bits: -14 debug -571> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.level: 32767 debug -570> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.strategy: 0 debug -569> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -568> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -567> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compression_opts.enabled: false debug -566> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -565> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -564> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -563> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_base: 67108864 debug -562> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.target_file_size_multiplier: 1 debug -561> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -560> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 debug -559> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -558> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -557> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -556> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -555> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -554> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -553> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -552> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -551> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -550> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -549> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.arena_block_size: 4194304 debug -548> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -547> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -546> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -545> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.disable_auto_compactions: 0 debug -544> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -543> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -542> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -541> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -540> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -539> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -538> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -537> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -536> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -535> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -534> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.table_properties_collectors: debug -533> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Opt ions.inplace_update_support: 0 debug -532> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -531> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -530> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -529> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.memtable_huge_page_size: 0 debug -528> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.bloom_locality: 0 debug -527> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.max_successive_merges: 0 debug -526> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -525> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.paranoid_file_checks: 0 debug -524> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.force_consistency_checks: 0 debug -523> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.report_bg_io_stats: 0 debug -522> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.ttl: 2592000 debug -521> 2021-11-16T21:23:28.661+0000 7f1d880be700 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -520> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-cn03/store.db/MANIFEST-531948 succeeded,manifest_file_number is 531948, next_file_number is 531950, last_sequence is 201837822, log_number is 531943,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -519> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 531943 debug -518> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808663528, "job": 1, "event": "recovery_started", "log_files": [531949]} debug -517> 2021-11-16T21:23:28.662+0000 7f1d880be700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #531949 mode 2 debug -516> 2021-11-16T21:23:28.663+0000 7f1d880be700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. debug -515> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808697864, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 531950, "file_size": 151829, "table_properties": {"data_size": 150555, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": 759, "raw_average_key_size": 25, "raw_value_size": 149707, "raw_average_value_size": 4990, "num_data_blocks": 8, "num_entries": 30, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097808, "oldest_key_time": 3, "file_creation_time": 0}} debug -514> 2021-11-16T21:23:28.696+0000 7f1d880be700 4 rocksdb: [version_set.cc:3826] Creating manifest 531951 debug -513> 2021-11-16T21:23:28.755+0000 7f1d880be700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097808756165, "job": 1, "event": "recovery_finished"} debug -512> 2021-11-16T21:23:28.797+0000 7f1d880be700 4 rocksdb: DB pointer 0x5607841e9800 debug -511> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- debug -510> 2021-11-16T21:23:28.797+0000 7f1d6eed6700 4 rocksdb: [db_impl/db_impl.cc:851] ** DB Stats ** Uptime(secs): 0.1 total, 0.1 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Uptime(secs): 0.1 total, 0.1 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 7.69 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 L6 2/0 100.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 108.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.03 0.00 1 0.034 0 0 Uptime(secs): 0.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 1.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** debug -509> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -508> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -507> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: cephx debug -506> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding auth protocol: none debug -505> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -504> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -503> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -502> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -501> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -500> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -499> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -498> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -497> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -496> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -495> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: crc debug -494> 2021-11-16T21:23:28.797+0000 7f1d880be700 5 AuthRegistry(0x560784d6ea40) adding con mode: secure debug -493> 2021-11-16T21:23:28.797+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -492> 2021-11-16T21:23:28.798+0000 7f1d880be700 0 starting mon.cn03 rank 2 at public addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] at bind addrs [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon_data /var/lib/ceph/mon/ceph-cn03 fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d debug -491> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -490> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -489> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: cephx debug -488> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding auth protocol: none debug -487> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -486> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -485> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -484> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -483> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -482> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -481> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -480> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -479> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -478> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -477> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: crc debug -476> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 AuthRegistry(0x560784d6e140) adding con mode: secure debug -475> 2021-11-16T21:23:28.798+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -474> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx debug -473> 2021-11-16T21:23:28.798+0000 7f1d880be700 5 adding auth protocol: cephx debug -472> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) debug -471> 2021-11-16T21:23:28.798+0000 7f1d880be700 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) debug -470> 2021-11-16T21:23:28.799+0000 7f1d880be700 1 mon.cn03@-1(???) e69 preinit fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d debug -469> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 new map debug -468> 2021-11-16T21:23:28.799+0000 7f1d880be700 0 mon.cn03@-1(???).mds e432 print_map e432 enable_multiple, ever_enabled_multiple: 1,1 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 2 Filesystem 'coldlogix' (2) fs_name coldlogix epoch 402 flags 12 created 2021-06-27T06:32:25.086015+0000 modified 2021-11-15T08:30:22.670221+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 17247 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=8444199} failed damaged stopped data_pools [12] metadata_pool 11 inline_data disabled balancer standby_count_wanted 1 [mds.coldlogix.cn01.dirzwe{0:8444199} state up:active seq 77 join_fscid=2 addr [v2:192.168.30.11:6850/3427514521,v1:192.168.30.11:6851/3427514521]] Filesystem 'btc' (5) fs_name btc epoch 423 flags 12 created 2021-11-15T08:49:14.520956+0000 modified 2021-11-15T08:49:25.936407+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 17308 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 i n 0 up {0=8534156} failed damaged stopped data_pools [19] metadata_pool 18 inline_data disabled balancer standby_count_wanted 1 [mds.btc.cn04.pyuaxe{0:8534156} state up:active seq 5 join_fscid=5 addr [v2:192.168.30.14:6800/4253241567,v1:192.168.30.14:6801/4253241567]] Standby daemons: [mds.btc.cn06.hqmwsx{-1:8534166} state up:standby seq 3 join_fscid=5 addr [v2:192.168.30.16:6848/446743436,v1:192.168.30.16:6849/446743436]] [mds.coldlogix.cn03.lyxzhp{-1:8604119} state up:standby seq 4 join_fscid=2 addr [v2:192.168.30.13:6800/2645334932,v1:192.168.30.13:6801/2645334932]] debug -467> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 3314933000854323200, adjusting msgr requires debug -466> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -465> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -464> 2021-11-16T21:23:28.800+0000 7f1d880be700 0 mon.cn03@-1(???).osd e17413 crush map has features 432629239337189376, adjusting msgr requires debug -463> 2021-11-16T21:23:28.800+0000 7f1d880be700 1 mon.cn03@-1(???).paxosservice(auth 10008..10244) refresh upgraded, format 0 -> 3 debug -462> 2021-11-16T21:23:28.800+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e0 loading version 500 debug -461> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 active server: [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106](8444129) debug -460> 2021-11-16T21:23:28.801+0000 7f1d880be700 4 mon.cn03@-1(???).mgr e500 mkfs or daemon transitioned to available, loading commands debug -459> 2021-11-16T21:23:28.802+0000 7f1d880be700 4 set_mon_vals no callback set debug -458> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals auth_allow_insecure_global_id_reclaim = false debug -457> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 debug -456> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals mon_allow_pool_delete = true debug -455> 2021-11-16T21:23:28.802+0000 7f1d880be700 10 set_mon_vals rbd_move_to_trash_on_remove = true debug -454> 2021-11-16T21:23:28.803+0000 7f1d880be700 4 set_mon_vals no callback set debug -453> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -452> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command compact hook 0x5607840e89a0 debug -451> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command smart hook 0x5607840e89a0 debug -450> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command mon_status hook 0x5607840e89a0 debug -449> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command heap hook 0x5607840e89a0 debug -448> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores dump hook 0x5607840e89a0 debug -447> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command connection scores reset hook 0x5607840e89a0 debug -446> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sync_force hook 0x5607840e89a0 debug -445> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hint hook 0x5607840e89a0 debug -444> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command add_bootstrap_peer_hintv hook 0x5607840e89a0 debug -443> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum enter hook 0x5607840e89a0 debug -442> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command quorum exit hook 0x5607840e89a0 debug -441> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command ops hook 0x5607840e89a0 debug -440> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command sessions hook 0x5607840e89a0 debug -439> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 asok(0x56078419c000) register_command dump_historic_ops hook 0x5607840e89a0 debug -438> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -437> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -436> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: cephx debug -435> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding auth protocol: none debug -434> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -433> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -432> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -431> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -430> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -429> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -428> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -427> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -426> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -425> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -424> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: crc debug -423> 2021-11-16T21:23:28.803+0000 7f1d880be700 5 AuthRegistry(0x5607841eb020) adding con mode: secure debug -422> 2021-11-16T21:23:28.803+0000 7f1d880be700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cn03/keyring debug -421> 2021-11-16T21:23:28.804+0000 7f1d880be700 2 mon.cn03@-1(???) e69 init debug -420> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Got map version 500 debug -419> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -418> 2021-11-16T21:23:28.805+0000 7f1d880be700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -417> 2021-11-16T21:23:28.805+0000 7f1d726dd700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d726dd700' had timed out after 0.000000000s debug -416> 2021-11-16T21:23:28.806+0000 7f1d880be700 0 mon.cn03@-1(probing) e69 my rank is now 2 (was -1) debug -415> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure stats_period=5 debug -414> 2021-11-16T21:23:28.806+0000 7f1d6ded4700 4 mgrc handle_mgr_configure updated stats threshold: 5 debug -413> 2021-11-16T21:23:28.807+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection debug -412> 2021-11-16T21:23:29.213+0000 7f1d716db700 5 mon.cn03@2(probing) e69 _ms_dispatch setting monitor caps on this connection debug -411> 2021-11-16T21:23:29.214+0000 7f1d716db700 0 log_channel(cluster) log [INF] : mon.cn03 calling monitor election debug -410> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client _send_to_mon log to self debug -409> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 debug -408> 2021-11-16T21:23:29.214+0000 7f1d716db700 10 log_client will send 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election debug -407> 2021-11-16T21:23:29.214+0000 7f1d716db 700 5 paxos.2).electionLogic(0) start -- can i be leader? debug -406> 2021-11-16T21:23:29.214+0000 7f1d716db700 1 paxos.2).electionLogic(5412) init, last seen epoch 5412 debug -405> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -404> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248762+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -403> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:29.248771+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -402> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.1 debug -401> 2021-11-16T21:23:29.247+0000 7f1d716db700 5 paxos.2).electionLogic(5413) defer to 1 debug -400> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3 debug -399> 2021-11-16T21:23:29.366+0000 7f1d716db700 5 paxos.2).electionLogic(5413) no, we already acked 1 debug -398> 2021-11-16T21:23:30.619+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1 debug -397> 2021-11-16T21:23:33.077+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -396> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.4 debug -395> 2021-11-16T21:23:33.098+0000 7f1d716db700 5 paxos.2).electionLogic(5413) ignoring old propose debug -394> 2021-11-16T21:23:34.411+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5413) handle_propose from mon.3 debug -393> 2021-11-16T21:23:34.440+0000 7f1d716db700 5 paxos.2).electionLogic(5415) no, we already acked 1 debug -392> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing) e69 _ms_dispatch setting monitor caps on this connection debug -391> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5415) handle_propose from mon.0 debug -390> 2021-11-16T21:23:35.010+0000 7f1d716db700 5 paxos.2).electionLogic(5415) ignoring old propose debug -389> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) election period ended debug -388> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 5 paxos.2).electionLogic(5415) start -- can i be leader? debug -387> 2021-11-16T21:23:35.249+0000 7f1d73ee0700 1 paxos.2).electionLogic(5415) init, last seen epoch 5415, mid-election, bumping debug -386> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_ack from mon.3 debug -385> 2021-11-16T21:23:35.341+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) so far i have { mon.2: features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) } debug -384> 2021-11-16T21:23:35.392+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5417) handle_propose from mon.0 debug -383> 2021-11-16T21:23:35.424+0000 7f1d716db700 5 paxos.2).electionLogic(5419) defer to 0 debug -382> 2021-11-16T21:23:36.621+0000 7f1d716db700 5 mon.cn03@2(electing) e69 waitlisting message auth(proto 0 35 bytes epoch 0) v1 debug -381> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256876+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -380> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256899+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -379> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256913+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -378> 2021-11-16T21:23:38.256+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.256925+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -377> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257153+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -376> 2021-11-16T21:23:38.257+0000 7f1d716db700 5 mon.cn03@2(electing).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.257169+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -375> 2021-11-16T21:23:38.342+0000 7f1d716db700 5 mon.cn03@2(electing).elector(5419) handle_victory from mon.0 quorum_features 4540138297136906239 mon_feature_t([kraken,luminous,mimic,osdmap-prune,nautilus,octopus,pacific,elector-pinging]) debug -374> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_quorum_to_compatset_features debug -373> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon) e69 apply_monmap_to_compatset_features debug -372> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382930+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -371> 2021-11-16T21:23:38.382+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos recovering c 19556917..19557603) is_readable = 0 - now=2021-11-16T21:23:38.382942+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557603 debug -370> 2021-11-16T21:23:38.450+0000 7f1d716db700 4 set_mon_vals no callback set debug -369> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -368> 2021-11-16T21:23:38.451+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -367> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -366> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451523+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -365> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451536+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -364> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451571+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -363> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451596+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -362> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451620+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -361> 2021-11-16T21:23:38.451+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557604) is_readable = 1 - now=2021-11-16T21:23:38.451641+0000 lease_expire=2021-11-16T21:23:43.413758+0000 has v0 lc 19557604 debug -360> 2021-11-16T21:23:38.452+0000 7f1d736df700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f1d736df700' had timed out after 0.000000000s debug -359> 2021-11-16T21:23:38.538+0000 7f1d706d9700 1 mon.cn03@2(peon) e69 handle_auth_request failed to assign global_id debug -358> 2021-11-16T21:23:38.541+0000 7f1d716db700 4 set_mon_vals no callback set debug -357> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr _map Got map version 500 debug -356> 2021-11-16T21:23:38.542+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -355> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -354> 2021-11-16T21:23:38.542+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557605) is_readable = 1 - now=2021-11-16T21:23:38.542983+0000 lease_expire=2021-11-16T21:23:43.507566+0000 has v0 lc 19557605 debug -353> 2021-11-16T21:23:38.635+0000 7f1d716db700 4 set_mon_vals no callback set debug -352> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -351> 2021-11-16T21:23:38.636+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -350> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -349> 2021-11-16T21:23:38.636+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557606) is_readable = 1 - now=2021-11-16T21:23:38.636621+0000 lease_expire=2021-11-16T21:23:43.603066+0000 has v0 lc 19557606 debug -348> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client handle_log_ack log(last 1) v1 debug -347> 2021-11-16T21:23:38.666+0000 7f1d716db700 10 log_client logged 2021-11-16T21:23:29.215196+0000 mon.cn03 (mon.2) 1 : cluster [INF] mon.cn03 calling monitor election debug -346> 2021-11-16T21:23:38.725+0000 7f1d716db700 4 set_mon_vals no callback set debug -345> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -344> 2021-11-16T21:23:38.726+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -343> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -342> 2021-11-16T21:23:38.726+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557607) is_readable = 1 - now=2021-11-16T21:23:38.726705+0000 lease_expire=2021-11-16T21:23:43.690619+0000 has v0 lc 19557607 debug -341> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783222+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607 debug -340> 2021-11-16T21:23:38.783+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos updating c 19556917..19557607) is_readable = 0 - now=2021-11-16T21:23:38.783242+0000 lease_expire=1970-01-01T00:00:00.000000+0000 has v0 lc 19557607 debug -339> 2021-11-16T21:23:38.841+0000 7f1d716db700 4 set_mon_vals no callback set debug -338> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -337> 2021-11-16T21:23:38.842+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -336> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -335> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.843331+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -334> 2021-11-16T21:23:38.843+0000 7f1d716db700 5 mon.cn03@2(peon).elector(5420) begin_peer_ping against 3 debug -333> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944355+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -332> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853ebef0 0x5607858deb60 auth_reply(proto 2 0 (0) Success) v1 debug -331> 2021-11-16T21:23:38.944+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.944811+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -330> 2021-11-16T21:23:38.944+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858df380 auth_reply(proto 2 0 (0) Success) v1 debug -329> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984545+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -328> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x5607858de820 auth_reply(proto 2 0 (0) Success) v1 debug -327> 2021-11-16T21:23:38.984+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:38.984931+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -326> 2021-11-16T21:23:38.984+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851533b0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1 debug -325> 2021-11-16T21:23:39.026+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.026942+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -324> 2021-11-16T21:23:39.026+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x5607858de9c0 auth_reply(proto 2 0 (0) Success) v1 debug -323> 2021-11-16T21:23:39.027+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.027319+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -322> 2021-11-16T21:23:39.027+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x5607858deea0 auth_reply(proto 2 0 (0) Success) v1 debug -321> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067500+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -320> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x5607858df520 auth_reply(proto 2 0 (0) Success) v1 debug -319> 2021-11-16T21:23:39.067+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.067791+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -318> 2021-11-16T21:23:39.067+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607858de680 auth_reply(proto 2 0 (0) Success) v1 debug -317> 2021-11-16T21:23:39.105+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.105918+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -316> 2021-11-16T21:23:39.105+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858dc9c0 auth_reply(proto 2 0 (0) Success) v1 debug -315> 2021-11-16T21:23:39.106+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.106361+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -314> 2021-11-16T21:23:39.106+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607858dc820 auth_reply(proto 2 0 (0) Success) v1 debug -313> 2021-11-16T21:23:39.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.217724+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -312> 2021-11-16T21:23:39.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e4b0 0x560785262b60 auth_reply(proto 2 0 (0) Success) v1 debug -311> 2021-11-16T21:23:39.218+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.218155+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -310> 2021-11-1 6T21:23:39.218+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x5607852636c0 auth_reply(proto 2 0 (0) Success) v1 debug -309> 2021-11-16T21:23:39.335+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336070+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -308> 2021-11-16T21:23:39.335+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858eb6c0 auth_reply(proto 2 0 (0) Success) v1 debug -307> 2021-11-16T21:23:39.336+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.336443+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -306> 2021-11-16T21:23:39.336+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858eab60 auth_reply(proto 2 0 (0) Success) v1 debug -305> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.388745+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -304> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x560785916680 auth_reply(proto 2 0 (0) Success) v1 debug -303> 2021-11-16T21:23:39.388+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.389042+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -302> 2021-11-16T21:23:39.388+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152870 0x560785916340 auth_reply(proto 2 0 (0) Success) v1 debug -301> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415381+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -300> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb770 0x5607858f2b60 auth_reply(proto 2 0 (0) Success) v1 debug -299> 2021-11-16T21:23:39.415+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.415676+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -298> 2021-11-16T21:23:39.415+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607858ec9c0 auth_reply(proto 2 0 (0) Success) v1 debug -297> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441170+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -296> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785912d00 auth_reply(proto 2 0 (0) Success) v1 debug -295> 2021-11-16T21:23:39.441+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.441498+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -294> 2021-11-16T21:23:39.441+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785912680 auth_reply(proto 2 0 (0) Success) v1 debug -293> 2021-11-16T21:23:39.547+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.547657+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -292> 2021-11-16T21:23:39.547+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x5607859124e0 auth_reply(proto 2 0 (0) Success) v1 debug -291> 2021-11-16T21:23:39.548+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557608) is_readable = 1 - now=2021-11-16T21:23:39.548146+0000 lease_expire=2021-11-16T21:23:43.807251+0000 has v0 lc 19557608 debug -290> 2021-11-16T21:23:39.548+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607852621a0 auth_reply(proto 2 0 (0) Success) v1 debug -289> 2021-11-16T21:23:39.717+0000 7f1d716db700 4 set_mon_vals no callback set debug -288> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -287> 2021-11-16T21:23:39.718+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -286> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -285> 2021-11-16T21:23:39.718+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.718923+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -284> 2021-11-16T21:23:39.796+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.796980+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -283> 2021-11-16T21:23:39.796+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x560785263860 auth_reply(proto 2 0 (0) Success) v1 debug -282> 2021-11-16T21:23:39.797+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.797502+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -281> 2021-11-16T21:23:39.797+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x5607858b36c0 auth_reply(proto 2 0 (0) Success) v1 debug -280> 2021-11-16T21:23:39.853+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854122+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -279> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e000 0x56078525eb60 auth_reply(proto 2 0 (0) Success) v1 debug -278> 2021-11-16T21:23:39.854+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.854658+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -277> 2021-11-16T21:23:39.854+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eb4a0 0x5607858eb520 auth_reply(proto 2 0 (0) Success) v1 debug -276> 2021-11-16T21:23:39.999+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:39.999601+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -275> 2021-11-16T21:23:39.999+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x560785263520 auth_reply(proto 2 0 (0) Success) v1 debug -274> 2021-11-16T21:23:40.000+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.000151+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -273> 2021-11-16T21:23:40.000+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x5607858b3040 auth_reply(proto 2 0 (0) Success) v1 debug -272> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078366+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -271> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eae10 0x5607852631e0 auth_reply(proto 2 0 (0) Success) v1 debug -270> 2021-11-16T21:23:40.078+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.078744+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -269> 2021-11-16T21:23:40.078+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eba40 0x560785263380 auth_reply(proto 2 0 (0) Success) v1 debug -268> 2021-11-16T21:23:40.176+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177084+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -267> 2021-11- 16T21:23:40.176+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x560785225520 auth_reply(proto 2 0 (0) Success) v1 debug -266> 2021-11-16T21:23:40.177+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.177745+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -265> 2021-11-16T21:23:40.177+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x5607858ad520 auth_reply(proto 2 0 (0) Success) v1 debug -264> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -263> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ea50 0x560785917380 auth_reply(proto 2 0 (0) Success) v1 debug -262> 2021-11-16T21:23:40.217+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.217665+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -261> 2021-11-16T21:23:40.217+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e690 0x560785916ea0 auth_reply(proto 2 0 (0) Success) v1 debug -260> 2021-11-16T21:23:40.274+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.274992+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -259> 2021-11-16T21:23:40.274+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ec30 0x56078525a820 auth_reply(proto 2 0 (0) Success) v1 debug -258> 2021-11-16T21:23:40.275+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.275761+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -257> 2021-11-16T21:23:40.275+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851521e0 0x560785258d00 auth_reply(proto 2 0 (0) Success) v1 debug -256> 2021-11-16T21:23:40.315+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.315817+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -255> 2021-11-16T21:23:40.315+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x56078591a4e0 auth_reply(proto 2 0 (0) Success) v1 debug -254> 2021-11-16T21:23:40.316+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.316340+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -253> 2021-11-16T21:23:40.316+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785259520 auth_reply(proto 2 0 (0) Success) v1 debug -252> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413338+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -251> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859131e0 auth_reply(proto 2 0 (0) Success) v1 debug -250> 2021-11-16T21:23:40.413+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.413824+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -249> 2021-11-16T21:23:40.413+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785916000 auth_reply(proto 2 0 (0) Success) v1 debug -248> 2021-11-16T21:23:40.493+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.493816+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -247> 2021-11-16T21:23:40.493+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851534a0 0x5607859161a0 auth_reply(proto 2 0 (0) Success) v1 debug -246> 2021-11-16T21:23:40.494+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.494344+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -245> 2021-11-16T21:23:40.494+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152d20 0x560785913380 auth_reply(proto 2 0 (0) Success) v1 debug -244> 2021-11-16T21:23:40.571+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.571680+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -243> 2021-11-16T21:23:40.571+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153950 0x560785916340 auth_reply(proto 2 0 (0) Success) v1 debug -242> 2021-11-16T21:23:40.572+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.572418+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -241> 2021-11-16T21:23:40.572+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785913520 auth_reply(proto 2 0 (0) Success) v1 debug -240> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634431+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -239> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ef00 0x5607859164e0 auth_reply(proto 2 0 (0) Success) v1 debug -238> 2021-11-16T21:23:40.634+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.634908+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -237> 2021-11-16T21:23:40.634+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eff0 0x560785913d40 auth_reply(proto 2 0 (0) Success) v1 debug -236> 2021-11-16T21:23:40.696+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.696476+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -235> 2021-11-16T21:23:40.696+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x5607858e2340 auth_reply(proto 2 0 (0) Success) v1 debug -234> 2021-11-16T21:23:40.697+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557609) is_readable = 1 - now=2021-11-16T21:23:40.697195+0000 lease_expire=2021-11-16T21:23:44.683434+0000 has v0 lc 19557609 debug -233> 2021-11-16T21:23:40.697+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x560785913860 auth_reply(proto 2 0 (0) Success) v1 debug -232> 2021-11-16T21:23:40.784+0000 7f1d716db700 4 set_mon_vals no callback set debug -231> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -230> 2021-11-16T21:23:40.785+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -229> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -228> 2021-11-16T21:23:40.785+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557610) is_readable = 1 - now=2021-11-16T21:23:40.785527+0000 lease_expire=2021-11-16T21:23:45.757378+0000 has v0 lc 19557610 debug -227> 2021-11-16T21:23:40.867+0000 7f1d716db700 4 set_mon_vals no callback set debug -226> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -225> 2021-11-16T21:23:40.868+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -224> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -223> 2021-11-16T21:23:40.868+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40 .868717+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -222> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910205+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -221> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152b40 0x5607858e9ba0 auth_reply(proto 2 0 (0) Success) v1 debug -220> 2021-11-16T21:23:40.910+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:40.910749+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -219> 2021-11-16T21:23:40.910+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x5607858ea340 auth_reply(proto 2 0 (0) Success) v1 debug -218> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038619+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -217> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153a40 0x560785910820 auth_reply(proto 2 0 (0) Success) v1 debug -216> 2021-11-16T21:23:41.038+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.038981+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -215> 2021-11-16T21:23:41.038+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152780 0x560785911520 auth_reply(proto 2 0 (0) Success) v1 debug -214> 2021-11-16T21:23:41.082+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.082731+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -213> 2021-11-16T21:23:41.082+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152ff0 0x5607859111e0 auth_reply(proto 2 0 (0) Success) v1 debug -212> 2021-11-16T21:23:41.083+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.083295+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -211> 2021-11-16T21:23:41.083+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152e10 0x560785911380 auth_reply(proto 2 0 (0) Success) v1 debug -210> 2021-11-16T21:23:41.167+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168120+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -209> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eaf00 0x5607859109c0 auth_reply(proto 2 0 (0) Success) v1 debug -208> 2021-11-16T21:23:41.168+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.168780+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -207> 2021-11-16T21:23:41.168+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607853eac30 0x560785911040 auth_reply(proto 2 0 (0) Success) v1 debug -206> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264181+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -205> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e0f0 0x560785910b60 auth_reply(proto 2 0 (0) Success) v1 debug -204> 2021-11-16T21:23:41.264+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.264798+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -203> 2021-11-16T21:23:41.264+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589ed20 0x560785910d00 auth_reply(proto 2 0 (0) Success) v1 debug -202> 2021-11-16T21:23:41.360+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.360828+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -201> 2021-11-16T21:23:41.360+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785153680 0x560785910ea0 auth_reply(proto 2 0 (0) Success) v1 debug -200> 2021-11-16T21:23:41.361+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.361219+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -199> 2021-11-16T21:23:41.361+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152f00 0x560785910000 auth_reply(proto 2 0 (0) Success) v1 debug -198> 2021-11-16T21:23:41.475+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.475815+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -197> 2021-11-16T21:23:41.475+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152000 0x56078590f6c0 auth_reply(proto 2 0 (0) Success) v1 debug -196> 2021-11-16T21:23:41.476+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.476381+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -195> 2021-11-16T21:23:41.476+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851525a0 0x56078590fd40 auth_reply(proto 2 0 (0) Success) v1 debug -194> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.531532+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -193> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x560785152a50 0x56078590f860 auth_reply(proto 2 0 (0) Success) v1 debug -192> 2021-11-16T21:23:41.531+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557611) is_readable = 1 - now=2021-11-16T21:23:41.532037+0000 lease_expire=2021-11-16T21:23:45.840970+0000 has v0 lc 19557611 debug -191> 2021-11-16T21:23:41.531+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x5607851520f0 0x56078590fa00 auth_reply(proto 2 0 (0) Success) v1 debug -190> 2021-11-16T21:23:42.200+0000 7f1d716db700 4 set_mon_vals no callback set debug -189> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -188> 2021-11-16T21:23:42.201+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -187> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -186> 2021-11-16T21:23:42.201+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557612) is_readable = 1 - now=2021-11-16T21:23:42.202649+0000 lease_expire=2021-11-16T21:23:47.173895+0000 has v0 lc 19557612 debug -185> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 set_mon_vals no callback set debug -184> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -183> 2021-11-16T21:23:42.292+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -182> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -181> 2021-11-16T21:23:42.292+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557613) is_readable = 1 - now=2021-11-16T21:23:42.294055+0000 lease_expire=2021-11-16T21:23:47.265226+0000 has v0 lc 19557613 debug -180> 2021-11-16T21:23:43.283+0000 7f1d716db700 4 set_mon_vals no callback set debug -179> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -178> 2021-11-16T21:23:43.285+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -177> 2021-11-1 6T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -176> 2021-11-16T21:23:43.285+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557614) is_readable = 1 - now=2021-11-16T21:23:43.286367+0000 lease_expire=2021-11-16T21:23:48.257095+0000 has v0 lc 19557614 debug -175> 2021-11-16T21:23:43.384+0000 7f1d716db700 4 set_mon_vals no callback set debug -174> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -173> 2021-11-16T21:23:43.385+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -172> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -171> 2021-11-16T21:23:43.385+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557615) is_readable = 1 - now=2021-11-16T21:23:43.386353+0000 lease_expire=2021-11-16T21:23:48.357251+0000 has v0 lc 19557615 debug -170> 2021-11-16T21:23:44.467+0000 7f1d716db700 4 set_mon_vals no callback set debug -169> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -168> 2021-11-16T21:23:44.468+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -167> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -166> 2021-11-16T21:23:44.468+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557616) is_readable = 1 - now=2021-11-16T21:23:44.469822+0000 lease_expire=2021-11-16T21:23:49.440271+0000 has v0 lc 19557616 debug -165> 2021-11-16T21:23:44.567+0000 7f1d716db700 4 set_mon_vals no callback set debug -164> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -163> 2021-11-16T21:23:44.568+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -162> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -161> 2021-11-16T21:23:44.568+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557617) is_readable = 1 - now=2021-11-16T21:23:44.569810+0000 lease_expire=2021-11-16T21:23:49.531775+0000 has v0 lc 19557617 debug -160> 2021-11-16T21:23:45.684+0000 7f1d716db700 4 set_mon_vals no callback set debug -159> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -158> 2021-11-16T21:23:45.685+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -157> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -156> 2021-11-16T21:23:45.685+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557618) is_readable = 1 - now=2021-11-16T21:23:45.686877+0000 lease_expire=2021-11-16T21:23:50.648710+0000 has v0 lc 19557618 debug -155> 2021-11-16T21:23:46.776+0000 7f1d716db700 4 set_mon_vals no callback set debug -154> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -153> 2021-11-16T21:23:46.777+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -152> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -151> 2021-11-16T21:23:46.777+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557619) is_readable = 1 - now=2021-11-16T21:23:46.778635+0000 lease_expire=2021-11-16T21:23:51.740053+0000 has v0 lc 19557619 debug -150> 2021-11-16T21:23:46.876+0000 7f1d716db700 4 set_mon_vals no callback set debug -149> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -148> 2021-11-16T21:23:46.877+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -147> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -146> 2021-11-16T21:23:46.877+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557620) is_readable = 1 - now=2021-11-16T21:23:46.878541+0000 lease_expire=2021-11-16T21:23:51.839977+0000 has v0 lc 19557620 debug -145> 2021-11-16T21:23:47.243+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #531955. Immutable memtables: 0. debug -144> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.244227) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 debug -143> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 531955 debug -142> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827244296, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 683, "num_deletes": 506, "total_data_size": 1462899, "memory_usage": 1475944, "flush_reason": "Manual Compaction"} debug -141> 2021-11-16T21:23:47.243+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #531956: started debug -140> 2021-11-16T21:23:47.243+0000 7f1d716db700 4 set_mon_vals no callback set debug -139> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -138> 2021-11-16T21:23:47.244+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -137> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -136> 2021-11-16T21:23:47.244+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557621) is_readable = 1 - now=2021-11-16T21:23:47.245909+0000 lease_expire=2021-11-16T21:23:52.207163+0000 has v0 lc 19557621 debug -135> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827293833, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 531956, "file_size": 966005, "table_properties": {"data_size": 962689, "index_size": 997, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1477, "raw_key_size": 12966, "raw_average_key_size": 22, "raw_value_size": 954089, "raw_average_value_size": 1667, "num_data_blocks": 29, "num_entries": 572, "num_deletions": 506, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637097809, "oldest_key_time": 1637097809, "file_creation_time": 1637097827}} debug -134> 2021-11-16T21:23:47.293+0000 7f1d76ee6700 4 rocksdb: [flush_job.cc:401] [default] [JOB 3] Level-0 flush table #531956: 966005 bytes OK debug -133> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.293881) [memtable_list.cc:447] [default] Level-0 commit table #531956 started debug -132> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327047) [memtable_list.cc:503] [default] Leve l-0 commit table #531956: memtable #1 done debug -131> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327058) EVENT_LOG_v1 {"time_micros": 1637097827327054, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} debug -130> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327073) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[3 0 0 0 0 0 2] max score 0.75 debug -129> 2021-11-16T21:23:47.326+0000 7f1d76ee6700 4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 3] Try to delete WAL files size 1458128, prev total WAL file size 1458455, number of live WAL files 2. debug -128> 2021-11-16T21:23:47.326+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -127> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:47.327469) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'logm .. 'logm; will stop at (end) debug -126> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 4] Compacting 3@0 + 2@6 files to L6, score -1.00 debug -125> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 3 Base level 0, inputs: [531956(943KB) 531950(148KB) 531947(7726KB)], [531944(64MB) 531945(36MB)] debug -124> 2021-11-16T21:23:47.326+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827327529, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [531956, 531950, 531947], "files_L6": [531944, 531945], "score": -1, "input_data_size": 114546999} debug -123> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531957: 3454 keys, 67198385 bytes debug -122> 2021-11-16T21:23:47.785+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097827785630, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531957, "file_size": 67198385, "table_properties": {"data_size": 67136484, "index_size": 52412, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 8645, "raw_key_size": 76781, "raw_average_key_size": 22, "raw_value_size": 67047935, "raw_average_value_size": 19411, "num_data_blocks": 1555, "num_entries": 3454, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}} debug -121> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 4] Generated table #531958: 1281 keys, 32015375 bytes debug -120> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828044065, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 531958, "file_size": 32015375, "table_properties": {"data_size": 31968485, "index_size": 42777, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 3269, "raw_key_size": 29915, "raw_average_key_size": 23, "raw_value_size": 31919151, "raw_average_value_size": 24917, "num_data_blocks": 1155, "num_entries": 1281, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1623830683, "oldest_key_time": 0, "file_creation_time": 1637097827}} debug -119> 2021-11-16T21:23:48.043+0000 7f1d776e7700 4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 4] Compacted 3@0 + 2@6 files to L6 => 99213760 bytes debug -118> 2021-11-16T21:23:48.066+0000 7f1d716db700 4 set_mon_vals no callback set debug -117> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -116> 2021-11-16T21:23:48.068+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -115> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -114> 2021-11-16T21:23:48.068+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557622) is_readable = 1 - now=2021-11-16T21:23:48.069418+0000 lease_expire=2021-11-16T21:23:52.960557+0000 has v0 lc 19557622 debug -113> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085683) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 159.9 rd, 138.5 wr, level 6, files in(3, 2) out(2) MB in(8.6, 100.6) out(94.6), read-write-amplify(23.7) write-amplify(11.0) OK, records in: 5811, records dropped: 1076 output_compression: NoCompression debug -112> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: (Original Log Time 2021/11/16-21:23:48.085701) EVENT_LOG_v1 {"time_micros": 1637097828085695, "job": 4, "event": "compaction_finished", "compaction_time_micros": 716586, "compaction_time_cpu_micros": 83813, "output_level": 6, "num_output_files": 2, "total_output_size": 99213760, "num_input_records": 5811, "num_output_records": 4735, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]} debug -111> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085923, "job": 4, "event": "table_file_deletion", "file_number": 531956} debug -110> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828085969, "job": 4, "event": "table_file_deletion", "file_number": 531950} debug -109> 2021-11-16T21:23:48.085+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828086487, "job": 4, "event": "table_file_deletion", "file_number": 531947} debug -108> 2021-11-16T21:23:48.088+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828089419, "job": 4, "event": "table_file_deletion", "file_number": 531945} debug -107> 2021-11-16T21:23:48.093+0000 7f1d776e7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637097828094233, "job": 4, "event": "table_file_deletion", "file_number": 531944} debug -106> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -105> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -104> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -103> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -102> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc :1443] [default] Manual compaction starting debug -101> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -100> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -99> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -98> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -97> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -96> 2021-11-16T21:23:48.093+0000 7f1d6ced2700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug -95> 2021-11-16T21:23:49.260+0000 7f1d716db700 4 set_mon_vals no callback set debug -94> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -93> 2021-11-16T21:23:49.261+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -92> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -91> 2021-11-16T21:23:49.261+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557623) is_readable = 1 - now=2021-11-16T21:23:49.262917+0000 lease_expire=2021-11-16T21:23:54.223482+0000 has v0 lc 19557623 debug -90> 2021-11-16T21:23:50.002+0000 7f1d716db700 4 set_mon_vals no callback set debug -89> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -88> 2021-11-16T21:23:50.003+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -87> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -86> 2021-11-16T21:23:50.003+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557624) is_readable = 1 - now=2021-11-16T21:23:50.004764+0000 lease_expire=2021-11-16T21:23:54.973508+0000 has v0 lc 19557624 debug -85> 2021-11-16T21:23:51.102+0000 7f1d716db700 4 set_mon_vals no callback set debug -84> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -83> 2021-11-16T21:23:51.103+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -82> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -81> 2021-11-16T21:23:51.103+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557625) is_readable = 1 - now=2021-11-16T21:23:51.104896+0000 lease_expire=2021-11-16T21:23:56.064982+0000 has v0 lc 19557625 debug -80> 2021-11-16T21:23:51.202+0000 7f1d716db700 4 set_mon_vals no callback set debug -79> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -78> 2021-11-16T21:23:51.203+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -77> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -76> 2021-11-16T21:23:51.203+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557626) is_readable = 1 - now=2021-11-16T21:23:51.204980+0000 lease_expire=2021-11-16T21:23:56.164769+0000 has v0 lc 19557626 debug -75> 2021-11-16T21:23:52.261+0000 7f1d716db700 4 set_mon_vals no callback set debug -74> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -73> 2021-11-16T21:23:52.262+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -72> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -71> 2021-11-16T21:23:52.262+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557627) is_readable = 1 - now=2021-11-16T21:23:52.263508+0000 lease_expire=2021-11-16T21:23:57.231263+0000 has v0 lc 19557627 debug -70> 2021-11-16T21:23:52.352+0000 7f1d716db700 4 set_mon_vals no callback set debug -69> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -68> 2021-11-16T21:23:52.353+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -67> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -66> 2021-11-16T21:23:52.354+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557628) is_readable = 1 - now=2021-11-16T21:23:52.355232+0000 lease_expire=2021-11-16T21:23:57.322723+0000 has v0 lc 19557628 debug -65> 2021-11-16T21:23:53.436+0000 7f1d716db700 4 set_mon_vals no callback set debug -64> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -63> 2021-11-16T21:23:53.437+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -62> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -61> 2021-11-16T21:23:53.437+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557629) is_readable = 1 - now=2021-11-16T21:23:53.438872+0000 lease_expire=2021-11-16T21:23:58.406301+0000 has v0 lc 19557629 debug -60> 2021-11-16T21:23:54.519+0000 7f1d716db700 4 set_mon_vals no callback set debug -59> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -58> 2021-11-16T21:23:54.520+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -57> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -56> 2021-11-16T21:23:54.521+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557630) is_readable = 1 - now=2021-11-16T21:23:54.522319+0000 lease_expire=2021-11-16T21:23:59.489316+0000 has v0 lc 19557630 debug -55> 2021-11-16T21:23:55.595+0000 7f1d716db700 4 set_mon_vals no callback set debug -54> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -53> 2021-11-16T21:23:55.596+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -52> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -51> 2021-11-16T21:23:55.596+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557631) is_readable = 1 - now=2021-11-16T21:23:55.597582+0000 lease_expire=2021-11-16T21:24:00.564365+0000 has v0 lc 19557631 debug -50> 2021-11-16T21:23:56.670+0000 7f1d716db700 4 set_mon_vals no callback set debug -49> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -48> 2021-11-16T21:23:56.671+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -47> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -46> 2021-11-16T21:23:56.671+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557632) is_readable = 1 - now=2021-11-16T21:23:56.672809+0000 lease_expire=2021-11-16T21:24:01.638988+0000 has v0 lc 19557632 debug -45> 2021-11-16T21:23:57.270+0000 7f1d716db700 4 set_mon_vals no callback set debug -44> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -43> 2021-11-16T21:23:57.271+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -42> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -41> 2021-11-16T21:23:57.271+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557633) is_readable = 1 - now=2021-11-16T21:23:57.272872+0000 lease_expire=2021-11-16T21:24:02.238966+0000 has v0 lc 19557633 debug -40> 2021-11-16T21:23:57.753+0000 7f1d716db700 4 set_mon_vals no callback set debug -39> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -38> 2021-11-16T21:23:57.755+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -37> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -36> 2021-11-16T21:23:57.755+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557634) is_readable = 1 - now=2021-11-16T21:23:57.756400+0000 lease_expire=2021-11-16T21:24:02.722461+0000 has v0 lc 19557634 debug -35> 2021-11-16T21:23:58.829+0000 7f1d716db700 4 set_mon_vals no callback set debug -34> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -33> 2021-11-16T21:23:58.830+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -32> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -31> 2021-11-16T21:23:58.830+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557635) is_readable = 1 - now=2021-11-16T21:23:58.831535+0000 lease_expire=2021-11-16T21:24:03.797240+0000 has v0 lc 19557635 debug -30> 2021-11-16T21:23:58.920+0000 7f1d716db700 4 set_mon_vals no callback set debug -29> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -28> 2021-11-16T21:23:58.921+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -27> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -26> 2021-11-16T21:23:58.921+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557636) is_readable = 1 - now=2021-11-16T21:23:58.922888+0000 lease_expire=2021-11-16T21:24:03.888665+0000 has v0 lc 19557636 debug -25> 2021-11-16T21:23:59.354+0000 7f1d716db700 4 set_mon_vals no callback set debug -24> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -23> 2021-11-16T21:23:59.355+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -22> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -21> 2021-11-16T21:23:59.355+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557637) is_readable = 1 - now=2021-11-16T21:23:59.356709+0000 lease_expire=2021-11-16T21:24:04.321986+0000 has v0 lc 19557637 debug -20> 2021-11-16T21:24:00.462+0000 7f1d716db700 4 set_mon_vals no callback set debug -19> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -18> 2021-11-16T21:24:00.463+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -17> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -16> 2021-11-16T21:24:00.464+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557638) is_readable = 1 - now=2021-11-16T21:24:00.465271+0000 lease_expire=2021-11-16T21:24:05.430188+0000 has v0 lc 19557638 debug -15> 2021-11-16T21:24:00.554+0000 7f1d716db700 4 set_mon_vals no callback set debug -14> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -13> 2021-11-16T21:24:00.555+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -12> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -11> 2021-11-16T21:24:00.555+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557639) is_readable = 1 - now=2021-11-16T21:24:00.556869+0000 lease_expire=2021-11-16T21:24:05.521766+0000 has v0 lc 19557639 debug -10> 2021-11-16T21:24:01.629+0000 7f1d716db700 4 set_mon_vals no callback set debug -9> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Got map version 500 debug -8> 2021-11-16T21:24:01.631+0000 7f1d716db700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/4203565106,v1:192.168.30.12:6849/4203565106] debug -7> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).monmap v69 apply_mon_features wait for service to be writeable debug -6> 2021-11-16T21:24:01.631+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.632342+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -5> 2021-11-16T21:24:01.873+0000 7f1d706d9700 0 --1- [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] >> conn(0x5607851bbc00 0x560784fc7800 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.100.19:52620/0) debug -4> 2021-11-16T21:24:01.873+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875099+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -3> 2021-11-16T21:24:01.873+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589e780 0x56078525b380 auth_reply(proto 2 0 (0) Success) v1 debug -2> 2021-11-16T21:24:01.874+0000 7f1d716db700 5 mon.cn03@2(peon).paxos(paxos active c 19556917..19557640) is_readable = 1 - now=2021-11-16T21:24:01.875450+0000 lease_expire=2021-11-16T21:24:06.605306+0000 has v0 lc 19557640 debug -1> 2021-11-16T21:24:01.874+0000 7f1d716db700 2 mon.cn03@2(peon) e69 send_reply 0x56078589eb40 0x560785259d40 auth_reply(proto 2 0 (0) Success) v1 debug 0> 2021-11-16T21:24:01.876+0000 7f1d716db700 -1 *** Caught signal (Aborted) ** in thread 7f1d716db700 thread_name:ms_dispatch ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1d7cd4eb20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f1d7c36c09b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f1d7c37253c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f1d7c372597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f1d7c3727f8] 8: /lib64/libstdc++.so.6(+0x92045) [0x7f1d7c36e045] 9: /usr/bin/ceph-mon(+0x4d8da6) [0x5607830d8da6] 10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x5607830cf329] 11: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x560782ec1258] 12: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x560782ee72ed] 13: (Monitor::_ms_dispatch(Message*)+0x670) [0x560782ee8910] 14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x560782f16fdc] 15: (DispatchQueue::entry()+0x126a) [0x7f1d7f48eb1a] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7f53eb71] 17: /lib64/libpthread.so.0(+0x814a) [0x7f1d7cd4414a] 18: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 139764358260480 / rstore_compact 139764375045888 / ms_dispatch 139764391831296 / rocksdb:dump_st 139764417009408 / msgr-worker-0 139764433794816 / ms_dispatch 139764450580224 / cpu_tp 139764467365632 / cpu_tp 139764475758336 / safe_timer 139764526114560 / rocksdb:high0 139764534507264 / rocksdb:low0 139764560197376 / admin_socket 139764813260544 / ceph-mon max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2021-11-16T21:24:01.877284Z_ec6af590-8752-4032-b581-ebb6ced84e72/log --- end dump of recent events --- meta file from the crash dump: [ceph: root@cn03 2021-11-16T21:24:41.375317Z_4cc0b19c-e1b1-4f7c-8ed4-780a7e64dd46]# cat meta { "crash_id": "2021-11-16T21:24:41.375317Z_4cc0b19c-e1b1-4f7c-8ed4-780a7e64dd46", "timestamp": "2021-11-16T21:24:41.375317Z", "process_name": "ceph-mon", "entity_name": "mon.cn03", "ceph_version": "16.2.4", "utsname_hostname": "cn03.ceph.la1.clx.corp", "utsname_sysname": "Linux", "utsname_release": "5.15.1-1.el8.elrepo.x86_64", "utsname_version": "#1 SMP Thu Nov 4 19:14:37 EDT 2021", "utsname_machine": "x86_64", "os_name": "CentOS Linux", "os_id": "centos", "os_version_id": "8", "os_version": "8", "backtrace": [ "/lib64/libpthread.so.0(+0x12b20) [0x7f9fea30db20]", "gsignal()", "abort()", "/lib64/libstdc++.so.6(+0x9009b) [0x7f9fe992b09b]", "/lib64/libstdc++.so.6(+0x9653c) [0x7f9fe993153c]", "/lib64/libstdc++.so.6(+0x96597) [0x7f9fe9931597]", "/lib64/libstdc++.so.6(+0x967f8) [0x7f9fe99317f8]", "/lib64/libstdc++.so.6(+0x92045) [0x7f9fe992d045]", "/usr/bin/ceph-mon(+0x4d8da6) [0x5610db0d8da6]", "(MDSMonitor::check_sub(Subscription*)+0x819) [0x5610db0cf329]", "(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xcd8) [0x5610daec1258]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d) [0x5610daee72ed]", "(Monitor::_ms_dispatch(Message*)+0x670) [0x5610daee8910]", "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5610daf16fdc]", "(DispatchQueue::entry()+0x126a) [0x7f9feca4db1a]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7f9fecafdb71]", "/lib64/libpthread.so.0(+0x814a) [0x7f9fea30314a]", "clone()" ] } Thank you -jeremy On Tuesday, Nov 16, 2021 at 5:58 AM, 胡 玮文 <huww98@xxxxxxxxxxx<mailto:huww98@xxxxxxxxxxx>> wrote: Hi Jeremy. Since you say the mons fail, could you share the logs of the failing mons? It is hard to diagnostic with little information. 发件人: Jeremy Hansen<mailto:jeremy@xxxxxxxxxx> 发送时间: 2021年11月16日 19:27 收件人: ceph-users<mailto:ceph-users@xxxxxxx> 主题: Re: mons fail as soon as I attempt to mount No ideas on this? It has me stumped. I’m going to revert my kernel upgrades if there isn’t a more logical explanation. Thanks > On Monday, Nov 15, 2021 at 2:33 AM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote: > This post references a kernel issue: > > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/6JDAPD5IR46JI6R6YGWQORDJTZ5Z2FIU/ > > I recently updated my ceph nodes to 5.15.1. Could this be my issue? > > -jeremy > > > > > > On Sunday, Nov 14, 2021 at 4:37 PM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote: > > I’m trying to mount a cephfs volume from a new machine. For some reason, it looks like all the mons fail when attempting to mount: > > > > [root@btc04 ~]# mount -t ceph :/ /mnt/ceph -o name=btc > > mount error: no mds server is up or the cluster is laggy > > [root@btc04 ~]# rpm -qa | grep ceph > > python3-cephfs-16.2.4-0.el8.x86_64 > > ceph-common-16.2.4-0.el8.x86_64 > > cephadm-16.2.4-0.el8.noarch > > python3-ceph-argparse-16.2.4-0.el8.x86_64 > > libcephfs2-16.2.4-0.el8.x86_64 > > python3-ceph-common-16.2.4-0.el8.x86_64 > > > > > > [ 51.105212] libceph: loaded (mon/osd proto 15/24) > > [ 51.145564] ceph: loaded (mds proto 32) > > [ 51.164266] libceph: mon3 (1)192.168.30.14:6789 session established > > [ 70.199453] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state OPEN) > > [ 70.199464] libceph: mon3 (1)192.168.30.14:6789 session lost, hunting for new mon > > [ 70.204400] libceph: mon0 (1)192.168.30.11:6789 session established > > [ 70.771652] libceph: mon0 (1)192.168.30.11:6789 socket closed (con state OPEN) > > [ 70.771670] libceph: mon0 (1)192.168.30.11:6789 session lost, hunting for new mon > > [ 70.774588] libceph: mon4 (1)192.168.30.15:6789 session established > > [ 71.234037] libceph: mon4 (1)192.168.30.15:6789 socket closed (con state OPEN) > > [ 71.234055] libceph: mon4 (1)192.168.30.15:6789 session lost, hunting for new mon > > [ 77.904722] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > [ 78.160614] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > [ 78.664602] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > [ 79.824787] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > [ 81.808526] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > [ 85.840430] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state V1_BANNER) > > > > > > Not really sure why… > > > > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.data all > > size: 4 > > min_size: 2 > > pg_num: 32 > > pgp_num: 32 > > crush_rule: replicated_rule > > hashpspool: true > > nodelete: false > > nopgchange: false > > nosizechange: false > > write_fadvise_dontneed: false > > noscrub: false > > nodeep-scrub: false > > use_gmt_hitset: 1 > > fast_read: 0 > > pg_autoscale_mode: on > > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.meta all > > size: 4 > > min_size: 2 > > pg_num: 32 > > pgp_num: 32 > > crush_rule: replicated_rule > > hashpspool: true > > nodelete: false > > nopgchange: false > > nosizechange: false > > write_fadvise_dontneed: false > > noscrub: false > > nodeep-scrub: false > > use_gmt_hitset: 1 > > fast_read: 0 > > recovery_priority: 5 > > pg_autoscale_mode: on > > pg_num_min: 16 > > pg_autoscale_bias: 4 > > > > > > The cluster becomes unhealthy but then clears shortly after the client times out. > > > > [ceph: root@cn01 /]# ceph -s > > cluster: > > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > health: HEALTH_OK > > > > services: > > mon: 5 daemons, quorum cn05,cn02,cn03,cn04,cn01 (age 6m) > > mgr: cn05.vpuwau(active, since 6d), standbys: cn02.arszct > > mds: 2/2 daemons up, 4 standby > > osd: 35 osds: 35 up (since 2d), 35 in (since 6d) > > > > data: > > volumes: 2/2 healthy > > pools: 6 pools, 289 pgs > > objects: 6.73M objects, 4.3 TiB > > usage: 17 TiB used, 108 TiB / 126 TiB avail > > pgs: 289 active+clean > > > > io: > > client: 0 B/s rd, 105 KiB/s wr, 2 op/s rd, 13 op/s wr > > > > > > > > -jeremy > > > > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |