On Tue, Dec 5, 2023 at 11:52 AM Eugen Block <eblock@xxxxxx> wrote: > You didn't paste the full output from the MON, I already asked for the > --no-pager output: > > journalctl -u --no-pager > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service Yeah, that would help. Even so, the output does contain some of the backtrace. It looks like this tracker issue: https://tracker.ceph.com/issues/56901 Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing an instance of 'std::invalid_argument' Dec 05 15:23:17 node01 bash[1969]: what(): stoull ... Dec 05 15:23:17 node01 bash[1969]: 9: (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] Dec 05 15:23:17 node01 bash[1969]: 10: (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] Dec 05 15:23:17 node01 bash[1969]: 11: (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) [0x55be1dbc93ad] So, here: https://github.com/ceph/ceph/blob/ece0d60728c6966000bb839fe46b2b8e2bdb5305/src/mon/LogMonitor.cc#L448 Looking at the bits on the ground, my best guess is that safe_read is stumbling into an EOF condition. In this case, log_external_backlog-->read_meta-->safe_read still returns 0 (even though it was EOF and arguably not a "success" condition), hence leaving the std::string where the configuration key is supposed to be loaded uninitialized and thus resulting in the abort in log_external_backlog(). Without a copy of a core or copy of the mon store to poke at, it may be tricky to recover this mon in its current state. Maybe check dmesg for hardware failures on a whim given the circumstances? Cheers, Tyler > Zitat von Manolis Daramas <mdaramas@xxxxxxxxxxxx>: > > > When this was first setup we had run this: > > > > ceph orch apply osd --all-available-devices > > > > Should not identify the newly added disk itself and add it on its own ? > > Regarding the mon.node01 daemon, did you see the output ? > > Do you understand what has caused this and what is the reason of not > > starting up ? > > I am stuck and do not know what to run in order to proceed. > > May you please guide me ? > > > > Thanks, > > > > Manolis > > > > > > -----Original Message----- > > From: Eugen Block <eblock@xxxxxx> > > Sent: Tuesday, December 5, 2023 5:45 PM > > To: Manolis Daramas <mdaramas@xxxxxxxxxxxx> > > Cc: ceph-users@xxxxxxx > > Subject: Re: Re: After hardware failure tried to > > recover ceph and followed instructions for recovery using OSDS > > > > Try to deploy the new osd so the PGs can recover as far as possible, > > wait for recovery. > > Try to bring the third MON back online while PGs recover. Then we'll > > see how far you get. > > > > Zitat von Manolis Daramas <mdaramas@xxxxxxxxxxxx>: > > > >> It has now removed it successfully. > >> > >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > >> -1 1.65987 root default > >> -3 0.29289 host node01 > >> 0 hdd 0.14650 osd.0 up 1.00000 1.00000 > >> 4 hdd 0.04880 osd.4 up 1.00000 1.00000 > >> 8 hdd 0.04880 osd.8 up 1.00000 1.00000 > >> 10 hdd 0.04880 osd.10 up 1.00000 1.00000 > >> -5 0.68349 host node02 > >> 2 hdd 0.14650 osd.2 up 1.00000 1.00000 > >> 5 hdd 0.04880 osd.5 up 1.00000 1.00000 > >> 7 hdd 0.04880 osd.7 up 1.00000 1.00000 > >> 9 hdd 0.04880 osd.9 up 1.00000 1.00000 > >> 12 hdd 0.39059 osd.12 up 1.00000 1.00000 > >> -7 0.68349 host node03 > >> 1 hdd 0.14650 osd.1 up 1.00000 1.00000 > >> 3 hdd 0.04880 osd.3 up 1.00000 1.00000 > >> 6 hdd 0.04880 osd.6 up 1.00000 1.00000 > >> 11 hdd 0.04880 osd.11 up 1.00000 1.00000 > >> 13 hdd 0.39059 osd.13 up 1.00000 1.00000 > >> > >> What are the next steps now ? > >> > >> Thanks, > >> > >> Manolis Daramas > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: Eugen Block <eblock@xxxxxx> > >> Sent: Tuesday, December 5, 2023 4:21 PM > >> To: Manolis Daramas <mdaramas@xxxxxxxxxxxx> > >> Cc: ceph-users@xxxxxxx > >> Subject: Re: Re: After hardware failure tried to > >> recover ceph and followed instructions for recovery using OSDS > >> > >> And how about this? > >> > >> osd crush rm osd.14 > >> > >> Zitat von Manolis Daramas <mdaramas@xxxxxxxxxxxx>: > >> > >>> Hi Eugen, > >>> > >>> I think that it cannot even be removed with ceph osd purge command, > >>> see below: > >>> > >>> root@node01:~# ceph osd purge 14 --force --yes-i-really-mean-it > >>> osd.14 does not exist > >>> > >>> root@node01:~# ceph osd tree > >>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > >>> -1 2.05046 root default > >>> -3 0.68349 host node01 > >>> 0 hdd 0.14650 osd.0 up 1.00000 1.00000 > >>> 4 hdd 0.04880 osd.4 up 1.00000 1.00000 > >>> 8 hdd 0.04880 osd.8 up 1.00000 1.00000 > >>> 10 hdd 0.04880 osd.10 up 1.00000 1.00000 > >>> 14 hdd 0.39059 osd.14 DNE 0 > >>> -5 0.68349 host node02 > >>> 2 hdd 0.14650 osd.2 up 1.00000 1.00000 > >>> 5 hdd 0.04880 osd.5 up 1.00000 1.00000 > >>> 7 hdd 0.04880 osd.7 up 1.00000 1.00000 > >>> 9 hdd 0.04880 osd.9 up 1.00000 1.00000 > >>> 12 hdd 0.39059 osd.12 up 1.00000 1.00000 > >>> -7 0.68349 host node03 > >>> 1 hdd 0.14650 osd.1 up 1.00000 1.00000 > >>> 3 hdd 0.04880 osd.3 up 1.00000 1.00000 > >>> 6 hdd 0.04880 osd.6 up 1.00000 1.00000 > >>> 11 hdd 0.04880 osd.11 up 1.00000 1.00000 > >>> 13 hdd 0.39059 osd.13 up 1.00000 1.00000 > >>> > >>> The output of journalctl with --no-pager option is below: > >>> > >>> -- Logs begin at Thu 2023-11-23 06:16:17 EET, end at Tue 2023-12-05 > >>> 16:01:49 EET. -- > >>> Dec 05 15:23:16 node01 systemd[1]: Started Ceph mon.node01 for > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37. > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > >>> (ceph:ceph) > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > >>> ceph-mon, pid 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > >>> empty --pid-file > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > >>> version: 6.15.5 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > >>> rocksdb_build_git_sha:@0@ > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > >>> Oct 25 2023 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > >>> 69YYUY1IJK4BA9QQDDK8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > >>> CURRENT > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > >>> IDENTITY > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > >>> MANIFEST-000226 size: 308 Bytes > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > >>> /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > >>> 000091.sst 000094.sst 000100.sst > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > >>> Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > >>> 0 ; > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.error_if_exists: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_if_missing: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.track_and_verify_wals_in_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.env: 0x55be1e3aea80 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.fs: Posix File System > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.info_log: 0x55be20918360 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_file_opening_threads: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.statistics: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_fsync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_log_file_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_manifest_file_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_file_time_to_roll: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.keep_log_file_num: 1000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.recycle_log_file_num: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_fallocate: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_writes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_io_for_flush_and_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_missing_column_families: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_log_dir: > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_dir: > >>> /var/lib/ceph/mon/ceph-node01/store.db > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_cache_numshardbits: 6 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_ttl_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_size_limit_MB: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_batch_group_size_bytes: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manifest_preallocation_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.is_fd_close_on_exec: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.advise_random_on_open: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_write_buffer_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_manager: 0x55be2168c990 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.access_hint_on_compaction_start: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.new_table_reader_for_compaction_inputs: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.random_access_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_adaptive_mutex: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limiter: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_file_manager.rate_bytes_per_sec: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_recovery_mode: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_thread_tracking: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_pipelined_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.unordered_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_concurrent_memtable_write: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_write_thread_adaptive_yield: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_max_yield_usec: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_slow_yield_usec: 3 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.row_cache: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_ingest_behind: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.preserve_deletes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.two_write_queues: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manual_wal_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.atomic_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_unnecessary_blocking_io: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.persist_stats_to_disk: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_dbid_to_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.file_checksum_gen_factory: Unknown > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.best_efforts_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bgerror_resume_count: 2147483647 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bgerror_resume_retry_interval: 1000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_data_in_errors: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_host_id: __hostname__ > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_jobs: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_compactions: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_subcompactions: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_shutdown: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.writable_file_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delayed_write_rate : 16777216 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_total_wal_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delete_obsolete_files_period_micros: 21600000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_dump_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_persist_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_history_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_open_files: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.strict_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_flushes: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > >>> algorithms supported: > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZSTDNotFinalCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > >>> supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kXpressCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4HCCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZlibCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kSnappyCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4Compression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kBZip2Compression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > >>> supported: Supported on x86 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4724] Recovering from manifest file: > >>> /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/column_family.cc:595] --------------- Options for column family > >>> [default]: > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.comparator: leveldb.BytewiseComparator > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.merge_operator: > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_partitioner_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_factory: SkipListFactory > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_factory: BlockBasedTable > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> table_factory options: flush_block_policy_factory: > >>> FlushBlockBySizePolicyFactory (0x55be2096afb0) > >>> Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> cache_index_and_filter_blocks_with_high_priority: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> pin_l0_filter_and_index_blocks_in_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> data_block_hash_table_util_ratio: 0.750000 > >>> Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > >>> Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > >>> Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > >>> Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > >>> Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > >>> Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: filter_policy: > >>> rocksdb.BuiltinBloomFilter > >>> Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_size: 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression: Disabled > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_insert_with_hint_prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.num_levels: 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_write_buffer_number_to_merge: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_size_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_file_num_compaction_trigger: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_slowdown_writes_trigger: 20 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_stop_writes_trigger: 36 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_base: 67108864 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_multiplier: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_base: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level_compaction_dynamic_level_bytes: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier: 10.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[0]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[1]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[2]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[3]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[4]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[5]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[6]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_sequential_skip_in_iterations: 8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_compaction_bytes: 1677721600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.arena_block_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.soft_pending_compaction_bytes_limit: 68719476736 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.hard_pending_compaction_bytes_limit: 274877906944 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limit_delay_max_milliseconds: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.disable_auto_compactions: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_style: kCompactionStyleLevel > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_pri: kMinOverlappingRatio > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.size_ratio: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.min_merge_width: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_merge_width: 4294967295 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_size_amplification_percent: > >>> 200 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.compression_size_percent: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.stop_style: > >>> kCompactionStopStyleTotalSize > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.max_table_files_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.allow_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_properties_collectors: > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_support: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_num_locks: 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_prefix_bloom_size_ratio: 0.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_whole_key_filtering: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_huge_page_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bloom_locality: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_successive_merges: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.optimize_filters_for_hits: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_file_checks: Dec 05 15:23:17 node01 bash[1969]: > >>> debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.force_consistency_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.report_bg_io_stats: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.ttl: 2592000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.periodic_compaction_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_files: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_blob_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_file_size: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_compression_type: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_garbage_collection: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_garbage_collection_age_cutoff: 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4764] Recovered from manifest > >>> file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> succeeded,manifest_file_number is 226, next_file_number is 228, > >>> last_sequence is 21201, log_number is 223,prev_log_number is > >>> 0,max_column_family is 0,min_log_number_to_keep is 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4779] Column family [default] (ID 0), log number > >>> is 223 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 230 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693048, "job": 1, "event": > >>> "recovery_started", "wal_files": [227]} > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 231 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693989, "job": 1, "event": > >>> "recovery_finished"} > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [file/delete_scheduler.cc:69] Deleted file > >>> /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > >>> rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > >>> 0x55be20931180 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > >>> 0x55be20a06000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:903] > >>> Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > >>> syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > >>> 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > >>> rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > >>> at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > >>> /var/lib/ceph/mon/ceph-node01 fsid > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > >>> preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 3314933000854323200, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing > >>> an instance of 'std::invalid_argument' > >>> Dec 05 15:23:17 node01 bash[1969]: what(): stoull > >>> Dec 05 15:23:17 node01 bash[1969]: *** Caught signal (Aborted) ** > >>> Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > >>> thread_name:ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > >>> Dec 05 15:23:17 node01 bash[1969]: 1: > >>> /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > >>> Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > >>> Dec 05 15:23:17 node01 bash[1969]: 3: abort() > >>> Dec 05 15:23:17 node01 bash[1969]: 4: > >>> /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > >>> Dec 05 15:23:17 node01 bash[1969]: 5: > >>> /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > >>> Dec 05 15:23:17 node01 bash[1969]: 6: > >>> /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > >>> Dec 05 15:23:17 node01 bash[1969]: 7: > >>> /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > >>> Dec 05 15:23:17 node01 bash[1969]: 8: > >>> /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > >>> Dec 05 15:23:17 node01 bash[1969]: 9: > >>> (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > >>> Dec 05 15:23:17 node01 bash[1969]: 10: > >>> (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > >>> Dec 05 15:23:17 node01 bash[1969]: 11: > >>> (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > >>> Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > >>> [0x55be1dbc93ad] > >>> Dec 05 15:23:17 node01 bash[1969]: 13: main() > >>> Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > >>> Dec 05 15:23:17 node01 bash[1969]: 15: _start() > >>> Dec 05 15:23:17 node01 bash[1969]: debug > >>> 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > >>> (Aborted) ** > >>> Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > >>> thread_name:ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > >>> Dec 05 15:23:17 node01 bash[1969]: 1: > >>> /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > >>> Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > >>> Dec 05 15:23:17 node01 bash[1969]: 3: abort() > >>> Dec 05 15:23:17 node01 bash[1969]: 4: > >>> /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > >>> Dec 05 15:23:17 node01 bash[1969]: 5: > >>> /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > >>> Dec 05 15:23:17 node01 bash[1969]: 6: > >>> /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > >>> Dec 05 15:23:17 node01 bash[1969]: 7: > >>> /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > >>> Dec 05 15:23:17 node01 bash[1969]: 8: > >>> /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > >>> Dec 05 15:23:17 node01 bash[1969]: 9: > >>> (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > >>> Dec 05 15:23:17 node01 bash[1969]: 10: > >>> (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > >>> Dec 05 15:23:17 node01 bash[1969]: 11: > >>> (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > >>> Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > >>> [0x55be1dbc93ad] > >>> Dec 05 15:23:17 node01 bash[1969]: 13: main() > >>> Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > >>> Dec 05 15:23:17 node01 bash[1969]: 15: _start() > >>> Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > >>> or `objdump -rdS <executable>` is needed to interpret this. > >>> Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > >>> Dec 05 15:23:17 node01 bash[1969]: debug -290> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command assert hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -289> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command abort hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -288> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command leak_some_memory hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -287> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perfcounters_dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -286> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 1 hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -285> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -284> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perfcounters_schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -283> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf histogram dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -282> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 2 hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -281> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -280> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf histogram schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -279> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf reset hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -278> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config show hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -277> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config help hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -276> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config set hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -275> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config unset hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -274> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config get hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -273> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config diff hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -272> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config diff get hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -271> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command injectargs hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -270> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log flush hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -269> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -268> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log reopen hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -267> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command dump_mempools hook 0x55be2168a068 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -266> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > >>> (ceph:ceph) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -265> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > >>> ceph-mon, pid 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -264> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > >>> empty --pid-file > >>> Dec 05 15:23:17 node01 bash[1969]: debug -263> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> init /var/run/ceph/ceph-mon.node01.asok > >>> Dec 05 15:23:17 node01 bash[1969]: debug -262> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> bind_and_listen /var/run/ceph/ceph-mon.node01.asok > >>> Dec 05 15:23:17 node01 bash[1969]: debug -261> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 0 hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -260> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command version hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -259> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command git_version hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -258> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command help hook 0x55be208de5b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -257> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command get_command_descriptions hook 0x55be208de5a0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -256> > >>> 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > >>> entry start > >>> Dec 05 15:23:17 node01 bash[1969]: debug -255> > >>> 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -254> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> level_compaction_dynamic_level_bytes = true > >>> Dec 05 15:23:17 node01 bash[1969]: debug -253> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> write_buffer_size = 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -252> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> compression = kNoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -251> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> level_compaction_dynamic_level_bytes = true > >>> Dec 05 15:23:17 node01 bash[1969]: debug -250> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> write_buffer_size = 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -249> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> compression = kNoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -248> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > >>> verify_sharding column families from rocksdb: [default] > >>> Dec 05 15:23:17 node01 bash[1969]: debug -247> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > >>> version: 6.15.5 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -246> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > >>> rocksdb_build_git_sha:@0@ > >>> Dec 05 15:23:17 node01 bash[1969]: debug -245> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > >>> Oct 25 2023 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -244> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > >>> Dec 05 15:23:17 node01 bash[1969]: debug -243> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > >>> 69YYUY1IJK4BA9QQDDK8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -242> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > >>> CURRENT > >>> Dec 05 15:23:17 node01 bash[1969]: debug -241> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > >>> IDENTITY > >>> Dec 05 15:23:17 node01 bash[1969]: debug -240> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > >>> MANIFEST-000226 size: 308 Bytes > >>> Dec 05 15:23:17 node01 bash[1969]: debug -239> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > >>> /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > >>> 000091.sst 000094.sst 000100.sst > >>> Dec 05 15:23:17 node01 bash[1969]: debug -238> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > >>> Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > >>> 0 ; > >>> Dec 05 15:23:17 node01 bash[1969]: debug -237> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.error_if_exists: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -236> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_if_missing: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -235> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -234> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.track_and_verify_wals_in_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -233> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.env: 0x55be1e3aea80 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -232> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.fs: Posix File System > >>> Dec 05 15:23:17 node01 bash[1969]: debug -231> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.info_log: 0x55be20918360 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -230> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_file_opening_threads: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -229> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.statistics: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -228> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_fsync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -227> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_log_file_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -226> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_manifest_file_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -225> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_file_time_to_roll: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -224> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.keep_log_file_num: 1000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -223> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.recycle_log_file_num: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -222> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_fallocate: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -221> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -220> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_writes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -219> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -218> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_io_for_flush_and_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -217> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_missing_column_families: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -216> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_log_dir: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -215> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_dir: > >>> /var/lib/ceph/mon/ceph-node01/store.db > >>> Dec 05 15:23:17 node01 bash[1969]: debug -214> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_cache_numshardbits: 6 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -213> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_ttl_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -212> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_size_limit_MB: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -211> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_batch_group_size_bytes: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -210> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manifest_preallocation_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -209> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.is_fd_close_on_exec: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -208> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.advise_random_on_open: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -207> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_write_buffer_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -206> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_manager: 0x55be2168c990 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -205> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.access_hint_on_compaction_start: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -204> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.new_table_reader_for_compaction_inputs: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -203> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.random_access_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -202> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_adaptive_mutex: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -201> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limiter: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -200> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_file_manager.rate_bytes_per_sec: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -199> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_recovery_mode: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -198> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_thread_tracking: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -197> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_pipelined_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -196> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.unordered_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -195> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_concurrent_memtable_write: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -194> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_write_thread_adaptive_yield: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -193> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_max_yield_usec: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -192> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_slow_yield_usec: 3 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -191> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.row_cache: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -190> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -189> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -188> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_ingest_behind: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -187> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.preserve_deletes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -186> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.two_write_queues: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -185> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manual_wal_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -184> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.atomic_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -183> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_unnecessary_blocking_io: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -182> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.persist_stats_to_disk: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -181> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_dbid_to_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -180> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -179> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.file_checksum_gen_factory: Unknown > >>> Dec 05 15:23:17 node01 bash[1969]: debug -178> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.best_efforts_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -177> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bgerror_resume_count: 2147483647 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -176> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bgerror_resume_retry_interval: 1000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -175> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_data_in_errors: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -174> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_host_id: __hostname__ > >>> Dec 05 15:23:17 node01 bash[1969]: debug -173> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_jobs: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -172> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_compactions: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -171> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_subcompactions: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -170> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_shutdown: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -169> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.writable_file_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -168> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delayed_write_rate : 16777216 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -167> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_total_wal_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -166> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delete_obsolete_files_period_micros: 21600000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -165> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_dump_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -164> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_persist_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -163> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_history_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -162> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_open_files: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -161> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -160> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -159> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.strict_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -158> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -157> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_flushes: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -156> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > >>> algorithms supported: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -155> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZSTDNotFinalCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -154> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > >>> supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -153> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kXpressCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -152> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4HCCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -151> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZlibCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -150> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kSnappyCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -149> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4Compression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -148> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kBZip2Compression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -147> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > >>> supported: Supported on x86 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -146> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4724] Recovering from manifest file: > >>> /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -145> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/column_family.cc:595] --------------- Options for column family > >>> [default]: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -144> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.comparator: leveldb.BytewiseComparator > >>> Dec 05 15:23:17 node01 bash[1969]: debug -143> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.merge_operator: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -142> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -141> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -140> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_partitioner_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -139> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_factory: SkipListFactory > >>> Dec 05 15:23:17 node01 bash[1969]: debug -138> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_factory: BlockBasedTable > >>> Dec 05 15:23:17 node01 bash[1969]: debug -137> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> table_factory options: flush_block_policy_factory: > >>> FlushBlockBySizePolicyFactory (0x55be2096afb0) > >>> Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> cache_index_and_filter_blocks_with_high_priority: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> pin_l0_filter_and_index_blocks_in_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> data_block_hash_table_util_ratio: 0.750000 > >>> Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > >>> Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > >>> Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > >>> Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > >>> Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > >>> Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: filter_policy: > >>> rocksdb.BuiltinBloomFilter > >>> Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -136> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_size: 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -135> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -134> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -133> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression: Disabled > >>> Dec 05 15:23:17 node01 bash[1969]: debug -132> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug -131> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_insert_with_hint_prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug -130> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.num_levels: 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -129> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_write_buffer_number_to_merge: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -128> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -127> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_size_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -126> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -125> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -124> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -123> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -122> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -121> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -120> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -119> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -118> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -117> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -116> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -115> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -114> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -113> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -112> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_file_num_compaction_trigger: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -111> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_slowdown_writes_trigger: 20 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -110> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_stop_writes_trigger: 36 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -109> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_base: 67108864 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -108> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_multiplier: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -107> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_base: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -106> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level_compaction_dynamic_level_bytes: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -105> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier: 10.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -104> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[0]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -103> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[1]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -102> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[2]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -101> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[3]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -100> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[4]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -99> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[5]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -98> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[6]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -97> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_sequential_skip_in_iterations: 8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -96> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_compaction_bytes: 1677721600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -95> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.arena_block_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -94> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.soft_pending_compaction_bytes_limit: 68719476736 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -93> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.hard_pending_compaction_bytes_limit: 274877906944 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -92> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limit_delay_max_milliseconds: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -91> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.disable_auto_compactions: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -90> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_style: kCompactionStyleLevel > >>> Dec 05 15:23:17 node01 bash[1969]: debug -89> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_pri: kMinOverlappingRatio > >>> Dec 05 15:23:17 node01 bash[1969]: debug -88> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.size_ratio: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -87> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.min_merge_width: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -86> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_merge_width: 4294967295 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -85> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_size_amplification_percent: > >>> 200 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -84> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.compression_size_percent: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -83> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.stop_style: > >>> kCompactionStopStyleTotalSize > >>> Dec 05 15:23:17 node01 bash[1969]: debug -82> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.max_table_files_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -81> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.allow_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -80> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_properties_collectors: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -79> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_support: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -78> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_num_locks: 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -77> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_prefix_bloom_size_ratio: 0.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -76> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_whole_key_filtering: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -75> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_huge_page_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -74> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bloom_locality: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -73> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_successive_merges: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -72> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.optimize_filters_for_hits: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -71> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_file_checks: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -70> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.force_consistency_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -69> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.report_bg_io_stats: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -68> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.ttl: 2592000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -67> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.periodic_compaction_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -66> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_files: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -65> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_blob_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -64> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_file_size: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -63> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_compression_type: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -62> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_garbage_collection: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -61> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_garbage_collection_age_cutoff: 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -60> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -59> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4764] Recovered from manifest > >>> file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> succeeded,manifest_file_number is 226, next_file_number is 228, > >>> last_sequence is 21201, log_number is 223,prev_log_number is > >>> 0,max_column_family is 0,min_log_number_to_keep is 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -58> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4779] Column family [default] (ID 0), log number > >>> is 223 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -57> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 230 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -56> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -55> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693048, "job": 1, "event": > >>> "recovery_started", "wal_files": [227]} > >>> Dec 05 15:23:17 node01 bash[1969]: debug -54> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -53> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 231 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -52> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -51> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693989, "job": 1, "event": > >>> "recovery_finished"} > >>> Dec 05 15:23:17 node01 bash[1969]: debug -50> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [file/delete_scheduler.cc:69] Deleted file > >>> /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > >>> rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -49> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > >>> 0x55be20931180 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -48> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > >>> 0x55be20a06000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -47> > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > >>> Dec 05 15:23:17 node01 bash[1969]: debug -46> > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:903] > >>> Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > >>> syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > >>> 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: debug -45> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -44> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -43> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -42> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: none > >>> Dec 05 15:23:17 node01 bash[1969]: debug -41> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -40> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -39> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -38> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -37> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -36> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -35> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -34> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -33> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -32> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -31> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -30> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -29> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > >>> loaded key file /var/lib/ceph/mon/ceph-node01/keyring > >>> Dec 05 15:23:17 node01 bash[1969]: debug -28> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > >>> rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > >>> at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > >>> /var/lib/ceph/mon/ceph-node01 fsid > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -27> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -26> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -25> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -24> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: none > >>> Dec 05 15:23:17 node01 bash[1969]: debug -23> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -22> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -21> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -20> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -19> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -18> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -17> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -16> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -15> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -14> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -13> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -12> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -11> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > >>> loaded key file /var/lib/ceph/mon/ceph-node01/keyring > >>> Dec 05 15:23:17 node01 bash[1969]: debug -10> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > >>> cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -9> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > >>> cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -8> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > >>> update_config to_monitors: true to_syslog: false syslog_facility: > >>> prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > >>> 12201) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -7> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > >>> update_config to_monitors: true to_syslog: false syslog_facility: > >>> prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > >>> 12201) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -6> > >>> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > >>> preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -5> > >>> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01@-1(???).mds > >>> e0 Unable to load 'last_metadata' > >>> Dec 05 15:23:17 node01 bash[1969]: debug -4> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 3314933000854323200, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -3> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -2> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -1> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug 0> > >>> 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > >>> (Aborted) ** > >>> Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > >>> thread_name:ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > >>> Dec 05 15:23:17 node01 bash[1969]: 1: > >>> /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > >>> Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > >>> Dec 05 15:23:17 node01 bash[1969]: 3: abort() > >>> Dec 05 15:23:17 node01 bash[1969]: 4: > >>> /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > >>> Dec 05 15:23:17 node01 bash[1969]: 5: > >>> /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > >>> Dec 05 15:23:17 node01 bash[1969]: 6: > >>> /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > >>> Dec 05 15:23:17 node01 bash[1969]: 7: > >>> /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > >>> Dec 05 15:23:17 node01 bash[1969]: 8: > >>> /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > >>> Dec 05 15:23:17 node01 bash[1969]: 9: > >>> (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > >>> Dec 05 15:23:17 node01 bash[1969]: 10: > >>> (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > >>> Dec 05 15:23:17 node01 bash[1969]: 11: > >>> (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > >>> Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > >>> [0x55be1dbc93ad] > >>> Dec 05 15:23:17 node01 bash[1969]: 13: main() > >>> Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > >>> Dec 05 15:23:17 node01 bash[1969]: 15: _start() > >>> Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > >>> or `objdump -rdS <executable>` is needed to interpret this. > >>> Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > >>> Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > >>> Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > >>> Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > >>> Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > >>> Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > >>> Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > >>> recent threads --- > >>> Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > >>> Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: log_file > >>> Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > >>> Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > >>> Dec 05 15:23:17 node01 bash[1969]: debug -290> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command assert hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -289> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command abort hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -288> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command leak_some_memory hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -287> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perfcounters_dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -286> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 1 hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -285> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -284> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perfcounters_schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -283> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf histogram dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -282> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 2 hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -281> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -280> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf histogram schema hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -279> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command perf reset hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -278> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config show hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -277> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config help hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -276> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config set hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -275> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config unset hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -274> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config get hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -273> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config diff hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -272> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command config diff get hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -271> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command injectargs hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -270> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log flush hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -269> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log dump hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -268> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command log reopen hook 0x55be208de8b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -267> > >>> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command dump_mempools hook 0x55be2168a068 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -266> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > >>> (ceph:ceph) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -265> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > >>> ceph-mon, pid 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -264> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > >>> empty --pid-file > >>> Dec 05 15:23:17 node01 bash[1969]: debug -263> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> init /var/run/ceph/ceph-mon.node01.asok > >>> Dec 05 15:23:17 node01 bash[1969]: debug -262> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> bind_and_listen /var/run/ceph/ceph-mon.node01.asok > >>> Dec 05 15:23:17 node01 bash[1969]: debug -261> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command 0 hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -260> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command version hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -259> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command git_version hook 0x55be2096b030 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -258> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command help hook 0x55be208de5b0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -257> > >>> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > >>> register_command get_command_descriptions hook 0x55be208de5a0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -256> > >>> 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > >>> entry start > >>> Dec 05 15:23:17 node01 bash[1969]: debug -255> > >>> 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -254> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> level_compaction_dynamic_level_bytes = true > >>> Dec 05 15:23:17 node01 bash[1969]: debug -253> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> write_buffer_size = 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -252> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> compression = kNoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -251> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> level_compaction_dynamic_level_bytes = true > >>> Dec 05 15:23:17 node01 bash[1969]: debug -250> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> write_buffer_size = 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -249> > >>> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > >>> compression = kNoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -248> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > >>> verify_sharding column families from rocksdb: [default] > >>> Dec 05 15:23:17 node01 bash[1969]: debug -247> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > >>> version: 6.15.5 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -246> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > >>> rocksdb_build_git_sha:@0@ > >>> Dec 05 15:23:17 node01 bash[1969]: debug -245> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > >>> Oct 25 2023 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -244> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > >>> Dec 05 15:23:17 node01 bash[1969]: debug -243> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > >>> 69YYUY1IJK4BA9QQDDK8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -242> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > >>> CURRENT > >>> Dec 05 15:23:17 node01 bash[1969]: debug -241> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > >>> IDENTITY > >>> Dec 05 15:23:17 node01 bash[1969]: debug -240> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > >>> MANIFEST-000226 size: 308 Bytes > >>> Dec 05 15:23:17 node01 bash[1969]: debug -239> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > >>> /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > >>> 000091.sst 000094.sst 000100.sst > >>> Dec 05 15:23:17 node01 bash[1969]: debug -238> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > >>> Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > >>> 0 ; > >>> Dec 05 15:23:17 node01 bash[1969]: debug -237> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.error_if_exists: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -236> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_if_missing: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -235> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -234> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.track_and_verify_wals_in_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -233> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.env: 0x55be1e3aea80 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -232> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.fs: Posix File System > >>> Dec 05 15:23:17 node01 bash[1969]: debug -231> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.info_log: 0x55be20918360 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -230> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_file_opening_threads: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -229> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.statistics: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -228> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_fsync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -227> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_log_file_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -226> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_manifest_file_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -225> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_file_time_to_roll: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -224> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.keep_log_file_num: 1000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -223> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.recycle_log_file_num: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -222> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_fallocate: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -221> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -220> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_mmap_writes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -219> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_reads: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -218> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_direct_io_for_flush_and_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -217> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.create_missing_column_families: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -216> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_log_dir: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -215> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_dir: > >>> /var/lib/ceph/mon/ceph-node01/store.db > >>> Dec 05 15:23:17 node01 bash[1969]: debug -214> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_cache_numshardbits: 6 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -213> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_ttl_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -212> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.WAL_size_limit_MB: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -211> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_batch_group_size_bytes: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -210> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manifest_preallocation_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -209> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.is_fd_close_on_exec: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -208> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.advise_random_on_open: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -207> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_write_buffer_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -206> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_manager: 0x55be2168c990 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -205> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.access_hint_on_compaction_start: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -204> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.new_table_reader_for_compaction_inputs: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -203> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.random_access_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -202> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.use_adaptive_mutex: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -201> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limiter: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -200> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_file_manager.rate_bytes_per_sec: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -199> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_recovery_mode: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -198> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_thread_tracking: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -197> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_pipelined_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -196> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.unordered_write: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -195> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_concurrent_memtable_write: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -194> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_write_thread_adaptive_yield: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -193> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_max_yield_usec: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -192> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_thread_slow_yield_usec: 3 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -191> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.row_cache: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -190> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -189> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -188> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_ingest_behind: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -187> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.preserve_deletes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -186> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.two_write_queues: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -185> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.manual_wal_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -184> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.atomic_flush: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -183> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_unnecessary_blocking_io: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -182> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.persist_stats_to_disk: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -181> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_dbid_to_manifest: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -180> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.log_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -179> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.file_checksum_gen_factory: Unknown > >>> Dec 05 15:23:17 node01 bash[1969]: debug -178> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.best_efforts_recovery: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -177> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bgerror_resume_count: 2147483647 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -176> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bgerror_resume_retry_interval: 1000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -175> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.allow_data_in_errors: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -174> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.db_host_id: __hostname__ > >>> Dec 05 15:23:17 node01 bash[1969]: debug -173> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_jobs: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -172> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_compactions: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -171> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_subcompactions: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -170> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.avoid_flush_during_shutdown: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -169> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.writable_file_max_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -168> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delayed_write_rate : 16777216 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -167> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_total_wal_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -166> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.delete_obsolete_files_period_micros: 21600000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -165> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_dump_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -164> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_persist_period_sec: 600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -163> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.stats_history_buffer_size: 1048576 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -162> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_open_files: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -161> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -160> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.wal_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -159> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.strict_bytes_per_sync: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -158> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_readahead_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -157> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_background_flushes: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -156> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > >>> algorithms supported: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -155> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZSTDNotFinalCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -154> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > >>> supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -153> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kXpressCompression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -152> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4HCCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -151> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kZlibCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -150> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kSnappyCompression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -149> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kLZ4Compression supported: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -148> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> kBZip2Compression supported: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -147> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > >>> supported: Supported on x86 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -146> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4724] Recovering from manifest file: > >>> /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -145> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> [db/column_family.cc:595] --------------- Options for column family > >>> [default]: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -144> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.comparator: leveldb.BytewiseComparator > >>> Dec 05 15:23:17 node01 bash[1969]: debug -143> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.merge_operator: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -142> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -141> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_filter_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -140> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.sst_partitioner_factory: None > >>> Dec 05 15:23:17 node01 bash[1969]: debug -139> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_factory: SkipListFactory > >>> Dec 05 15:23:17 node01 bash[1969]: debug -138> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_factory: BlockBasedTable > >>> Dec 05 15:23:17 node01 bash[1969]: debug -137> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> table_factory options: flush_block_policy_factory: > >>> FlushBlockBySizePolicyFactory (0x55be2096afb0) > >>> Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> cache_index_and_filter_blocks_with_high_priority: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> pin_l0_filter_and_index_blocks_in_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> data_block_hash_table_util_ratio: 0.750000 > >>> Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > >>> Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > >>> Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > >>> Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > >>> Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > >>> Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > >>> Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > >>> Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > >>> Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: filter_policy: > >>> rocksdb.BuiltinBloomFilter > >>> Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -136> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.write_buffer_size: 33554432 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -135> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -134> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -133> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression: Disabled > >>> Dec 05 15:23:17 node01 bash[1969]: debug -132> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug -131> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_insert_with_hint_prefix_extractor: nullptr > >>> Dec 05 15:23:17 node01 bash[1969]: debug -130> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.num_levels: 7 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -129> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_write_buffer_number_to_merge: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -128> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_number_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -127> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_write_buffer_size_to_maintain: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -126> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -125> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -124> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -123> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -122> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -121> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -120> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bottommost_compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -119> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.window_bits: -14 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -118> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.level: 32767 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -117> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.strategy: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -116> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.max_dict_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -115> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.zstd_max_train_bytes: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -114> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.parallel_threads: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -113> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compression_opts.enabled: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -112> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_file_num_compaction_trigger: 4 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -111> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_slowdown_writes_trigger: 20 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -110> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level0_stop_writes_trigger: 36 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -109> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_base: 67108864 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -108> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.target_file_size_multiplier: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -107> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_base: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -106> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.level_compaction_dynamic_level_bytes: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -105> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier: 10.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -104> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[0]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -103> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[1]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -102> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[2]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -101> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[3]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -100> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[4]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -99> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[5]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -98> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_bytes_for_level_multiplier_addtl[6]: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -97> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_sequential_skip_in_iterations: 8 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -96> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_compaction_bytes: 1677721600 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -95> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.arena_block_size: 4194304 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -94> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.soft_pending_compaction_bytes_limit: 68719476736 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -93> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.hard_pending_compaction_bytes_limit: 274877906944 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -92> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.rate_limit_delay_max_milliseconds: 100 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -91> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.disable_auto_compactions: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -90> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_style: kCompactionStyleLevel > >>> Dec 05 15:23:17 node01 bash[1969]: debug -89> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_pri: kMinOverlappingRatio > >>> Dec 05 15:23:17 node01 bash[1969]: debug -88> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.size_ratio: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -87> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.min_merge_width: 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -86> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_merge_width: 4294967295 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -85> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.max_size_amplification_percent: > >>> 200 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -84> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.compression_size_percent: -1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -83> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_universal.stop_style: > >>> kCompactionStopStyleTotalSize > >>> Dec 05 15:23:17 node01 bash[1969]: debug -82> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.max_table_files_size: 1073741824 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -81> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.compaction_options_fifo.allow_compaction: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -80> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.table_properties_collectors: > >>> Dec 05 15:23:17 node01 bash[1969]: debug -79> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_support: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -78> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.inplace_update_num_locks: 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -77> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_prefix_bloom_size_ratio: 0.000000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -76> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_whole_key_filtering: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -75> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.memtable_huge_page_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -74> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.bloom_locality: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -73> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.max_successive_merges: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -72> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.optimize_filters_for_hits: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -71> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.paranoid_file_checks: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -70> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.force_consistency_checks: 1 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -69> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.report_bg_io_stats: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -68> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.ttl: 2592000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -67> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.periodic_compaction_seconds: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -66> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_files: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -65> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.min_blob_size: 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -64> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_file_size: 268435456 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -63> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_compression_type: NoCompression > >>> Dec 05 15:23:17 node01 bash[1969]: debug -62> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.enable_blob_garbage_collection: false > >>> Dec 05 15:23:17 node01 bash[1969]: debug -61> > >>> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > >>> Options.blob_garbage_collection_age_cutoff: 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -60> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -59> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4764] Recovered from manifest > >>> file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > >>> succeeded,manifest_file_number is 226, next_file_number is 228, > >>> last_sequence is 21201, log_number is 223,prev_log_number is > >>> 0,max_column_family is 0,min_log_number_to_keep is 0 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -58> > >>> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4779] Column family [default] (ID 0), log number > >>> is 223 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -57> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 230 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -56> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -55> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693048, "job": 1, "event": > >>> "recovery_started", "wal_files": [227]} > >>> Dec 05 15:23:17 node01 bash[1969]: debug -54> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -53> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:4082] Creating manifest 231 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -52> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > >>> [db/version_set.cc:3457] More existing levels in DB than needed. > >>> max_bytes_for_level_multiplier may not be guaranteed. > >>> Dec 05 15:23:17 node01 bash[1969]: debug -51> > >>> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > >>> {"time_micros": 1701782597693989, "job": 1, "event": > >>> "recovery_finished"} > >>> Dec 05 15:23:17 node01 bash[1969]: debug -50> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [file/delete_scheduler.cc:69] Deleted file > >>> /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > >>> rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -49> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > >>> [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > >>> 0x55be20931180 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -48> > >>> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > >>> 0x55be20a06000 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -47> > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > >>> Dec 05 15:23:17 node01 bash[1969]: debug -46> > >>> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > >>> [db/db_impl/db_impl.cc:903] > >>> Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > >>> syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > >>> keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > >>> 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > >>> 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > >>> Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > >>> H:M:S, 0.0 percent > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > >>> Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > >>> 0.0 0.00 0.00 0 0.000 0 0 > >>> Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > >>> Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > >>> Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > >>> KeyIn KeyDrop > >>> Dec 05 15:23:17 node01 bash[1969]: > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > >>> Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > >>> interval 0.000 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > >>> 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > >>> interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > >>> Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > >>> write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > >>> Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > >>> 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > >>> level0_numfiles_with_compaction, 0 stop for > >>> pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > >>> memtable_compaction, 0 memtable_slowdown, interval 0 total count > >>> Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > >>> Level [default] ** > >>> Dec 05 15:23:17 node01 bash[1969]: debug -45> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -44> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -43> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -42> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding auth protocol: none > >>> Dec 05 15:23:17 node01 bash[1969]: debug -41> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -40> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -39> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -38> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -37> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -36> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -35> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -34> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -33> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -32> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -31> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -30> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167a140) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -29> > >>> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > >>> loaded key file /var/lib/ceph/mon/ceph-node01/keyring > >>> Dec 05 15:23:17 node01 bash[1969]: debug -28> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > >>> rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > >>> at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > >>> /var/lib/ceph/mon/ceph-node01 fsid > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -27> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -26> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -25> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -24> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding auth protocol: none > >>> Dec 05 15:23:17 node01 bash[1969]: debug -23> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -22> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -21> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -20> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -19> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -18> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -17> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -16> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -15> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -14> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -13> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: crc > >>> Dec 05 15:23:17 node01 bash[1969]: debug -12> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > >>> AuthRegistry(0x55be2167aa40) adding con mode: secure > >>> Dec 05 15:23:17 node01 bash[1969]: debug -11> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > >>> loaded key file /var/lib/ceph/mon/ceph-node01/keyring > >>> Dec 05 15:23:17 node01 bash[1969]: debug -10> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > >>> cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -9> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > >>> cephx > >>> Dec 05 15:23:17 node01 bash[1969]: debug -8> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > >>> update_config to_monitors: true to_syslog: false syslog_facility: > >>> prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > >>> 12201) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -7> > >>> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > >>> update_config to_monitors: true to_syslog: false syslog_facility: > >>> prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > >>> 12201) > >>> Dec 05 15:23:17 node01 bash[1969]: debug -6> > >>> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > >>> preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > >>> Dec 05 15:23:17 node01 bash[1969]: debug -5> > >>> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01@-1(???).mds > >>> e0 Unable to load 'last_metadata' > >>> Dec 05 15:23:17 node01 bash[1969]: debug -4> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 3314933000854323200, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -3> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -2> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug -1> > >>> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd > >>> e9898 crush map has features 432629239337189376, adjusting msgr > >>> requires > >>> Dec 05 15:23:17 node01 bash[1969]: debug 0> > >>> 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > >>> (Aborted) ** > >>> Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > >>> thread_name:ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > >>> Dec 05 15:23:17 node01 bash[1969]: 1: > >>> /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > >>> Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > >>> Dec 05 15:23:17 node01 bash[1969]: 3: abort() > >>> Dec 05 15:23:17 node01 bash[1969]: 4: > >>> /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > >>> Dec 05 15:23:17 node01 bash[1969]: 5: > >>> /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > >>> Dec 05 15:23:17 node01 bash[1969]: 6: > >>> /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > >>> Dec 05 15:23:17 node01 bash[1969]: 7: > >>> /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > >>> Dec 05 15:23:17 node01 bash[1969]: 8: > >>> /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > >>> Dec 05 15:23:17 node01 bash[1969]: 9: > >>> (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > >>> Dec 05 15:23:17 node01 bash[1969]: 10: > >>> (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > >>> Dec 05 15:23:17 node01 bash[1969]: 11: > >>> (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > >>> Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > >>> [0x55be1dbc93ad] > >>> Dec 05 15:23:17 node01 bash[1969]: 13: main() > >>> Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > >>> Dec 05 15:23:17 node01 bash[1969]: 15: _start() > >>> Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > >>> or `objdump -rdS <executable>` is needed to interpret this. > >>> Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > >>> Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > >>> Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > >>> Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > >>> Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > >>> Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > >>> Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > >>> Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > >>> Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > >>> Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > >>> recent threads --- > >>> Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > >>> Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > >>> Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > >>> Dec 05 15:23:17 node01 bash[1969]: log_file > >>> /var/lib/ceph/crash/2023-12-05T13:23:17.739483Z_174a5e49-22df-4c2f-be14-4e88cc65b2f2/log > >>> Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > >>> Dec 05 15:23:17 node01 systemd[1]: > >>> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: Main > >>> process exited, code=exited, status=134/n/a > >>> Dec 05 15:23:18 node01 systemd[1]: > >>> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: Failed > >>> with result 'exit-code'. > >>> Dec 05 15:23:29 node01 systemd[1]: > >>> ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: > >>> Scheduled restart job, restart counter is at 1. > >>> Dec 05 15:23:29 node01 systemd[1]: Stopped Ceph mon.node01 for > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37. > >>> Dec 05 15:23:29 node01 systemd[1]: Started Ceph mon.node01 for > >>> be4304e4-b0d5-11ec-8c6a-2965d4229f37. > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 set uid:gid to 167:167 > >>> (ceph:ceph) > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 ceph version 17.2.7 > >>> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > >>> ceph-mon, pid 7 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 pidfile_write: ignore > >>> empty --pid-file > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 0 load: jerasure load: lrc > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: RocksDB > >>> version: 6.15.5 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Git sha > >>> rocksdb_build_git_sha:@0@ > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compile date > >>> Oct 25 2023 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB SUMMARY > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB Session ID: > >>> OQS561SAY9M8P46I122B > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: CURRENT file: > >>> CURRENT > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: IDENTITY file: > >>> IDENTITY > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: MANIFEST file: > >>> MANIFEST-000231 size: 308 Bytes > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: SST files in > >>> /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > >>> 000091.sst 000094.sst 000100.sst > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Write Ahead > >>> Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000232.log size: > >>> 0 ; > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.error_if_exists: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.create_if_missing: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.paranoid_checks: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.track_and_verify_wals_in_manifest: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.env: 0x562241625a80 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.fs: Posix File System > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.info_log: 0x562243b9a360 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_file_opening_threads: 16 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.statistics: (nil) > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.use_fsync: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_log_file_size: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_manifest_file_size: 1073741824 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.log_file_time_to_roll: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.keep_log_file_num: 1000 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.recycle_log_file_num: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_fallocate: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_mmap_reads: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_mmap_writes: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.use_direct_reads: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.use_direct_io_for_flush_and_compaction: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.create_missing_column_families: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.db_log_dir: > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.wal_dir: > >>> /var/lib/ceph/mon/ceph-node01/store.db > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.table_cache_numshardbits: 6 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.WAL_ttl_seconds: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.WAL_size_limit_MB: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_write_batch_group_size_bytes: 1048576 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.manifest_preallocation_size: 4194304 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.is_fd_close_on_exec: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.advise_random_on_open: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.db_write_buffer_size: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.write_buffer_manager: 0x56224490e990 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.access_hint_on_compaction_start: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.new_table_reader_for_compaction_inputs: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.random_access_max_buffer_size: 1048576 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.use_adaptive_mutex: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.rate_limiter: (nil) > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.sst_file_manager.rate_bytes_per_sec: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.wal_recovery_mode: 2 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.enable_thread_tracking: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.enable_pipelined_write: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.unordered_write: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_concurrent_memtable_write: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.enable_write_thread_adaptive_yield: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.write_thread_max_yield_usec: 100 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.write_thread_slow_yield_usec: 3 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.row_cache: None > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.wal_filter: None > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.avoid_flush_during_recovery: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_ingest_behind: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.preserve_deletes: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.two_write_queues: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.manual_wal_flush: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.atomic_flush: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.avoid_unnecessary_blocking_io: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.persist_stats_to_disk: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.write_dbid_to_manifest: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.log_readahead_size: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.file_checksum_gen_factory: Unknown > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.best_efforts_recovery: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_bgerror_resume_count: 2147483647 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.bgerror_resume_retry_interval: 1000000 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.allow_data_in_errors: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.db_host_id: __hostname__ > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_background_jobs: 2 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_background_compactions: -1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_subcompactions: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.avoid_flush_during_shutdown: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.writable_file_max_buffer_size: 1048576 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.delayed_write_rate : 16777216 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_total_wal_size: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.delete_obsolete_files_period_micros: 21600000000 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.stats_dump_period_sec: 600 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.stats_persist_period_sec: 600 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.stats_history_buffer_size: 1048576 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_open_files: -1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.bytes_per_sync: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.wal_bytes_per_sync: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.strict_bytes_per_sync: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.compaction_readahead_size: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_background_flushes: -1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compression > >>> algorithms supported: > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kZSTDNotFinalCompression supported: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTD > >>> supported: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kXpressCompression supported: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kLZ4HCCompression supported: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kZlibCompression supported: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kSnappyCompression supported: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kLZ4Compression supported: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> kBZip2Compression supported: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Fast CRC32 > >>> supported: Supported on x86 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> [db/version_set.cc:4724] Recovering from manifest file: > >>> /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000231 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> [db/column_family.cc:595] --------------- Options for column family > >>> [default]: > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.comparator: leveldb.BytewiseComparator > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.merge_operator: > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.compaction_filter: None > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.compaction_filter_factory: None > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.sst_partitioner_factory: None > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.memtable_factory: SkipListFactory > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.table_factory: BlockBasedTable > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> table_factory options: flush_block_policy_factory: > >>> FlushBlockBySizePolicyFactory (0x562243becfb0) > >>> Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: > >>> cache_index_and_filter_blocks_with_high_priority: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: > >>> pin_l0_filter_and_index_blocks_in_cache: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: pin_top_level_index_and_filter: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: index_type: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: data_block_index_type: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: index_shortening: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: > >>> data_block_hash_table_util_ratio: 0.750000 > >>> Dec 05 15:23:29 node01 bash[6375]: hash_index_allow_collision: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: checksum: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: no_block_cache: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: block_cache: 0x562243c1f090 > >>> Dec 05 15:23:29 node01 bash[6375]: block_cache_name: BinnedLRUCache > >>> Dec 05 15:23:29 node01 bash[6375]: block_cache_options: > >>> Dec 05 15:23:29 node01 bash[6375]: capacity : 536870912 > >>> Dec 05 15:23:29 node01 bash[6375]: num_shard_bits : 4 > >>> Dec 05 15:23:29 node01 bash[6375]: strict_capacity_limit : 0 > >>> Dec 05 15:23:29 node01 bash[6375]: high_pri_pool_ratio: 0.000 > >>> Dec 05 15:23:29 node01 bash[6375]: block_cache_compressed: (nil) > >>> Dec 05 15:23:29 node01 bash[6375]: persistent_cache: (nil) > >>> Dec 05 15:23:29 node01 bash[6375]: block_size: 4096 > >>> Dec 05 15:23:29 node01 bash[6375]: block_size_deviation: 10 > >>> Dec 05 15:23:29 node01 bash[6375]: block_restart_interval: 16 > >>> Dec 05 15:23:29 node01 bash[6375]: index_block_restart_interval: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: metadata_block_size: 4096 > >>> Dec 05 15:23:29 node01 bash[6375]: partition_filters: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: use_delta_encoding: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: filter_policy: > >>> rocksdb.BuiltinBloomFilter > >>> Dec 05 15:23:29 node01 bash[6375]: whole_key_filtering: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: verify_compression: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: read_amp_bytes_per_bit: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: format_version: 4 > >>> Dec 05 15:23:29 node01 bash[6375]: enable_index_compression: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: block_align: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.write_buffer_size: 33554432 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_write_buffer_number: 2 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.compression: NoCompression > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.bottommost_compression: Disabled > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.prefix_extractor: nullptr > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.memtable_insert_with_hint_prefix_extractor: nullptr > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.num_levels: 7 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.min_write_buffer_number_to_merge: 1 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > >>> Options.max_write_buffer_number_to_maintain: 0 > >>> Dec 05 15:23:29 node01 bash[6375]: debug > >>> 2023-12-05T13:23:29.303+0000 7fd6a1 > > [Text gekürzt] > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |