Hello everyone,
We had a recent power failure on a server which hosts a 3-node ceph
cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think
that we may have lost some of our data if not all of them.
We have followed the instructions on
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds but with no
luck.
We have kept a backup of store.db folder on all 3 nodes prior the
below steps.
We have stopped ceph.target on all 3 nodes.
We have run the first part of the script and we have altered it
according to our configuration:
ms=/root/mon-store
mkdir $ms
hosts="node01 node02 node03"
# collect the cluster map from stopped OSDs
for host in $hosts; do
rsync -avz $ms/. root@$host:$ms.remote
rm -rf $ms
ssh root@$host <<EOF
for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do
ceph-objectstore-tool --data-path \$osd --no-mon-config --op
update-mon-db --mon-store-path $ms.remote
done
EOF
rsync -avz root@$host:$ms.remote/. $ms
done
and the results were:
for node01
sd.0 : 0 osdmaps trimmed, 673 osdmaps added.
osd.10 : 9225 osdmaps trimmed, 0 osdmaps added.
Mount failed with '(5) Input/output error'
osd.4 : 0 osdmaps trimmed, 0 osdmaps added.
osd.8 : 0 osdmaps trimmed, 0 osdmaps added.
receiving incremental file list
created directory /root/mon-store
./
kv_backend
store.db/
store.db/000008.sst
store.db/000014.sst
store.db/000020.sst
store.db/000022.log
store.db/CURRENT
store.db/IDENTITY
store.db/LOCK
store.db/MANIFEST-000021
store.db/OPTIONS-000018
store.db/OPTIONS-000024
sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec
total size is 7,869,025 speedup is 27.44
sending incremental file list
created directory /root/mon-store.remote
./
kv_backend
store.db/
store.db/000008.sst
store.db/000014.sst
store.db/000020.sst
store.db/000022.log
store.db/CURRENT
store.db/IDENTITY
store.db/LOCK
store.db/MANIFEST-000021
store.db/OPTIONS-000018
store.db/OPTIONS-000024
sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec
total size is 7,869,025 speedup is 27.44
for node02
osd.12 : 0 osdmaps trimmed, 0 osdmaps added.
osd.2 : 0 osdmaps trimmed, 0 osdmaps added.
osd.5 : 0 osdmaps trimmed, 0 osdmaps added.
osd.7 : 0 osdmaps trimmed, 0 osdmaps added.
osd.9 : 0 osdmaps trimmed, 0 osdmaps added.
receiving incremental file list
created directory /root/mon-store
./
kv_backend
store.db/
store.db/000008.sst
store.db/000014.sst
store.db/000020.sst
store.db/000026.sst
store.db/000032.sst
store.db/000038.sst
store.db/000044.sst
store.db/000050.sst
store.db/000052.log
store.db/CURRENT
store.db/IDENTITY
store.db/LOCK
store.db/MANIFEST-000051
store.db/OPTIONS-000048
store.db/OPTIONS-000054
sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec
total size is 7,875,746 speedup is 27.02
sending incremental file list
created directory /root/mon-store.remote
./
kv_backend
store.db/
store.db/000008.sst
store.db/000014.sst
store.db/000020.sst
store.db/000026.sst
store.db/000032.sst
store.db/000038.sst
store.db/000044.sst
store.db/000050.sst
store.db/000052.log
store.db/CURRENT
store.db/IDENTITY
store.db/LOCK
store.db/MANIFEST-000051
store.db/OPTIONS-000048
store.db/OPTIONS-000054
sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec
total size is 7,875,746 speedup is 27.02
for node03
osd.1 : 0 osdmaps trimmed, 0 osdmaps added.
osd.11 : 0 osdmaps trimmed, 0 osdmaps added.
osd.13 : 0 osdmaps trimmed, 0 osdmaps added.
osd.3 : 0 osdmaps trimmed, 0 osdmaps added.
osd.6 : 0 osdmaps trimmed, 0 osdmaps added.
receiving incremental file list
created directory /root/mon-store
./
kv_backend
store.db/
store.db/000008.sst
store.db/000014.sst
store.db/000020.sst
store.db/000026.sst
store.db/000032.sst
store.db/000038.sst
store.db/000044.sst
store.db/000050.sst
store.db/000056.sst
store.db/000062.sst
store.db/000068.sst
store.db/000074.sst
store.db/000080.sst
store.db/000082.log
store.db/CURRENT
store.db/IDENTITY
store.db/LOCK
store.db/MANIFEST-000081
store.db/OPTIONS-000078
store.db/OPTIONS-000084
sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec
total size is 7,882,477 speedup is 26.62
Then we have run the (in order to rebuild the monstore DB and fix it):
ceph-monstore-tool /root/mon-store rebuild -- --keyring
/etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03
and the output is below:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date
Oct 25 2023
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID:
OS2T69IQ02SU5OKHBI40
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file:
IDENTITY
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file:
MANIFEST-000081 size: 1083 Bytes
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in
/root/mon-store/store.db dir, Total Num: 13, files: 000008.sst
000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst
000050.sst 000056.sst
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead
Log file in /root/mon-store/store.db: 000082.log size: 244 ;
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.error_if_exists: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.create_if_missing: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.paranoid_checks: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.track_and_verify_wals_in_manifest: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.env: 0x56017c8d1c20
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.fs: Posix File System
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.info_log: 0x56017d4c3860
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_file_opening_threads: 16
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.statistics: (nil)
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.use_fsync: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_log_file_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_manifest_file_size: 1073741824
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.log_file_time_to_roll: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.keep_log_file_num: 1000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.recycle_log_file_num: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_fallocate: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_mmap_reads: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_mmap_writes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.use_direct_reads: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.use_direct_io_for_flush_and_compaction: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.create_missing_column_families: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.db_log_dir:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.wal_dir: /root/mon-store/store.db
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.table_cache_numshardbits: 6
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.WAL_ttl_seconds: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.WAL_size_limit_MB: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_write_batch_group_size_bytes: 1048576
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.manifest_preallocation_size: 4194304
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.is_fd_close_on_exec: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.advise_random_on_open: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.db_write_buffer_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.write_buffer_manager: 0x56017d1f6a20
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.access_hint_on_compaction_start: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.new_table_reader_for_compaction_inputs: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.random_access_max_buffer_size: 1048576
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.use_adaptive_mutex: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.rate_limiter: (nil)
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.wal_recovery_mode: 2
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.enable_thread_tracking: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.enable_pipelined_write: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.unordered_write: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_concurrent_memtable_write: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.enable_write_thread_adaptive_yield: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.write_thread_max_yield_usec: 100
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.write_thread_slow_yield_usec: 3
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.row_cache: None
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.wal_filter: None
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.avoid_flush_during_recovery: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_ingest_behind: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.preserve_deletes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.two_write_queues: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.manual_wal_flush: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.atomic_flush: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.avoid_unnecessary_blocking_io: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.persist_stats_to_disk: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.write_dbid_to_manifest: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.log_readahead_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.file_checksum_gen_factory: Unknown
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.best_efforts_recovery: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bgerror_resume_count: 2147483647
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bgerror_resume_retry_interval: 1000000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.allow_data_in_errors: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.db_host_id: __hostname__
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_background_jobs: 2
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_background_compactions: -1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_subcompactions: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.avoid_flush_during_shutdown: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.writable_file_max_buffer_size: 1048576
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.delayed_write_rate : 16777216
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_total_wal_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.stats_dump_period_sec: 600
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.stats_persist_period_sec: 600
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.stats_history_buffer_size: 1048576
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_open_files: -1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bytes_per_sync: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.wal_bytes_per_sync: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.strict_bytes_per_sync: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_readahead_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_background_flushes: -1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression
algorithms supported:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kZSTDNotFinalCompression supported: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kXpressCompression supported: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kLZ4HCCompression supported: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kLZ4Compression supported: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kBZip2Compression supported: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kZlibCompression supported: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
kSnappyCompression supported: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32
supported: Supported on x86
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
[db/version_set.cc:4724] Recovering from manifest file:
/root/mon-store/store.db/MANIFEST-000081
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
[db/column_family.cc:595] --------------- Options for column family
[default]:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.merge_operator:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_filter: None
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_filter_factory: None
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.sst_partitioner_factory: None
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.memtable_factory: SkipListFactory
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.table_factory: BlockBasedTable
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x56017d234f80)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x56017d22f610
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 4
enable_index_compression: 1
block_align: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.write_buffer_size: 33554432
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_write_buffer_number: 2
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression: NoCompression
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression: Disabled
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.prefix_extractor: nullptr
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.num_levels: 7
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_write_buffer_size_to_maintain: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.window_bits: -14
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.level: 32767
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.strategy: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.max_dict_bytes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.parallel_threads: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bottommost_compression_opts.enabled: false
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.window_bits: -14
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.level: 32767
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.strategy: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.zstd_max_train_bytes: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.parallel_threads: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compression_opts.enabled: false
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.level0_stop_writes_trigger: 36
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.target_file_size_base: 67108864
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.target_file_size_multiplier: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_compaction_bytes: 1677721600
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.arena_block_size: 4194304
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.disable_auto_compactions: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_pri: kMinOverlappingRatio
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent:
200
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.table_properties_collectors:
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.inplace_update_support: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.inplace_update_num_locks: 10000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.memtable_whole_key_filtering: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.memtable_huge_page_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.bloom_locality: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.max_successive_merges: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.optimize_filters_for_hits: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.paranoid_file_checks: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.force_consistency_checks: 1
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.report_bg_io_stats: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.ttl: 2592000
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.periodic_compaction_seconds: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.enable_blob_files: false
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.min_blob_size: 0
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.blob_file_size: 268435456
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.blob_compression_type: NoCompression
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.enable_blob_garbage_collection: false
2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb:
Options.blob_garbage_collection_age_cutoff: 0.250000
2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
[db/version_set.cc:4764] Recovered from manifest
file:/root/mon-store/store.db/MANIFEST-000081
succeeded,manifest_file_number is 81, next_file_number is 83,
last_sequence is 21183, log_number is 77,prev_log_number is
0,max_column_family is 0,min_log_number_to_keep is 0
2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
[db/version_set.cc:4779] Column family [default] (ID 0), log number
is 77
2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb:
[db/version_set.cc:4082] Creating manifest 85
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784162798, "job": 1, "event":
"recovery_started", "wal_files": [82]}
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
[db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2
2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb:
[table/block_based/filter_policy.cc:991] Using legacy Bloom filter
with high (20) bits/key. Dramatic filter space and/or accuracy
improvement is available with format_version>=5.
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784163944, "cf_name": "default", "job": 1,
"event": "table_file_creation", "file_number": 86, "file_size":
1266, "file_checksum": "", "file_checksum_func_name": "Unknown",
"table_properties": {"data_size": 238, "index_size": 40,
"index_partitions": 0, "top_level_index_size": 0,
"index_key_is_user_key": 1, "index_value_is_delta_encoded": 1,
"filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24,
"raw_value_size": 148, "raw_average_value_size": 49,
"num_data_blocks": 1, "num_entries": 3, "num_deletions": 0,
"num_merge_operands": 0, "num_range_deletions": 0, "format_version":
0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "default",
"column_family_id": 0, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "wind
ow_bits=-14; level=32767; strategy=0; max_dict_bytes=0;
zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784,
"oldest_key_time": 0, "file_creation_time": 0, "db_id":
"53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id":
"OS2T69IQ02SU5OKHBI40"}}
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
[db/version_set.cc:4082] Creating manifest 87
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784166273, "job": 1, "event":
"recovery_finished"}
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
[db/column_family.cc:983] [default] Increasing compaction threads
because we have 14 level-0 files
2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb:
[db/db_impl/db_impl_open.cc:1700] SstFileManager instance
0x56017d230700
2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer
0x56017df56000
adding auth for 'client.admin':
auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with
caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *})
2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb:
[db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting
14@0 files to L6, score 3.50
2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb:
[db/compaction/compaction_job.cc:1887] [default] Compaction start
summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B)
74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B)
38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)]
2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784169200, "job": 3, "event":
"compaction_started", "compaction_reason": "LevelL0FilesNum",
"files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8],
"score": 3.5, "input_data_size": 7870219}
2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb:
[db/db_impl/db_impl.cc:901] ------- DUMPING STATS -------
2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb:
[db/db_impl/db_impl.cc:903]
** DB Stats **
Uptime(secs): 0.0 total, 0.0 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per
commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per
commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00
MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB)
Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 1.1 0.00 0.00
1 0.001 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Uptime(secs): 0.0 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction,
0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for
pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0
memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB)
Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.00 0.00
0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 1.1 0.00 0.00
1 0.001 0 0
Uptime(secs): 0.0 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction,
0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for
pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0
memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
[db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated
table #91: 1366 keys, 7566988 bytes
2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784213586, "cf_name": "default", "job": 3,
"event": "table_file_creation", "file_number": 91, "file_size":
7566988, "file_checksum": "", "file_checksum_func_name": "Unknown",
"table_properties": {"data_size": 7541895, "index_size": 20610,
"index_partitions": 0, "top_level_index_size": 0,
"index_key_is_user_key": 1, "index_value_is_delta_encoded": 1,
"filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size":
21, "raw_value_size": 7503048, "raw_average_value_size": 5492,
"num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0,
"num_merge_operands": 0, "num_range_deletions": 0, "format_version":
0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "default",
"column_family_id": 0, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression", "c
ompression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ",
"creation_time": 1700216681, "oldest_key_time": 0,
"file_creation_time": 1700216784, "db_id":
"53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id":
"OS2T69IQ02SU5OKHBI40"}}
2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
[db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted
14@0 files to L6 => 7566988 bytes
2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb:
[db/version_set.cc:3457] More existing levels in DB than needed.
max_bytes_for_level_multiplier may not be guaranteed.
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log
Time 2023/11/17-12:26:24.215298)
[db/compaction/compaction_job.cc:812] [default] compacted to: base
level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0
0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files
in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0)
write-amplify(1.0) OK, records in: 19842, records dropped: 18476
output_compression: NoCompression
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log
Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros":
1700216784215306, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 44437, "compaction_time_cpu_micros":
40923, "output_level": 6, "num_output_files": 1,
"total_output_size": 7566988, "num_input_records": 19842,
"num_output_records": 1366, "num_subcompactions": 1,
"output_compression": "NoCompression",
"num_single_delete_mismatches": 0, "num_single_delete_fallthrough":
0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215520, "job": 3, "event":
"table_file_deletion", "file_number": 86}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215570, "job": 3, "event":
"table_file_deletion", "file_number": 80}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215603, "job": 3, "event":
"table_file_deletion", "file_number": 74}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215641, "job": 3, "event":
"table_file_deletion", "file_number": 68}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215672, "job": 3, "event":
"table_file_deletion", "file_number": 62}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215708, "job": 3, "event":
"table_file_deletion", "file_number": 56}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215739, "job": 3, "event":
"table_file_deletion", "file_number": 50}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215772, "job": 3, "event":
"table_file_deletion", "file_number": 44}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215804, "job": 3, "event":
"table_file_deletion", "file_number": 38}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215831, "job": 3, "event":
"table_file_deletion", "file_number": 32}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215858, "job": 3, "event":
"table_file_deletion", "file_number": 26}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215888, "job": 3, "event":
"table_file_deletion", "file_number": 20}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784215952, "job": 3, "event":
"table_file_deletion", "file_number": 14}
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb:
[file/delete_scheduler.cc:69] Deleted file
/root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec
0, total_trash_size 0 max_trash_db_ratio 0.250000
2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1700216784216804, "job": 3, "event":
"table_file_deletion", "file_number": 8}
update_mkfs generating seed initial monmap
epoch 0
fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37
last_changed 2023-11-17T12:26:24.222814+0200
created 2023-11-17T12:26:24.222814+0200
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01
1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02
2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03
2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb:
[db/db_impl/db_impl.cc:446] Shutdown: canceling all background work
2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb:
[db/db_impl/db_impl.cc:625] Shutdown complete
Then we copied the /root/mon-store/store.db folder across on all 3
nodes and tried to start ceph.target service again.
The output on node01 is below:
d31781fa6b4c quay.io/ceph/ceph
"/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc
e385c32651d2 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10
904f522c4cb5 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0
033edf99a98e quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4
70344a6e87a0 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8
905b782aedcf quay.io/prometheus/prometheus:v2.43.0
"/bin/prometheus --c..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01
ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0
"/bin/node_exporter ..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01
459c46f4bdb7 quay.io/ceph/ceph
"/usr/bin/ceph-mgr -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx
cacfe8abcbbf quay.io/ceph/ceph
"/usr/bin/ceph-crash..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01
e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0
"/bin/alertmanager -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01
d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c
'grafana..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01
The output on node02 is below:
2aec62685dee quay.io/ceph/ceph
"/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp
249b04f32f8c quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5
a2c96f56b517 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2
87496d374a29 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12
55fe47765917 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9
76171e25dbde quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7
220472e8c1bf quay.io/ceph/ceph
"/usr/bin/ceph-mgr -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu
0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0
"/bin/node_exporter ..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02
4e638003fa2e quay.io/ceph/ceph
"/usr/bin/ceph-crash..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02
42719d5cfdbf quay.io/ceph/ceph
"/usr/bin/ceph-mon -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02
The output on node03 is below:
7e5879dce643 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11
d53996ff33b9 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3
e1ac5a8b87d3 quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1
f4cda871218d quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13
969e670dc47c quay.io/ceph/ceph
"/usr/bin/ceph-osd -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6
a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0
"/bin/node_exporter ..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03
835c3893a3f4 quay.io/ceph/ceph
"/usr/bin/ceph-crash..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03
bfa6f5b989ea quay.io/ceph/ceph
"/usr/bin/ceph-mon -..." 2 days ago Up 2 days
ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03
# ceph -s (output below):
cluster:
id: be4304e4-b0d5-11ec-8c6a-2965d4229f37
health: HEALTH_ERR
20 stray daemon(s) not managed by cephadm
3 stray host(s) with 20 daemon(s) not managed by cephadm
1/3 mons down, quorum node02,node03
1/523510 objects unfound (0.000%)
3 nearfull osd(s)
1 osds exist in the crush map but not in the osdmap
Low space hindering backfill (add storage if this
doesn't resolve itself): 20 pgs backfill_toofull
Possible data damage: 1 pg recovery_unfound
Degraded data redundancy: 74666/1570530 objects degraded
(4.754%), 21 pgs degraded, 21 pgs undersized
3 pool(s) nearfull
services:
mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01
mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu
osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs
data:
pools: 3 pools, 161 pgs
objects: 523.51k objects, 299 GiB
usage: 1014 GiB used, 836 GiB / 1.8 TiB avail
pgs: 74666/1570530 objects degraded (4.754%)
1/523510 objects unfound (0.000%)
140 active+clean
20 active+undersized+degraded+remapped+backfill_toofull
1. active+recovery_unfound+undersized+degraded+remapped
# ceph fs ls (output below):
No filesystems enabled
It looks like that we have a problem with the orchestrator now
(we've lost cephadm orchestrator) and we also cannot see the
filesystem.
May you please assist since we are not able to mount the filesystem ?
Thank you,
Manolis Daramas
Under the General Data Protection Regulation (GDPR) (EU) 2016/679,
Motivian as Data Controller has a legal duty to protect any
information collected from you via email. Information contained in
this email and any attachments may be privileged or confidential and
intended for the exclusive use of the original recipient. If you
have received this email by mistake, please advise the sender
immediately and delete the email, including emptying your deleted
email box. Information included in this email is reserved to named
addressee's eyes only. You may not share this message or any of its
attachments to anyone. Please note that as the recipient, it is your
responsibility to check the email for malicious software. Motivian
puts the security of the client at a high priority. Therefore, we
have put efforts into ensuring that the message is error and
virus-free. Unfortunately, full security of the email cannot be
ensured as, despite our efforts, the data included in emails could
be infected,
intercepted, or corrupted. Therefore, the recipient should check
the email for threats with proper software, as the sender does not
accept liability for any damage inflicted by viewing the content of
this email.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
