Hi, I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below: bash-4.4$ ceph -s cluster: id: 3f271841-6188-47c1-b3fd-90fd4f978c76 health: HEALTH_WARN 7 daemons have recently crashed 4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops. services: mon: 3 daemons, quorum a,b,d (age 9h) mgr: b(active, since 14h), standbys: a osd: 4 osds: 0 up, 4 in (since 9h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: All osds are down. I checked the osds logs following this. Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Until now I haven't restored that in the course. Thanks, Ben osd0 log: debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 set uid:gid to 167:167 (ceph:ceph) debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process ceph-osd, pid 1 debug 2023-03-04T04:41:50.620+0000 7f824d7043c0 0 pidfile_write: ignore empty --pid-file debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:50.622+0000 7f824d7043c0 1 bdev(0x5570b19ccc00 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:50.934+0000 7f824d7043c0 1 bdev(0x5570b19cd400 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:51.481+0000 7f824d7043c0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal debug 2023-03-04T04:41:51.482+0000 7f824d7043c0 -1 Falling back to public interface debug 2023-03-04T04:41:51.492+0000 7f824d7043c0 0 load: jerasure load: lrc debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:51.493+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.024+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.025+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.051+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 0 osd.0:0.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.052+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 0 osd.0:1.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.089+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 0 osd.0:2.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.323+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.324+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 0 osd.0:3.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.733+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.734+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 0 osd.0:4.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:52.871+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:52.872+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 0 osd.0:5.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:53.393+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:53.394+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 0 osd.0:6.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.061+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.062+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) close debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_max_osd_capacity #op shards: 8 max osd capacity(iops) per shard: 464.18 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0500000 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000110 debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops debug 2023-03-04T04:41:54.503+0000 7f824d7043c0 0 osd.0:7.OSDShard using op scheduler mClockScheduler debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.504+0000 7f824d7043c0 1 bdev(0x5570b27f2000 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bdev(0x5570b27f3800 /var/lib/ceph/osd/ceph-0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 100 GiB debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs mount debug 2023-03-04T04:41:54.505+0000 7f824d7043c0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluefs mount shared_bdev_used = 0 debug 2023-03-04T04:41:54.508+0000 7f824d7043c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: RocksDB version: 6.15.5 debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2023-03-04T04:41:54.510+0000 7f824d7043c0 4 rocksdb: Compile date Oct 17 2022 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB SUMMARY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: DB Session ID: TK70G5I4T4HSLQWUMAF4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: CURRENT file: CURRENT debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: IDENTITY file: IDENTITY debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: MANIFEST file: MANIFEST-000407 size: 5232 Bytes debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db dir, Total Num: 25, files: 000067.sst 000085.sst 000098.sst 000298.sst 000304.sst 000348.sst 000355.sst 000356.sst 000366.sst debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Write Ahead Log file in db.wal: 000408.log size: 5147 ; debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.error_if_exists: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_if_missing: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.env: 0x5570b2806450 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.fs: Legacy File System debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.info_log: 0x5570b285af80 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_file_opening_threads: 16 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.statistics: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_fsync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_log_file_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.keep_log_file_num: 1000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.recycle_log_file_num: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_fallocate: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_mmap_writes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_reads: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.create_missing_column_families: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_log_dir: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_dir: db.wal debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.advise_random_on_open: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_write_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_manager: 0x5570b28064e0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limiter: (nil) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_recovery_mode: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_thread_tracking: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_pipelined_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.unordered_write: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.row_cache: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_ingest_behind: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.preserve_deletes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.two_write_queues: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.manual_wal_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.atomic_flush: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.log_readahead_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.file_checksum_gen_factory: Unknown debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.best_efforts_recovery: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.allow_data_in_errors: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.db_host_id: __hostname__ debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_jobs: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_compactions: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_subcompactions: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_open_files: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_background_flushes: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Compression algorithms supported: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZSTD supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kXpressCompression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kZlibCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kSnappyCompression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kLZ4Compression supported: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: kBZip2Compression supported: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: db/MANIFEST-000407 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 0, name: default) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_filter_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.sst_partitioner_factory: None debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_factory: BlockBasedTable debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570b1916ef0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x5570b1988f30 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.write_buffer_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression: Disabled debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.num_levels: 7 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.level: 32767 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.strategy: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.parallel_threads: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compression_opts.enabled: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_base: 67108864 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.arena_block_size: 33554432 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.disable_auto_compactions: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.table_properties_collectors: debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_support: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.bloom_locality: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.max_successive_merges: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.paranoid_file_checks: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.force_consistency_checks: 1 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.report_bg_io_stats: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.ttl: 2592000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_files: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.min_blob_size: 0 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_file_size: 268435456 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_compression_type: NoCompression debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.enable_blob_garbage_collection: false debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 1, name: m-0) debug 2023-03-04T04:41:54.511+0000 7f824d7043c0 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [m-0]: ... ... ... ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.23 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 26.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 29.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-1] ** ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 4.97 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L1 1/0 22.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 4/0 27.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-2] ** ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 1.89 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 L1 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-0] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-0] ** ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.0 0.00 0.00 1 0.000 0 0 L1 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 30.5 69.6 67.4 0.13 0.05 1 0.130 20K 2511 Sum 1/0 8.74 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4155.5 69.4 67.3 0.13 0.05 2 0.065 20K 2511 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-1] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.6 67.4 0.13 0.05 1 0.130 20K 2511 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.01 GB write, 0.00 MB/s write, 0.01 GB read, 0.00 MB/s read, 0.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-1] ** ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 2.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 L1 1/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-2] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.6 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [O-2] ** ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 11.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [L] ** ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 L1 1/0 19.96 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.9 113.3 54.8 0.36 0.15 1 0.365 450K 365K Sum 1/0 19.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17800.7 113.2 54.7 0.36 0.15 2 0.182 450K 365K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.3 54.8 0.36 0.15 1 0.365 450K 365K User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 Uptime(secs): 35400.1 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.02 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.4 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [P] ** debug 2023-03-04T14:32:10.864+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:32:40.999+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:11.317+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:33:42.245+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:13.196+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:34:44.048+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:14.100+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:35:44.757+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:15.544+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map debug 2023-03-04T14:36:46.148+0000 7f8242cf0700 1 osd.0 3466 tick checking mon for new map _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx