Good morning,
i checked today the osd.8 and the log shows again the same error
bluefs _allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid, capacity
0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
any idea why that could be?
Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof
<harald.freidhof@xxxxxxxxxxxxxx
<mailto:harald.freidhof@xxxxxxxxxxxxxx>>:
Hi Igor,
today i repaired one osd node and all osd´s on the node,
creating them new again....
after that i waited for the rebalance/recovery process
and the cluster was healthy after some hours..
i notices that the osd.2 does not have any more this
error in the log.
but i noticed it now on the same node on osd.8... so i
did the test that you suggested on osd.8
it took nearly 20minutes to compact those db´s on the
bluestore, but it helped... the problem on osd.8 is gone...
*so the problem that i have with the alloc on the wal
device seems to be random on different nodes and osd´s
and looks like it comes, stay a while and disappears
after a longer while...
*
here are the results that you suggested:
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore
bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 1044480,
"bluefs_used": 48317333504
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 79842762752,
"bluefs_used": 108062048256
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5510727389184,
"bluefs_used": 0,
"bluefs max available": 5508815847424
}
}
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB)
2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976,
db_total:178509578240, slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 304 MiB 7.9 GiB 0 B 0 B
0 B 9.7 MiB 1
WAL 45 GiB 100 GiB 0 B 0 B
0 B 144 GiB 2319
DB 0 B 276 MiB 0 B 0 B
0 B 249 MiB 47
SLOW 0 B 0 B 0 B 0 B
0 B 0 B 0
TOTALS 45 GiB 109 GiB 0 B 0 B
0 B 0 B 2367
MAXIMUMS:
LOG 304 MiB 7.9 GiB 0 B 0 B
0 B 20 MiB
WAL 45 GiB 149 GiB 0 B 0 B
0 B 192 GiB
DB 0 B 762 MiB 0 B 0 B
0 B 738 MiB
SLOW 0 B 0 B 0 B 0 B
0 B 0 B
TOTALS 45 GiB 150 GiB 0 B 0 B
0 B 0 B
---
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x400000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb:
[db_impl/db_impl_write.cc:1668] [L] New memtable created
with log file: #13656. Immutable memtables: 1.
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs
_allocate unable to allocate 0x100000 on bdev 0,
allocator name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb:
(Original Log Time 2021/10/06-19:51:34.996331)
[db_impl/db_impl_compaction_flush.cc:2198] Calling
FlushMemTableToOutputFile with column family [L], flush
slots available 1, compaction slots available 1, flush
slots scheduled 1, compaction slots scheduled 0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb:
[flush_job.cc:321] [L] [JOB 8859] Flushing memtable with
next log file: 13655
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb:
[flush_job.cc:321] [L] [JOB 8859] Flushing memtable with
next log file: 13656
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb:
EVENT_LOG_v1 {"time_micros": 1633549894998273, "job":
8859, "event": "flush_started", "num_memtables": 2,
"num_entries": 3662, "num_deletes": 0,
"total_data_size": 130482337, "memory_usage": 132976224,
"flush_reason": "Write Buffer Full"}
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb:
[flush_job.cc:350] [L] [JOB 8859] Level-0 flush table
#13657: started
Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb:
EVENT_LOG_v1 {"time_micros": 1633549895008271,
"cf_name": "L", "job": 8859, "event":
"table_file_creation", "file_number": 13657,
"file_size": 2952537, "table_properties": {"data_size":
2951222, "index_size": 267, "index_partitions": 0,
"top_level_index_size": 0, "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0, "filter_size": 197,
"raw_key_size": 1120, "raw_average_key_size": 16,
"raw_value_size": 2950151, "raw_average_value_size":
42145, "num_data_blocks": 9, "num_entries": 70,
"num_deletions": 61, "num_merge_operands": 0,
"num_range_deletions": 0, "format_version": 0,
"fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "L",
"column_family_id": 10, "comparator":
"leveldb.BytewiseComparator", "merge_operator":
"nullptr", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression":
"NoCompression", "compression_options":
"window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ",
"creation_time": 1633549854, "oldest_key_time":
1633549854, "file_creation_time": 1633549894}}
Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb:
[flush_job.cc:401] [L] [JOB 8859] Level-0 flush table
#13657: 2952537 bytes OK
---
root@cd88-ceph-osdh-01:~# ceph osd set noout
root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8
root@cd88-ceph-osdh-01:~# ceph orch ps
...
osd.7 cd133-ceph-osdh-01 running (4h) 44s ago -
2738M 5528M 16.2.5 6933c2a0b7dd 8a98ae61f0eb
osd.8 cd88-ceph-osdh-01 stopped 5s ago -
- 5528M <unknown> <unknown> <unknown>
osd.9 cd133k-ceph-osdh-01 running (3d) 5m ago -
4673M 8966M 16.2.5 6933c2a0b7dd 0ff7584b1808
...
---
root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/
compact
2021-10-06T21:53:50.559+0200 7f87bde3c240 0
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db_and_around read-only:0 repair:0
2021-10-06T21:53:50.559+0200 7f87bde3c240 1
bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000, 5.5 TiB)
block_size 4096 (4 KiB) rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45
data 0.06
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open size 187904819200 (0x2bc0000000, 175 GiB)
block_size 4096 (4 KiB) rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
add_block_device bdev 1 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000, 5.5 TiB)
block_size 4096 (4 KiB) rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
add_block_device bdev 2 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
size 5.5 TiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open size 48318382080 (0xb40000000, 45 GiB) block_size
4096 (4 KiB) non-rotational discard supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
add_block_device bdev 0 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
_init_alloc new, id 0, allocator name bluefs-wal,
allocator type hybrid, capacity 0xb40000000, block size
0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
_init_alloc new, id 1, allocator name bluefs-db,
allocator type hybrid, capacity 0x2bc0000000, block size
0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs
_init_alloc shared, id 2, capacity 0x57541c00000, block
size 0x10000
2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs
mount shared_bdev_used = 0
2021-10-06T21:53:50.655+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to db,178509578240
db.slow,5701113793740
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old nid_max 167450
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old blobid_max 30720
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta freelist_type bitmap
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta ondisk_format 4 compat_ondisk_format 3
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta min_alloc_size 0x1000
2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init
2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist
_read_cfg
2021-10-06T22:01:33.347+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc opening allocation metadata
2021-10-06T22:01:41.031+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc loaded 5.0 TiB in 37191 extents, allocator
type hybrid, capacity 0x57541c00000, block size 0x1000,
free 0x502f8f9a000, fragmentation 2.76445e-05
2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount
2021-10-06T22:01:41.043+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
close
2021-10-06T22:01:43.623+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
close
2021-10-06T22:01:54.727+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open size 187904819200 (0x2bc0000000, 175 GiB)
block_size 4096 (4 KiB) rotational discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
add_block_device bdev 1 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000, 5.5 TiB)
block_size 4096 (4 KiB) rotational discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
add_block_device bdev 2 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
size 5.5 TiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T22:01:54.995+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open size 48318382080 (0xb40000000, 45 GiB) block_size
4096 (4 KiB) non-rotational discard supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
add_block_device bdev 0 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
_init_alloc new, id 0, allocator name bluefs-wal,
allocator type hybrid, capacity 0xb40000000, block size
0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
_init_alloc new, id 1, allocator name bluefs-db,
allocator type hybrid, capacity 0x2bc0000000, block size
0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs
_init_alloc shared, id 2, capacity 0x57541c00000, block
size 0x10000
2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs
mount shared_bdev_used = 0
2021-10-06T22:01:55.079+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to db,178509578240
db.slow,5701113793740
2021-10-06T22:09:36.519+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:09:54.067+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
umount
2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount
2021-10-06T22:09:54.079+0200 7f87bde3c240 1
bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
close
2021-10-06T22:09:56.612+0200 7f87bde3c240 1
bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
close
2021-10-06T22:10:07.520+0200 7f87bde3c240 1
bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist
shutdown
2021-10-06T22:10:07.692+0200 7f87bde3c240 1
bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
---
root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8
---
root@cd88-ceph-osdh-01:/# ceph -s
cluster:
id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
health: HEALTH_OK
services:
mon: 3 daemons, quorum
cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01
(age 15h)
mgr: cd133-ceph-mon-01.mzapob(active, since 15h),
standbys: cd133k-ceph-mon-01.imikwh
osd: 60 osds: 60 up (since 2m), 60 in (since 3h)
rgw: 4 daemons active (2 hosts, 1 zones)
tcmu-runner: 10 portals active (2 hosts)
data:
pools: 6 pools, 361 pgs
objects: 2.46M objects, 8.0 TiB
usage: 33 TiB used, 304 TiB / 338 TiB avail
pgs: 361 active+clean
io:
client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674
op/s wr
---
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore
bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
*"free": 41354784768,
* "bluefs_used": 6963593216
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 187302928384,
"bluefs_used": 601882624
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5507531620352,
"bluefs_used": 0,
"bluefs max available": 5505566572544
}
}
---
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB)
1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB)
2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976,
db_total:178509578240, slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 12 MiB 18 MiB 0 B 0 B
0 B 10 MiB 0
WAL 6.5 GiB 0 B 0 B 0 B 0 B
6.4 GiB 102
DB 0 B 573 MiB 0 B 0 B
0 B 557 MiB 22
SLOW 0 B 0 B 0 B 0 B
0 B 0 B 0
TOTALS 6.5 GiB 591 MiB 0 B 0 B
0 B 0 B 125
MAXIMUMS:
LOG 12 MiB 18 MiB 0 B 0 B
0 B 17 MiB
WAL 45 GiB 101 GiB 0 B 0 B
0 B 145 GiB
DB 0 B 688 MiB 0 B 0 B
0 B 670 MiB
SLOW 0 B 0 B 0 B 0 B
0 B 0 B
TOTALS 45 GiB 101 GiB 0 B 0 B
0 B 0 B
----
Here are the osd.2... the problem disapeared from alone
very strange...
root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB)
1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB)
2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976,
db_total:178509578240, slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 920 MiB 4.0 GiB 0 B 0 B
0 B 10 MiB 1
WAL 31 GiB 17 GiB 0 B 0 B
0 B 48 GiB 765
DB 0 B 193 MiB 0 B 0 B
0 B 175 MiB 30
SLOW 0 B 0 B 0 B 0 B
0 B 0 B 0
TOTALS 32 GiB 21 GiB 0 B 0 B
0 B 0 B 796
MAXIMUMS:
LOG 920 MiB 4.0 GiB 0 B 0 B
0 B 17 MiB
WAL 45 GiB 149 GiB 0 B 0 B
0 B 192 GiB
DB 0 B 762 MiB 0 B 0 B
0 B 741 MiB
SLOW 0 B 0 B 0 B 0 B
0 B 0 B
TOTALS 45 GiB 153 GiB 0 B 0 B
0 B 0 B
root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore
bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 15043915776,
"bluefs_used": 33274462208
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 169235963904,
"bluefs_used": 18668847104
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5602453327872,
"bluefs_used": 0,
"bluefs max available": 5600865222656
}
}
Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov
<igor.fedotov@xxxxxxxx <mailto:igor.fedotov@xxxxxxxx>>:
On 10/6/2021 4:25 PM, José H. Freidhof wrote:
> hi,
>
> no risk no fun 😂 okay
> I have reset the settings you mentioned to standard.
>
> what you exactly mean with taking offline the osd?
ceph orch daemon stop
> osd.2? or mark down?
"daemon stop" is enough. You might want to set
noout flag before that
though...
>
> for the command which path i use? you mean:
>
> bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
yep
>
>
> Igor Fedotov <ifedotov@xxxxxxx
<mailto:ifedotov@xxxxxxx>> schrieb am Mi., 6. Okt.
2021, 13:33:
>
>> On 10/6/2021 2:16 PM, José H. Freidhof wrote:
>>> Hi Igor,
>>>
>>> yes i have some osd settings set :-) here are my
ceph config dump. those
>>> settings are from a redhat document for
bluestore devices
>>> maybe it is that setting causing this problem?
"advanced
>>> mon_compact_on_trim false"???
>> OMG!!!
>>
>> No - mon_compact_on_trim has nothing to deal with
bluestore.
>>
>> Highly likely it's bluestore_rocksdb_options
which hurts...
>> Documentations tend to fall behind the best
practices.... I would
>> strongly discourage you from using non-default
settings unless it's
>> absolutely clear why this is necessary.
>>
>> Even at the first glance the following settings
(just a few ones I'm
>> completely aware) are suboptimal/non-recommended:
>>
>> rocksdb_perf
>>
>> bluefs_sync_write
>>
>> bluefs_csum_type
>>
>>
>> Not to mention bluestore_rocksdb_options which
hasn't got much adoption
>> so far and apparently greatly alters rocksdb
behavior...
>>
>>
>> So I would suggest to revert rocksdb options back
to default, run the
>> compaction and if it succeeds monitor the OSD for
a while. Then if it
>> works fine - apply the same for others
>>
>>
>> Hope this helps,
>>
>> Igor
>>
>>
>>
>>> i will test it this afternoon... at the moment
are everything semi
>>> prodcuctive and i need to repair one osd node..
because i think of this
>>> reason the osds crashed on the node and the osd
container crashes with a
>>> dump while coming up now.
>>> need first to replicate all between all three
nodes and then i can take
>>> offline the osd.2.and test your command. i will
inform you later...
>>>
>>> root@cd88-ceph-osdh-01:/# ceph config dump
>>> WHO MASK
>>> LEVEL OPTION
VALUE
>>>
>>>
>>>
>>>
>>>
>>> RO
>>> global
>>> advanced leveldb_max_open_files 131072
>>>
>>>
>>>
>>>
>>> global
>>> advanced mon_compact_on_trim false
>>>
>>>
>>>
>>>
>>> global
>>> dev ms_crc_data
false
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_deep_scrub_interval
1209600.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_max_scrubs 16
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_scrub_load_threshold 0.010000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_scrub_max_interval
1209600.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_scrub_min_interval
86400.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced perf
true
>>>
>>>
>>>
>>>
>>> global
>>> advanced rbd_readahead_disable_after_bytes
0
>>>
>>>
>>>
>>>
>>> global
>>> advanced rbd_readahead_max_bytes 4194304
>>>
>>>
>>>
>>>
>>> global
>>> advanced rocksdb_perf true
>>>
>>>
>>>
>>>
>>> global
>>> advanced throttler_perf_counter false
>>>
>>>
>>>
>>>
>>> mon
>>> advanced auth_allow_insecure_global_id_reclaim
false
>>>
>>>
>>>
>>>
>>> mon
>>> advanced cluster_network 10.50.50.0/24
<http://10.50.50.0/24>
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mon
>>> advanced mon_osd_down_out_interval 300
>>>
>>>
>>>
>>>
>>> mon
>>> advanced public_network 10.50.50.0/24
<http://10.50.50.0/24>
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced mgr/cephadm/container_init True
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced mgr/cephadm/device_enhanced_scan
true
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced mgr/cephadm/migration_current 2
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced mgr/cephadm/warn_on_stray_daemons
false
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced mgr/cephadm/warn_on_stray_hosts false
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> advanced bluefs_sync_write true
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_autotune true
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_kv_ratio 0.200000
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_meta_ratio 0.800000
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_size 2147483648
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_size_hdd 2147483648
>>>
>>>
>>>
>>>
>>> osd
>>> advanced bluestore_csum_type none
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_extent_map_shard_max_size 200
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_extent_map_shard_min_size 50
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_extent_map_shard_target_size 100
>>>
>>>
>>>
>>>
>>> osd
>>> advanced bluestore_rocksdb_options
>>>
>>
compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>>> *
>>> osd
>>> advanced mon_osd_cache_size 1024
>>>
>>>
>>>
>>>
>>> osd
>>> dev ms_crc_data
false
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_map_share_max_epochs 5
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_max_backfills 1
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_max_pg_log_entries 10
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_memory_cache_min 3000000000
>>>
>>>
>>>
>>>
>>> osd
>> host:cd133-ceph-osdh-01
>>> basic osd_memory_target 5797322383
>>>
>>>
>>>
>>>
>>> osd
>> host:cd133k-ceph-osdh-01
>>> basic osd_memory_target 9402402385
>>>
>>>
>>>
>>>
>>> osd host:cd88-ceph-osdh-01
>>> basic osd_memory_target 5797322096
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_memory_target_autotune true
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_min_pg_log_entries 10
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_op_num_shards 8
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> advanced osd_op_num_threads_per_shard 2
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> dev osd_pg_log_dups_tracked 10
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_pg_log_trim_min 10
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_recovery_max_active 3
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_recovery_max_single_start 1
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_recovery_sleep 0.000000
>>>
>>>
>>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor
Fedotov <ifedotov@xxxxxxx <mailto:ifedotov@xxxxxxx>
>>> :
>>>
>>>> Jose,
>>>>
>>>> In fact 48GB is a way too much for WAL drive -
usually the write ahead
>> log
>>>> tend to be 2-4 GBs.
>>>>
>>>> But in your case it's ~150GB, while DB itself
is very small (146MB!!!):
>>>>
>>>> WAL 45 GiB 111 GiB 0 B 0
B 0 B
>>>> 154 GiB 2400
>>>>
>>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>>> 146 MiB 30
>>>>
>>>>
>>>> which means that there are some issues with
RocksDB's WAL processing,
>>>> which needs some troubleshooting...
>>>>
>>>> Curious if other OSDs are suffering from the
same and whether you have
>> any
>>>> custom settings for your OSD(s)?
>>>>
>>>> Additionally you might want to try the
following command to compact this
>>>> specific OSD manually and check if this would
normalize the DB layout -
>> the
>>>> majority of data has to be at DB level not WAL.
Please share the
>> resulting
>>>> layout (reported by "ceph daemon osd.2 bluefs
stats" command) after the
>>>> compaction is fulfiled and OSD is restarted.
>>>>
>>>> The compaction command to be applied on an
offline OSD:
>> "ceph-kvstore-tool
>>>> bluestore-kv <path-to-osd> compact"
>>>>
>>>> Even if the above works great please refrain
from applying that
>> compaction
>>>> to every OSD - let's see how that "compacted"
OSD evolves.Would WAL grow
>>>> again or not?
>>>>
>>>> Thanks,
>>>>
>>>> Igor
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote:
>>>>
>>>> Hello Igor,
>>>>
>>>> yes the volume is nvme wal partitions for the
bluestore devicegroups
>> are
>>>> only 48gb each
>>>>
>>>> on each osd node are 1 nvme with 1tb splitted
in 20 lvs with 48gb (WAL)
>>>> on each osd node are 4 ssd with 1tb splitted in
5 lvs with 175gb
>> (rock.db)
>>>> on each osd node are 20 hdd with 5.5tb with 1
lvs (block.db)
>>>>
>>>> each blustore have 1 partition nvme,ssd and hdd
like described in the
>>>> documentation
>>>>
>>
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
<https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/>
>>>> is this to small or can i adjust the max
allocation on the wal nvme
>> device
>>>> in the ceph configuration?
>>>> i know that the ssd and nvme are to small for
those 5.5tb disk... its 1%
>>>> only ot the rotation disk.
>>>> i am new in ceph and still or always learning,
but we are in a little
>>>> hurry because our other datastores are old and
full.
>>>>
>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2
bluestore bluefs device
>> info
>>>> {
>>>> "dev": {
>>>> "device": "BDEV_WAL",
>>>> "total": 48318377984,
>>>> "free": 1044480,
>>>> "bluefs_used": 48317333504
>>>> },
>>>> "dev": {
>>>> "device": "BDEV_DB",
>>>> "total": 187904811008,
>>>> "free": 68757217280,
>>>> "bluefs_used": 119147593728
>>>> },
>>>> "dev": {
>>>> "device": "BDEV_SLOW",
>>>> "total": 6001172414464,
>>>> "free": 5624912359424,
>>>> "bluefs_used": 0,
>>>> "bluefs max available": 5624401231872
>>>> }
>>>> }
>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2
bluefs stats
>>>> 0 : device size 0xb3ffff000 : using
0xb3ff00000(45 GiB)
>>>> 1 : device size 0x2bbfffe000 : using
0x1bbeb00000(111 GiB)
>>>> 2 : device size 0x57541c00000 : using
0x579b592000(350 GiB)
>>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>> db_total:178509578240,
>>>> slow_total:5701113793740, db_avail:103884521472
>>>> Usage matrix:
>>>> DEV/LEV WAL DB SLOW * *
>>>> REAL FILES
>>>> LOG 124 MiB 2.3 GiB 0 B
0 B 0 B
>>>> 7.5 MiB 1
>>>> WAL 45 GiB 111 GiB 0 B 0
B 0 B
>>>> 154 GiB 2400
>>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>>> 146 MiB 30
>>>> SLOW 0 B 0 B 0 B 0 B 0 B
>> 0
>>>> B 0
>>>> TOTALS 45 GiB 113 GiB 0 B 0
B 0 B
>> 0
>>>> B 2431
>>>> MAXIMUMS:
>>>> LOG 124 MiB 2.3 GiB 0 B
0 B 0 B
>> 17
>>>> MiB
>>>> WAL 45 GiB 149 GiB 0 B 0
B 0 B
>>>> 192 GiB
>>>> DB 0 B 762 MiB 0 B 0 B 0 B
>>>> 741 MiB
>>>> SLOW 0 B 0 B 0 B 0 B 0 B
>> 0 B
>>>> TOTALS 45 GiB 150 GiB 0 B 0
B 0 B
>> 0 B
>>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor
Fedotov <
>> ifedotov@xxxxxxx <mailto:ifedotov@xxxxxxx>>:
>>>>> Hey Jose,
>>>>>
>>>>> it looks like your WAL volume is out of space
which looks weird given
>>>>> its capacity = 48Gb.
>>>>>
>>>>> Could you please share the output of the
following commands:
>>>>>
>>>>> ceph daemon osd.N bluestore bluefs device info
>>>>>
>>>>> ceph daemon osd.N bluefs stats
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Igor
>>>>>
>>>>>
>>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote:
>>>>>> Hello together
>>>>>>
>>>>>> we have a running ceph pacific 16.2.5 cluster
and i found this
>> messages
>>>>> in
>>>>>> the service logs of the osd daemons.
>>>>>>
>>>>>> we have three osd nodes .. each node has
20osds as bluestore with
>>>>>> nvme/ssd/hdd
>>>>>>
>>>>>> is this a bug or maybe i have some settings
wrong?
>>>>>>
>>>>>>
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:25.821+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:29.857+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.073+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x400000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.405+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.465+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.529+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f38eebd4700 4 rocksdb:
[db_impl/db_impl_write.cc:1668] [L] New
>>>>> memtable
>>>>>> created with log file: #9588. Immutable
memtables: 1.
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f38eebd4700 1 bluefs _allocate unable to
allocate 0x100000 on bdev
>> 0,
>>>>>> allocator name bluefs-wal, allocator type
hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.547575)
>>>>>> [db_impl/db_impl_compaction_flush.cc:2198]
Calling
>>>>>> FlushMemTableToOutputFile with column family
[L], flush slots
>> available
>>>>> 1,
>>>>>> compaction slots available 1, flush slots
scheduled 1, compaction
>> slots
>>>>>> scheduled 0
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321]
[L] [JOB 5709] Flushing
>>>>>> memtable with next log file: 9587
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321]
[L] [JOB 5709] Flushing
>>>>>> memtable with next log file: 9588
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850547916,
>>>>>> "job": 5709, "event": "flush_started",
"num_memtables": 2,
>>>>> "num_entries":
>>>>>> 4146, "num_deletes": 0, "total_data_size":
127203926, "memory_usage":
>>>>>> 130479920, "flush_reason": "Write Buffer Full"}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350]
[L] [JOB 5709] Level-0
>> flush
>>>>>> table #9589: started
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850559292,
>>>>>> "cf_name": "L", "job": 5709, "event":
"table_file_creation",
>>>>> "file_number":
>>>>>> 9589, "file_size": 3249934,
"table_properties": {"data_size": 3247855,
>>>>>> "index_size": 1031, "index_partitions": 0,
"top_level_index_size": 0,
>>>>>> "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0,
>>>>>> "filter_size": 197, "raw_key_size": 1088,
"raw_average_key_size": 16,
>>>>>> "raw_value_size": 3246252,
"raw_average_value_size": 47739,
>>>>>> "num_data_blocks": 36, "num_entries": 68,
"num_deletions": 32,
>>>>>> "num_merge_operands": 0,
"num_range_deletions": 0, "format_version":
>> 0,
>>>>>> "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter",
>>>>>> "column_family_name": "L",
"column_family_id": 10, "comparator":
>>>>>> "leveldb.BytewiseComparator",
"merge_operator": "nullptr",
>>>>>> "prefix_extractor_name": "nullptr",
"property_collectors": "[]",
>>>>>> "compression": "NoCompression",
"compression_options":
>> "window_bits=-14;
>>>>>> level=32767; strategy=0; max_dict_bytes=0;
zstd_max_train_bytes=0;
>>>>>> enabled=0; ", "creation_time": 1633511730,
"oldest_key_time":
>>>>> 1633511730,
>>>>>> "file_creation_time": 1633511850}}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401]
[L] [JOB 5709] Level-0
>> flush
>>>>>> table #9589: 3249934 bytes OK
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.559362)
>>>>>> [memtable_list.cc:447] [L] Level-0 commit
table #9589 started
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.559583)
>>>>>> [memtable_list.cc:503] [L] Level-0 commit
table #9589: memtable #1
>> done
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.559586)
>>>>>> [memtable_list.cc:503] [L] Level-0 commit
table #9589: memtable #2
>> done
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.559601)
>>>>>> EVENT_LOG_v1 {"time_micros":
1633511850559593, "job": 5709, "event":
>>>>>> "flush_finished", "output_compression":
"NoCompression", "lsm_state":
>>>>> [8,
>>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.559638)
>>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L]
Level summary: files[8 1
>>>>> 0 0
>>>>>> 0 0 0] max score 1.00
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1676] [L] [JOB
>>>>> 5710]
>>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1680] [L]
>>>>> Compaction
>>>>>> start summary: Base version 3090 Base level
0, inputs: [9589(3173KB)
>>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB)
9576(6417KB) 9573(1078KB)
>>>>> 9570(405KB)
>>>>>> 9567(29KB)], [9564(1115KB)]
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850559956,
>>>>>> "job": 5710, "event": "compaction_started",
"compaction_reason":
>>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586,
9583, 9580, 9576, 9573,
>>>>> 9570,
>>>>>> 9567], "files_L1": [9564], "score": 1,
"input_data_size": 19542092}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1349] [L] [JOB
>>>>> 5710]
>>>>>> Generated table #9590: 36 keys, 3249524 bytes
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850582987,
>>>>>> "cf_name": "L", "job": 5710, "event":
"table_file_creation",
>>>>> "file_number":
>>>>>> 9590, "file_size": 3249524,
"table_properties": {"data_size": 3247449,
>>>>>> "index_size": 1031, "index_partitions": 0,
"top_level_index_size": 0,
>>>>>> "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0,
>>>>>> "filter_size": 197, "raw_key_size": 576,
"raw_average_key_size": 16,
>>>>>> "raw_value_size": 3246252,
"raw_average_value_size": 90173,
>>>>>> "num_data_blocks": 36, "num_entries": 36,
"num_deletions": 0,
>>>>>> "num_merge_operands": 0,
"num_range_deletions": 0, "format_version":
>> 0,
>>>>>> "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter",
>>>>>> "column_family_name": "L",
"column_family_id": 10, "comparator":
>>>>>> "leveldb.BytewiseComparator",
"merge_operator": "nullptr",
>>>>>> "prefix_extractor_name": "nullptr",
"property_collectors": "[]",
>>>>>> "compression": "NoCompression",
"compression_options":
>> "window_bits=-14;
>>>>>> level=32767; strategy=0; max_dict_bytes=0;
zstd_max_train_bytes=0;
>>>>>> enabled=0; ", "creation_time": 1633471854,
"oldest_key_time": 0,
>>>>>> "file_creation_time": 1633511850}}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1415] [L] [JOB
>>>>> 5710]
>>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.583469)
>>>>>> [compaction/compaction_job.cc:760] [L]
compacted to: files[0 1 0 0 0 0
>>>>> 0]
>>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr,
level 1, files in(8, 1)
>>>>> out(1)
>>>>>> MB in(17.5, 1.1) out(3.1),
read-write-amplify(1.2) write-amplify(0.2)
>>>>> OK,
>>>>>> records in: 376, records dropped: 340
output_compression:
>> NoCompression
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>> 2021/10/06-09:17:30.583498)
>>>>>> EVENT_LOG_v1 {"time_micros":
1633511850583485, "job": 5710, "event":
>>>>>> "compaction_finished",
"compaction_time_micros": 23098,
>>>>>> "compaction_time_cpu_micros": 20039,
"output_level": 1,
>>>>> "num_output_files":
>>>>>> 1, "total_output_size": 3249524,
"num_input_records": 376,
>>>>>> "num_output_records": 36,
"num_subcompactions": 1,
>> "output_compression":
>>>>>> "NoCompression",
"num_single_delete_mismatches": 0,
>>>>>> "num_single_delete_fallthrough": 0,
"lsm_state": [0, 1, 0, 0, 0, 0,
>> 0]}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583615,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9589}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583648,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9586}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583675,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9583}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583709,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9580}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583739,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9576}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583769,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9573}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583804,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9570}
>>>>>> cd88-ceph-osdh-01 bash[6283]: debug
2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1
{"time_micros":
>> 1633511850583835,
>>>>>> "job": 5710, "event": "table_file_deletion",
"file_number": 9567}
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
>>>>>> To unsubscribe send an email to
ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
>>>> --
