Hi Igor,
today i repaired one osd node and all osd´s on the node, creating them new
again....
after that i waited for the rebalance/recovery process and the cluster was
healthy after some hours..
i notices that the osd.2 does not have any more this error in the log.
but i noticed it now on the same node on osd.8... so i did the test that
you suggested on osd.8
it took nearly 20minutes to compact those db´s on the bluestore, but it
helped... the problem on osd.8 is gone...
*so the problem that i have with the alloc on the wal device seems to be
random on different nodes and osd´s and looks like it comes, stay a while
and disappears after a longer while...*
here are the results that you suggested:
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 1044480,
"bluefs_used": 48317333504
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 79842762752,
"bluefs_used": 108062048256
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5510727389184,
"bluefs_used": 0,
"bluefs max available": 5508815847424
}
}
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB)
2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
slow_total:5701113793740, db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 304 MiB 7.9 GiB 0 B 0 B 0 B 9.7
MiB 1
WAL 45 GiB 100 GiB 0 B 0 B 0 B 144
GiB 2319
DB 0 B 276 MiB 0 B 0 B 0 B 249
MiB 47
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
0
TOTALS 45 GiB 109 GiB 0 B 0 B 0 B 0 B
2367
MAXIMUMS:
LOG 304 MiB 7.9 GiB 0 B 0 B 0 B 20
MiB
WAL 45 GiB 149 GiB 0 B 0 B 0 B 192
GiB
DB 0 B 762 MiB 0 B 0 B 0 B 738
MiB
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
TOTALS 45 GiB 150 GiB 0 B 0 B 0 B 0 B
---
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb:
[db_impl/db_impl_write.cc:1668] [L] New memtable created with log file:
#13656. Immutable memtables: 1.
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time
2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198]
Calling FlushMemTableToOutputFile with column family [L], flush slots
available 1, compaction slots available 1, flush slots scheduled 1,
compaction slots scheduled 0
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
[L] [JOB 8859] Flushing memtable with next log file: 13655
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
[L] [JOB 8859] Flushing memtable with next log file: 13656
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1633549894998273, "job": 8859, "event": "flush_started",
"num_memtables": 2, "num_entries": 3662, "num_deletes": 0,
"total_data_size": 130482337, "memory_usage": 132976224, "flush_reason":
"Write Buffer Full"}
Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350]
[L] [JOB 8859] Level-0 flush table #13657: started
Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event":
"table_file_creation", "file_number": 13657, "file_size": 2952537,
"table_properties": {"data_size": 2951222, "index_size": 267,
"index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size":
1120, "raw_average_key_size": 16, "raw_value_size": 2950151,
"raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70,
"num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0,
"format_version": 0, "fixed_key_len": 0, "filter_policy":
"rocksdb.BuiltinBloomFilter", "column_family_name": "L",
"column_family_id": 10, "comparator": "leveldb.BytewiseComparator",
"merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
"property_collectors": "[]", "compression": "NoCompression",
"compression_options": "window_bits=-14; level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
1633549854, "oldest_key_time": 1633549854, "file_creation_time":
1633549894}}
Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401]
[L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK
---
root@cd88-ceph-osdh-01:~# ceph osd set noout
root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8
root@cd88-ceph-osdh-01:~# ceph orch ps
...
osd.7 cd133-ceph-osdh-01
running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd
8a98ae61f0eb
osd.8 cd88-ceph-osdh-01
stopped 5s ago - - 5528M <unknown> <unknown>
<unknown>
osd.9 cd133k-ceph-osdh-01
running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd
0ff7584b1808
...
---
root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact
2021-10-06T21:53:50.559+0200 7f87bde3c240 0
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db_and_around read-only:0 repair:0
2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev 1
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev 2
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
5.5 TiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
non-rotational discard supported
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev 0
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id 0,
allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
block size 0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id 1,
allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
block size 0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc shared, id
2, capacity 0x57541c00000, block size 0x10000
2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount shared_bdev_used
= 0
2021-10-06T21:53:50.655+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to db,178509578240
db.slow,5701113793740
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old nid_max 167450
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old blobid_max 30720
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta freelist_type bitmap
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta ondisk_format 4 compat_ondisk_format 3
2021-10-06T22:01:32.715+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta min_alloc_size 0x1000
2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init
2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg
2021-10-06T22:01:33.347+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc opening allocation metadata
2021-10-06T22:01:41.031+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid,
capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000,
fragmentation 2.76445e-05
2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount
2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
rotational discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev 1
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev 2
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
5.5 TiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
non-rotational discard supported
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev 0
path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id 0,
allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
block size 0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id 1,
allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
block size 0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc shared, id
2, capacity 0x57541c00000, block size 0x10000
2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount shared_bdev_used
= 0
2021-10-06T22:01:55.079+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to db,178509578240
db.slow,5701113793740
2021-10-06T22:09:36.519+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:09:54.067+0200 7f87bde3c240 1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount
2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount
2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown
2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
---
root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8
---
root@cd88-ceph-osdh-01:/# ceph -s
cluster:
id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
health: HEALTH_OK
services:
mon: 3 daemons, quorum
cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h)
mgr: cd133-ceph-mon-01.mzapob(active, since 15h), standbys:
cd133k-ceph-mon-01.imikwh
osd: 60 osds: 60 up (since 2m), 60 in (since 3h)
rgw: 4 daemons active (2 hosts, 1 zones)
tcmu-runner: 10 portals active (2 hosts)
data:
pools: 6 pools, 361 pgs
objects: 2.46M objects, 8.0 TiB
usage: 33 TiB used, 304 TiB / 338 TiB avail
pgs: 361 active+clean
io:
client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr
---
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
* "free": 41354784768,* "bluefs_used": 6963593216
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 187302928384,
"bluefs_used": 601882624
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5507531620352,
"bluefs_used": 0,
"bluefs max available": 5505566572544
}
}
---
root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB)
1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB)
2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
slow_total:5701113793740, db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 12 MiB 18 MiB 0 B 0 B 0 B 10
MiB 0
WAL 6.5 GiB 0 B 0 B 0 B 0 B 6.4
GiB 102
DB 0 B 573 MiB 0 B 0 B 0 B 557
MiB 22
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
0
TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B 0 B
125
MAXIMUMS:
LOG 12 MiB 18 MiB 0 B 0 B 0 B 17
MiB
WAL 45 GiB 101 GiB 0 B 0 B 0 B 145
GiB
DB 0 B 688 MiB 0 B 0 B 0 B 670
MiB
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
TOTALS 45 GiB 101 GiB 0 B 0 B 0 B 0 B
----
Here are the osd.2... the problem disapeared from alone
very strange...
root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB)
1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB)
2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB)
RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
slow_total:5701113793740, db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW * *
REAL FILES
LOG 920 MiB 4.0 GiB 0 B 0 B 0 B 10
MiB 1
WAL 31 GiB 17 GiB 0 B 0 B 0 B 48
GiB 765
DB 0 B 193 MiB 0 B 0 B 0 B 175
MiB 30
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
0
TOTALS 32 GiB 21 GiB 0 B 0 B 0 B 0 B
796
MAXIMUMS:
LOG 920 MiB 4.0 GiB 0 B 0 B 0 B 17
MiB
WAL 45 GiB 149 GiB 0 B 0 B 0 B 192
GiB
DB 0 B 762 MiB 0 B 0 B 0 B 741
MiB
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
TOTALS 45 GiB 153 GiB 0 B 0 B 0 B 0 B
root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 15043915776,
"bluefs_used": 33274462208
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 169235963904,
"bluefs_used": 18668847104
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5602453327872,
"bluefs_used": 0,
"bluefs max available": 5600865222656
}
}
Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov <
igor.fedotov@xxxxxxxx>:
>
> On 10/6/2021 4:25 PM, José H. Freidhof wrote:
> > hi,
> >
> > no risk no fun 😂 okay
> > I have reset the settings you mentioned to standard.
> >
> > what you exactly mean with taking offline the osd? ceph orch daemon stop
> > osd.2? or mark down?
> "daemon stop" is enough. You might want to set noout flag before that
> though...
> >
> > for the command which path i use? you mean:
> >
> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
> yep
> >
> >
> > Igor Fedotov <ifedotov@xxxxxxx> schrieb am Mi., 6. Okt. 2021, 13:33:
> >
> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote:
> >>> Hi Igor,
> >>>
> >>> yes i have some osd settings set :-) here are my ceph config dump.
> those
> >>> settings are from a redhat document for bluestore devices
> >>> maybe it is that setting causing this problem? "advanced
> >>> mon_compact_on_trim false"???
> >> OMG!!!
> >>
> >> No - mon_compact_on_trim has nothing to deal with bluestore.
> >>
> >> Highly likely it's bluestore_rocksdb_options which hurts...
> >> Documentations tend to fall behind the best practices.... I would
> >> strongly discourage you from using non-default settings unless it's
> >> absolutely clear why this is necessary.
> >>
> >> Even at the first glance the following settings (just a few ones I'm
> >> completely aware) are suboptimal/non-recommended:
> >>
> >> rocksdb_perf
> >>
> >> bluefs_sync_write
> >>
> >> bluefs_csum_type
> >>
> >>
> >> Not to mention bluestore_rocksdb_options which hasn't got much adoption
> >> so far and apparently greatly alters rocksdb behavior...
> >>
> >>
> >> So I would suggest to revert rocksdb options back to default, run the
> >> compaction and if it succeeds monitor the OSD for a while. Then if it
> >> works fine - apply the same for others
> >>
> >>
> >> Hope this helps,
> >>
> >> Igor
> >>
> >>
> >>
> >>> i will test it this afternoon... at the moment are everything semi
> >>> prodcuctive and i need to repair one osd node.. because i think of this
> >>> reason the osds crashed on the node and the osd container crashes with
> a
> >>> dump while coming up now.
> >>> need first to replicate all between all three nodes and then i can take
> >>> offline the osd.2.and test your command. i will inform you later...
> >>>
> >>> root@cd88-ceph-osdh-01:/# ceph config dump
> >>> WHO MASK
> >>> LEVEL OPTION VALUE
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> RO
> >>> global
> >>> advanced leveldb_max_open_files 131072
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced mon_compact_on_trim false
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> dev ms_crc_data false
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced osd_deep_scrub_interval
> 1209600.000000
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced osd_max_scrubs 16
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced osd_scrub_load_threshold 0.010000
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced osd_scrub_max_interval
> 1209600.000000
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced osd_scrub_min_interval 86400.000000
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced perf true
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced rbd_readahead_disable_after_bytes 0
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced rbd_readahead_max_bytes 4194304
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced rocksdb_perf true
> >>>
> >>>
> >>>
> >>>
> >>> global
> >>> advanced throttler_perf_counter false
> >>>
> >>>
> >>>
> >>>
> >>> mon
> >>> advanced auth_allow_insecure_global_id_reclaim false
> >>>
> >>>
> >>>
> >>>
> >>> mon
> >>> advanced cluster_network 10.50.50.0/24
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mon
> >>> advanced mon_osd_down_out_interval 300
> >>>
> >>>
> >>>
> >>>
> >>> mon
> >>> advanced public_network 10.50.50.0/24
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mgr
> >>> advanced mgr/cephadm/container_init True
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mgr
> >>> advanced mgr/cephadm/device_enhanced_scan true
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mgr
> >>> advanced mgr/cephadm/migration_current 2
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mgr
> >>> advanced mgr/cephadm/warn_on_stray_daemons false
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> mgr
> >>> advanced mgr/cephadm/warn_on_stray_hosts false
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> osd
> >>> advanced bluefs_sync_write true
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_cache_autotune true
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_cache_kv_ratio 0.200000
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_cache_meta_ratio 0.800000
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_cache_size 2147483648
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_cache_size_hdd 2147483648
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced bluestore_csum_type none
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_extent_map_shard_max_size 200
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_extent_map_shard_min_size 50
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev bluestore_extent_map_shard_target_size 100
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced bluestore_rocksdb_options
> >>>
> >>
> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
> >>> *
> >>> osd
> >>> advanced mon_osd_cache_size 1024
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev ms_crc_data false
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_map_share_max_epochs 5
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_max_backfills 1
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev osd_max_pg_log_entries 10
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev osd_memory_cache_min 3000000000
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >> host:cd133-ceph-osdh-01
> >>> basic osd_memory_target 5797322383
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >> host:cd133k-ceph-osdh-01
> >>> basic osd_memory_target 9402402385
> >>>
> >>>
> >>>
> >>>
> >>> osd
> host:cd88-ceph-osdh-01
> >>> basic osd_memory_target 5797322096
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_memory_target_autotune true
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev osd_min_pg_log_entries 10
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_op_num_shards 8
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> osd
> >>> advanced osd_op_num_threads_per_shard 2
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *
> >>> osd
> >>> dev osd_pg_log_dups_tracked 10
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> dev osd_pg_log_trim_min 10
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_recovery_max_active 3
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_recovery_max_single_start 1
> >>>
> >>>
> >>>
> >>>
> >>> osd
> >>> advanced osd_recovery_sleep 0.000000
> >>>
> >>>
> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov <
> ifedotov@xxxxxxx
> >>> :
> >>>
> >>>> Jose,
> >>>>
> >>>> In fact 48GB is a way too much for WAL drive - usually the write ahead
> >> log
> >>>> tend to be 2-4 GBs.
> >>>>
> >>>> But in your case it's ~150GB, while DB itself is very small
> (146MB!!!):
> >>>>
> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
> >>>> 154 GiB 2400
> >>>>
> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
> >>>> 146 MiB 30
> >>>>
> >>>>
> >>>> which means that there are some issues with RocksDB's WAL processing,
> >>>> which needs some troubleshooting...
> >>>>
> >>>> Curious if other OSDs are suffering from the same and whether you have
> >> any
> >>>> custom settings for your OSD(s)?
> >>>>
> >>>> Additionally you might want to try the following command to compact
> this
> >>>> specific OSD manually and check if this would normalize the DB layout
> -
> >> the
> >>>> majority of data has to be at DB level not WAL. Please share the
> >> resulting
> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command) after
> the
> >>>> compaction is fulfiled and OSD is restarted.
> >>>>
> >>>> The compaction command to be applied on an offline OSD:
> >> "ceph-kvstore-tool
> >>>> bluestore-kv <path-to-osd> compact"
> >>>>
> >>>> Even if the above works great please refrain from applying that
> >> compaction
> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would WAL
> grow
> >>>> again or not?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Igor
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote:
> >>>>
> >>>> Hello Igor,
> >>>>
> >>>> yes the volume is nvme wal partitions for the bluestore devicegroups
> >> are
> >>>> only 48gb each
> >>>>
> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb
> (WAL)
> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb
> >> (rock.db)
> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db)
> >>>>
> >>>> each blustore have 1 partition nvme,ssd and hdd like described in the
> >>>> documentation
> >>>>
> >>
> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
> >>>> is this to small or can i adjust the max allocation on the wal nvme
> >> device
> >>>> in the ceph configuration?
> >>>> i know that the ssd and nvme are to small for those 5.5tb disk... its
> 1%
> >>>> only ot the rotation disk.
> >>>> i am new in ceph and still or always learning, but we are in a little
> >>>> hurry because our other datastores are old and full.
> >>>>
> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device
> >> info
> >>>> {
> >>>> "dev": {
> >>>> "device": "BDEV_WAL",
> >>>> "total": 48318377984,
> >>>> "free": 1044480,
> >>>> "bluefs_used": 48317333504
> >>>> },
> >>>> "dev": {
> >>>> "device": "BDEV_DB",
> >>>> "total": 187904811008,
> >>>> "free": 68757217280,
> >>>> "bluefs_used": 119147593728
> >>>> },
> >>>> "dev": {
> >>>> "device": "BDEV_SLOW",
> >>>> "total": 6001172414464,
> >>>> "free": 5624912359424,
> >>>> "bluefs_used": 0,
> >>>> "bluefs max available": 5624401231872
> >>>> }
> >>>> }
> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB)
> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB)
> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
> >> db_total:178509578240,
> >>>> slow_total:5701113793740, db_avail:103884521472
> >>>> Usage matrix:
> >>>> DEV/LEV WAL DB SLOW * *
> >>>> REAL FILES
> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
> >>>> 7.5 MiB 1
> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
> >>>> 154 GiB 2400
> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
> >>>> 146 MiB 30
> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
> >> 0
> >>>> B 0
> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B
> >> 0
> >>>> B 2431
> >>>> MAXIMUMS:
> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
> >> 17
> >>>> MiB
> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
> >>>> 192 GiB
> >>>> DB 0 B 762 MiB 0 B 0 B 0 B
> >>>> 741 MiB
> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
> >> 0 B
> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B
> >> 0 B
> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <
> >> ifedotov@xxxxxxx>:
> >>>>> Hey Jose,
> >>>>>
> >>>>> it looks like your WAL volume is out of space which looks weird given
> >>>>> its capacity = 48Gb.
> >>>>>
> >>>>> Could you please share the output of the following commands:
> >>>>>
> >>>>> ceph daemon osd.N bluestore bluefs device info
> >>>>>
> >>>>> ceph daemon osd.N bluefs stats
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Igor
> >>>>>
> >>>>>
> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote:
> >>>>>> Hello together
> >>>>>>
> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this
> >> messages
> >>>>> in
> >>>>>> the service logs of the osd daemons.
> >>>>>>
> >>>>>> we have three osd nodes .. each node has 20osds as bluestore with
> >>>>>> nvme/ssd/hdd
> >>>>>>
> >>>>>> is this a bug or maybe i have some settings wrong?
> >>>>>>
> >>>>>>
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New
> >>>>> memtable
> >>>>>> created with log file: #9588. Immutable memtables: 1.
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev
> >> 0,
> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
> >> 0xb40000000,
> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.547575)
> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling
> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots
> >> available
> >>>>> 1,
> >>>>>> compaction slots available 1, flush slots scheduled 1, compaction
> >> slots
> >>>>>> scheduled 0
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
> >>>>>> memtable with next log file: 9587
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
> >>>>>> memtable with next log file: 9588
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850547916,
> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2,
> >>>>> "num_entries":
> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926,
> "memory_usage":
> >>>>>> 130479920, "flush_reason": "Write Buffer Full"}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] Level-0
> >> flush
> >>>>>> table #9589: started
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850559292,
> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation",
> >>>>> "file_number":
> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size":
> 3247855,
> >>>>>> "index_size": 1031, "index_partitions": 0, "top_level_index_size":
> 0,
> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
> >>>>>> "filter_size": 197, "raw_key_size": 1088, "raw_average_key_size":
> 16,
> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739,
> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32,
> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, "format_version":
> >> 0,
> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
> >>>>>> "compression": "NoCompression", "compression_options":
> >> "window_bits=-14;
> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time":
> >>>>> 1633511730,
> >>>>>> "file_creation_time": 1633511850}}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] Level-0
> >> flush
> >>>>>> table #9589: 3249934 bytes OK
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.559362)
> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.559583)
> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #1
> >> done
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.559586)
> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #2
> >> done
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.559601)
> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709, "event":
> >>>>>> "flush_finished", "output_compression": "NoCompression",
> "lsm_state":
> >>>>> [8,
> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.559638)
> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary:
> files[8 1
> >>>>> 0 0
> >>>>>> 0 0 0] max score 1.00
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] [L]
> [JOB
> >>>>> 5710]
> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L]
> >>>>> Compaction
> >>>>>> start summary: Base version 3090 Base level 0, inputs: [9589(3173KB)
> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB)
> >>>>> 9570(405KB)
> >>>>>> 9567(29KB)], [9564(1115KB)]
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850559956,
> >>>>>> "job": 5710, "event": "compaction_started", "compaction_reason":
> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, 9573,
> >>>>> 9570,
> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size": 19542092}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] [L]
> [JOB
> >>>>> 5710]
> >>>>>> Generated table #9590: 36 keys, 3249524 bytes
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850582987,
> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation",
> >>>>> "file_number":
> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size":
> 3247449,
> >>>>>> "index_size": 1031, "index_partitions": 0, "top_level_index_size":
> 0,
> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
> >>>>>> "filter_size": 197, "raw_key_size": 576, "raw_average_key_size": 16,
> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173,
> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0,
> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, "format_version":
> >> 0,
> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
> >>>>>> "compression": "NoCompression", "compression_options":
> >> "window_bits=-14;
> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0,
> >>>>>> "file_creation_time": 1633511850}}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] [L]
> [JOB
> >>>>> 5710]
> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.583469)
> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0 0
> 0 0
> >>>>> 0]
> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8, 1)
> >>>>> out(1)
> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2)
> write-amplify(0.2)
> >>>>> OK,
> >>>>>> records in: 376, records dropped: 340 output_compression:
> >> NoCompression
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
> >> 2021/10/06-09:17:30.583498)
> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710, "event":
> >>>>>> "compaction_finished", "compaction_time_micros": 23098,
> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1,
> >>>>> "num_output_files":
> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376,
> >>>>>> "num_output_records": 36, "num_subcompactions": 1,
> >> "output_compression":
> >>>>>> "NoCompression", "num_single_delete_mismatches": 0,
> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0,
> >> 0]}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583615,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9589}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583648,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9586}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583675,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9583}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583709,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9580}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583739,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9576}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583769,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9573}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583804,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9570}
> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
> >> 1633511850583835,
> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9567}
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>> --
> >>>>
> >>>> Mit freundlichen Grüßen,
> >>>>
> >>>> -
> >>>>
> >>>> José H. Freidhof
> >>>>
> >>>> Reyerhütterstrasse 130b
> >>>> 41065 Mönchengladbach
> >>>> eMail: harald.freidhof@xxxxxxxxx
> >>>> mobil: +49 (0) 1523 – 717 7801
> >>>>
> >>>>
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
--
Mit freundlichen Grüßen,
-
José H. Freidhof
Reyerhütterstrasse 130b
41065 Mönchengladbach
eMail: harald.freidhof@xxxxxxxxx
mobil: +49 (0) 1523 – 717 7801
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
