Hi Igor,
yesss
Am Do., 7. Okt. 2021 um 19:24 Uhr schrieb Igor Fedotov <
igor.fedotov@xxxxxxxx>:
> Hey Jose,
>
> so you applied default settings and compacted osd.8, right?
>
> This helped for a short time and now it's back to the same state, right?
>
>
> Igor
>
>
> On 10/7/2021 12:46 PM, José H. Freidhof wrote:
>
> Good morning,
>
> i checked today the osd.8 and the log shows again the same error
> bluefs _allocate unable to allocate 0x100000 on bdev 0, allocator name
> bluefs-wal, allocator type hybrid, capacity 0xb40000000, block size
> 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>
> any idea why that could be?
>
> Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof <
> harald.freidhof@xxxxxxxxxxxxxx>:
>
>> Hi Igor,
>>
>> today i repaired one osd node and all osd´s on the node, creating them
>> new again....
>> after that i waited for the rebalance/recovery process and the cluster
>> was healthy after some hours..
>>
>> i notices that the osd.2 does not have any more this error in the log.
>> but i noticed it now on the same node on osd.8... so i did the test that
>> you suggested on osd.8
>>
>> it took nearly 20minutes to compact those db´s on the bluestore, but it
>> helped... the problem on osd.8 is gone...
>>
>>
>> *so the problem that i have with the alloc on the wal device seems to be
>> random on different nodes and osd´s and looks like it comes, stay a while
>> and disappears after a longer while... *
>>
>> here are the results that you suggested:
>>
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
>> {
>> "dev": {
>> "device": "BDEV_WAL",
>> "total": 48318377984,
>> "free": 1044480,
>> "bluefs_used": 48317333504
>> },
>> "dev": {
>> "device": "BDEV_DB",
>> "total": 187904811008,
>> "free": 79842762752,
>> "bluefs_used": 108062048256
>> },
>> "dev": {
>> "device": "BDEV_SLOW",
>> "total": 6001172414464,
>> "free": 5510727389184,
>> "bluefs_used": 0,
>> "bluefs max available": 5508815847424
>> }
>> }
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
>> 1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB)
>> 2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB)
>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>> Usage matrix:
>> DEV/LEV WAL DB SLOW * *
>> REAL FILES
>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B
>> 9.7 MiB 1
>> WAL 45 GiB 100 GiB 0 B 0 B 0 B
>> 144 GiB 2319
>> DB 0 B 276 MiB 0 B 0 B 0 B
>> 249 MiB 47
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B 0
>> TOTALS 45 GiB 109 GiB 0 B 0 B 0 B 0
>> B 2367
>> MAXIMUMS:
>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B
>> 20 MiB
>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>> 192 GiB
>> DB 0 B 762 MiB 0 B 0 B 0 B
>> 738 MiB
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B
>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B 0
>> B
>>
>> ---
>>
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb:
>> [db_impl/db_impl_write.cc:1668] [L] New memtable created with log file:
>> #13656. Immutable memtables: 1.
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>> fragmentation 0, allocated 0x0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time
>> 2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198]
>> Calling FlushMemTableToOutputFile with column family [L], flush slots
>> available 1, compaction slots available 1, flush slots scheduled 1,
>> compaction slots scheduled 0
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
>> [L] [JOB 8859] Flushing memtable with next log file: 13655
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
>> [L] [JOB 8859] Flushing memtable with next log file: 13656
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
>> {"time_micros": 1633549894998273, "job": 8859, "event": "flush_started",
>> "num_memtables": 2, "num_entries": 3662, "num_deletes": 0,
>> "total_data_size": 130482337, "memory_usage": 132976224, "flush_reason":
>> "Write Buffer Full"}
>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350]
>> [L] [JOB 8859] Level-0 flush table #13657: started
>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
>> {"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event":
>> "table_file_creation", "file_number": 13657, "file_size": 2952537,
>> "table_properties": {"data_size": 2951222, "index_size": 267,
>> "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
>> 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size":
>> 1120, "raw_average_key_size": 16, "raw_value_size": 2950151,
>> "raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70,
>> "num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0,
>> "format_version": 0, "fixed_key_len": 0, "filter_policy":
>> "rocksdb.BuiltinBloomFilter", "column_family_name": "L",
>> "column_family_id": 10, "comparator": "leveldb.BytewiseComparator",
>> "merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
>> "property_collectors": "[]", "compression": "NoCompression",
>> "compression_options": "window_bits=-14; level=32767; strategy=0;
>> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
>> 1633549854, "oldest_key_time": 1633549854, "file_creation_time":
>> 1633549894}}
>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401]
>> [L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:~# ceph osd set noout
>> root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8
>> root@cd88-ceph-osdh-01:~# ceph orch ps
>> ...
>> osd.7 cd133-ceph-osdh-01
>> running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd
>> 8a98ae61f0eb
>> osd.8 cd88-ceph-osdh-01
>> stopped 5s ago - - 5528M <unknown> <unknown>
>> <unknown>
>> osd.9 cd133k-ceph-osdh-01
>> running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd
>> 0ff7584b1808
>> ...
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact
>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 0
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_db_and_around read-only:0 repair:0
>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>> discard not supported
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
>> rotational discard not supported
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 1 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>> size 175 GiB
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>> discard not supported
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
>> 5.5 TiB
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
>> non-rotational discard supported
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 0 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>> size 45 GiB
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
>> block size 0x100000
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
>> block size 0x100000
>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc shared,
>> id 2, capacity 0x57541c00000, block size 0x10000
>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount
>> shared_bdev_used = 0
>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _prepare_db_environment set db_paths to db,178509578240
>> db.slow,5701113793740
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_db opened rocksdb path db options
>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_super_meta old nid_max 167450
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_super_meta old blobid_max 30720
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_super_meta freelist_type bitmap
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_super_meta ondisk_format 4 compat_ondisk_format 3
>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_super_meta min_alloc_size 0x1000
>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init
>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg
>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _init_alloc opening allocation metadata
>> 2021-10-06T22:01:41.031+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid,
>> capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000,
>> fragmentation 2.76445e-05
>> 2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount
>> 2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
>> 2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
>> 2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
>> rotational discard not supported
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 1 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>> size 175 GiB
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>> discard not supported
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
>> 5.5 TiB
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
>> non-rotational discard supported
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
>> 0 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>> size 45 GiB
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
>> block size 0x100000
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
>> block size 0x100000
>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc shared,
>> id 2, capacity 0x57541c00000, block size 0x10000
>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount
>> shared_bdev_used = 0
>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _prepare_db_environment set db_paths to db,178509578240
>> db.slow,5701113793740
>> 2021-10-06T22:09:36.519+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>> _open_db opened rocksdb path db options
>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
>> 2021-10-06T22:09:54.067+0200 7f87bde3c240 1
>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount
>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount
>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
>> 2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
>> 2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>> 2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown
>> 2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800
>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:/# ceph -s
>> cluster:
>> id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
>> health: HEALTH_OK
>>
>> services:
>> mon: 3 daemons, quorum
>> cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h)
>> mgr: cd133-ceph-mon-01.mzapob(active, since 15h), standbys:
>> cd133k-ceph-mon-01.imikwh
>> osd: 60 osds: 60 up (since 2m), 60 in (since 3h)
>> rgw: 4 daemons active (2 hosts, 1 zones)
>> tcmu-runner: 10 portals active (2 hosts)
>>
>> data:
>> pools: 6 pools, 361 pgs
>> objects: 2.46M objects, 8.0 TiB
>> usage: 33 TiB used, 304 TiB / 338 TiB avail
>> pgs: 361 active+clean
>>
>> io:
>> client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr
>>
>>
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
>> {
>> "dev": {
>> "device": "BDEV_WAL",
>> "total": 48318377984,
>>
>> * "free": 41354784768, * "bluefs_used": 6963593216
>> },
>> "dev": {
>> "device": "BDEV_DB",
>> "total": 187904811008,
>> "free": 187302928384,
>> "bluefs_used": 601882624
>> },
>> "dev": {
>> "device": "BDEV_SLOW",
>> "total": 6001172414464,
>> "free": 5507531620352,
>> "bluefs_used": 0,
>> "bluefs max available": 5505566572544
>> }
>> }
>>
>> ---
>>
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
>> 0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB)
>> 1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB)
>> 2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB)
>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>> Usage matrix:
>> DEV/LEV WAL DB SLOW * *
>> REAL FILES
>> LOG 12 MiB 18 MiB 0 B 0 B 0 B
>> 10 MiB 0
>> WAL 6.5 GiB 0 B 0 B 0 B 0 B
>> 6.4 GiB 102
>> DB 0 B 573 MiB 0 B 0 B 0 B
>> 557 MiB 22
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B 0
>> TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B 0
>> B 125
>> MAXIMUMS:
>> LOG 12 MiB 18 MiB 0 B 0 B 0 B
>> 17 MiB
>> WAL 45 GiB 101 GiB 0 B 0 B 0 B
>> 145 GiB
>> DB 0 B 688 MiB 0 B 0 B 0 B
>> 670 MiB
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B
>> TOTALS 45 GiB 101 GiB 0 B 0 B 0 B 0
>> B
>>
>> ----
>>
>>
>> Here are the osd.2... the problem disapeared from alone
>> very strange...
>>
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
>> 0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB)
>> 1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB)
>> 2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB)
>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>> Usage matrix:
>> DEV/LEV WAL DB SLOW * *
>> REAL FILES
>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B
>> 10 MiB 1
>> WAL 31 GiB 17 GiB 0 B 0 B 0 B
>> 48 GiB 765
>> DB 0 B 193 MiB 0 B 0 B 0 B
>> 175 MiB 30
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B 0
>> TOTALS 32 GiB 21 GiB 0 B 0 B 0 B 0
>> B 796
>> MAXIMUMS:
>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B
>> 17 MiB
>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>> 192 GiB
>> DB 0 B 762 MiB 0 B 0 B 0 B
>> 741 MiB
>> SLOW 0 B 0 B 0 B 0 B 0 B 0
>> B
>> TOTALS 45 GiB 153 GiB 0 B 0 B 0 B 0
>> B
>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info
>> {
>> "dev": {
>> "device": "BDEV_WAL",
>> "total": 48318377984,
>> "free": 15043915776,
>> "bluefs_used": 33274462208
>> },
>> "dev": {
>> "device": "BDEV_DB",
>> "total": 187904811008,
>> "free": 169235963904,
>> "bluefs_used": 18668847104
>> },
>> "dev": {
>> "device": "BDEV_SLOW",
>> "total": 6001172414464,
>> "free": 5602453327872,
>> "bluefs_used": 0,
>> "bluefs max available": 5600865222656
>> }
>> }
>>
>>
>>
>>
>>
>>
>> Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov <
>> igor.fedotov@xxxxxxxx>:
>>
>>>
>>> On 10/6/2021 4:25 PM, José H. Freidhof wrote:
>>> > hi,
>>> >
>>> > no risk no fun 😂 okay
>>> > I have reset the settings you mentioned to standard.
>>> >
>>> > what you exactly mean with taking offline the osd? ceph orch daemon
>>> stop
>>> > osd.2? or mark down?
>>> "daemon stop" is enough. You might want to set noout flag before that
>>> though...
>>> >
>>> > for the command which path i use? you mean:
>>> >
>>> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
>>> yep
>>> >
>>> >
>>> > Igor Fedotov <ifedotov@xxxxxxx> schrieb am Mi., 6. Okt. 2021, 13:33:
>>> >
>>> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote:
>>> >>> Hi Igor,
>>> >>>
>>> >>> yes i have some osd settings set :-) here are my ceph config dump.
>>> those
>>> >>> settings are from a redhat document for bluestore devices
>>> >>> maybe it is that setting causing this problem? "advanced
>>> >>> mon_compact_on_trim false"???
>>> >> OMG!!!
>>> >>
>>> >> No - mon_compact_on_trim has nothing to deal with bluestore.
>>> >>
>>> >> Highly likely it's bluestore_rocksdb_options which hurts...
>>> >> Documentations tend to fall behind the best practices.... I would
>>> >> strongly discourage you from using non-default settings unless it's
>>> >> absolutely clear why this is necessary.
>>> >>
>>> >> Even at the first glance the following settings (just a few ones I'm
>>> >> completely aware) are suboptimal/non-recommended:
>>> >>
>>> >> rocksdb_perf
>>> >>
>>> >> bluefs_sync_write
>>> >>
>>> >> bluefs_csum_type
>>> >>
>>> >>
>>> >> Not to mention bluestore_rocksdb_options which hasn't got much
>>> adoption
>>> >> so far and apparently greatly alters rocksdb behavior...
>>> >>
>>> >>
>>> >> So I would suggest to revert rocksdb options back to default, run the
>>> >> compaction and if it succeeds monitor the OSD for a while. Then if it
>>> >> works fine - apply the same for others
>>> >>
>>> >>
>>> >> Hope this helps,
>>> >>
>>> >> Igor
>>> >>
>>> >>
>>> >>
>>> >>> i will test it this afternoon... at the moment are everything semi
>>> >>> prodcuctive and i need to repair one osd node.. because i think of
>>> this
>>> >>> reason the osds crashed on the node and the osd container crashes
>>> with a
>>> >>> dump while coming up now.
>>> >>> need first to replicate all between all three nodes and then i can
>>> take
>>> >>> offline the osd.2.and test your command. i will inform you later...
>>> >>>
>>> >>> root@cd88-ceph-osdh-01:/# ceph config dump
>>> >>> WHO MASK
>>> >>> LEVEL OPTION VALUE
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> RO
>>> >>> global
>>> >>> advanced leveldb_max_open_files 131072
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced mon_compact_on_trim false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> dev ms_crc_data false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced osd_deep_scrub_interval
>>> 1209600.000000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced osd_max_scrubs 16
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced osd_scrub_load_threshold 0.010000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced osd_scrub_max_interval
>>> 1209600.000000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced osd_scrub_min_interval
>>> 86400.000000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced perf true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced rbd_readahead_disable_after_bytes 0
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced rbd_readahead_max_bytes 4194304
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced rocksdb_perf true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> global
>>> >>> advanced throttler_perf_counter false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> mon
>>> >>> advanced auth_allow_insecure_global_id_reclaim false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> mon
>>> >>> advanced cluster_network 10.50.50.0/24
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mon
>>> >>> advanced mon_osd_down_out_interval 300
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> mon
>>> >>> advanced public_network 10.50.50.0/24
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mgr
>>> >>> advanced mgr/cephadm/container_init True
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mgr
>>> >>> advanced mgr/cephadm/device_enhanced_scan true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mgr
>>> >>> advanced mgr/cephadm/migration_current 2
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mgr
>>> >>> advanced mgr/cephadm/warn_on_stray_daemons false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> mgr
>>> >>> advanced mgr/cephadm/warn_on_stray_hosts false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> osd
>>> >>> advanced bluefs_sync_write true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_cache_autotune true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_cache_kv_ratio 0.200000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_cache_meta_ratio 0.800000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_cache_size 2147483648
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_cache_size_hdd 2147483648
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced bluestore_csum_type none
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_extent_map_shard_max_size 200
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_extent_map_shard_min_size 50
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev bluestore_extent_map_shard_target_size 100
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced bluestore_rocksdb_options
>>> >>>
>>> >>
>>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>>> >>> *
>>> >>> osd
>>> >>> advanced mon_osd_cache_size 1024
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev ms_crc_data false
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_map_share_max_epochs 5
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_max_backfills 1
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev osd_max_pg_log_entries 10
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev osd_memory_cache_min 3000000000
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >> host:cd133-ceph-osdh-01
>>> >>> basic osd_memory_target 5797322383
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >> host:cd133k-ceph-osdh-01
>>> >>> basic osd_memory_target 9402402385
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> host:cd88-ceph-osdh-01
>>> >>> basic osd_memory_target 5797322096
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_memory_target_autotune true
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev osd_min_pg_log_entries 10
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_op_num_shards 8
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> osd
>>> >>> advanced osd_op_num_threads_per_shard 2
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *
>>> >>> osd
>>> >>> dev osd_pg_log_dups_tracked 10
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> dev osd_pg_log_trim_min 10
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_recovery_max_active 3
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_recovery_max_single_start 1
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> osd
>>> >>> advanced osd_recovery_sleep 0.000000
>>> >>>
>>> >>>
>>> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov <
>>> ifedotov@xxxxxxx
>>> >>> :
>>> >>>
>>> >>>> Jose,
>>> >>>>
>>> >>>> In fact 48GB is a way too much for WAL drive - usually the write
>>> ahead
>>> >> log
>>> >>>> tend to be 2-4 GBs.
>>> >>>>
>>> >>>> But in your case it's ~150GB, while DB itself is very small
>>> (146MB!!!):
>>> >>>>
>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>>> >>>> 154 GiB 2400
>>> >>>>
>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>> >>>> 146 MiB 30
>>> >>>>
>>> >>>>
>>> >>>> which means that there are some issues with RocksDB's WAL
>>> processing,
>>> >>>> which needs some troubleshooting...
>>> >>>>
>>> >>>> Curious if other OSDs are suffering from the same and whether you
>>> have
>>> >> any
>>> >>>> custom settings for your OSD(s)?
>>> >>>>
>>> >>>> Additionally you might want to try the following command to compact
>>> this
>>> >>>> specific OSD manually and check if this would normalize the DB
>>> layout -
>>> >> the
>>> >>>> majority of data has to be at DB level not WAL. Please share the
>>> >> resulting
>>> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command) after
>>> the
>>> >>>> compaction is fulfiled and OSD is restarted.
>>> >>>>
>>> >>>> The compaction command to be applied on an offline OSD:
>>> >> "ceph-kvstore-tool
>>> >>>> bluestore-kv <path-to-osd> compact"
>>> >>>>
>>> >>>> Even if the above works great please refrain from applying that
>>> >> compaction
>>> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would WAL
>>> grow
>>> >>>> again or not?
>>> >>>>
>>> >>>> Thanks,
>>> >>>>
>>> >>>> Igor
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote:
>>> >>>>
>>> >>>> Hello Igor,
>>> >>>>
>>> >>>> yes the volume is nvme wal partitions for the bluestore
>>> devicegroups
>>> >> are
>>> >>>> only 48gb each
>>> >>>>
>>> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb
>>> (WAL)
>>> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb
>>> >> (rock.db)
>>> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db)
>>> >>>>
>>> >>>> each blustore have 1 partition nvme,ssd and hdd like described in
>>> the
>>> >>>> documentation
>>> >>>>
>>> >>
>>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
>>> >>>> is this to small or can i adjust the max allocation on the wal nvme
>>> >> device
>>> >>>> in the ceph configuration?
>>> >>>> i know that the ssd and nvme are to small for those 5.5tb disk...
>>> its 1%
>>> >>>> only ot the rotation disk.
>>> >>>> i am new in ceph and still or always learning, but we are in a
>>> little
>>> >>>> hurry because our other datastores are old and full.
>>> >>>>
>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device
>>> >> info
>>> >>>> {
>>> >>>> "dev": {
>>> >>>> "device": "BDEV_WAL",
>>> >>>> "total": 48318377984,
>>> >>>> "free": 1044480,
>>> >>>> "bluefs_used": 48317333504
>>> >>>> },
>>> >>>> "dev": {
>>> >>>> "device": "BDEV_DB",
>>> >>>> "total": 187904811008,
>>> >>>> "free": 68757217280,
>>> >>>> "bluefs_used": 119147593728
>>> >>>> },
>>> >>>> "dev": {
>>> >>>> "device": "BDEV_SLOW",
>>> >>>> "total": 6001172414464,
>>> >>>> "free": 5624912359424,
>>> >>>> "bluefs_used": 0,
>>> >>>> "bluefs max available": 5624401231872
>>> >>>> }
>>> >>>> }
>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
>>> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
>>> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB)
>>> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB)
>>> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>>> >> db_total:178509578240,
>>> >>>> slow_total:5701113793740, db_avail:103884521472
>>> >>>> Usage matrix:
>>> >>>> DEV/LEV WAL DB SLOW * *
>>> >>>> REAL FILES
>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>>> >>>> 7.5 MiB 1
>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>>> >>>> 154 GiB 2400
>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>> >>>> 146 MiB 30
>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> >> 0
>>> >>>> B 0
>>> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B
>>> >> 0
>>> >>>> B 2431
>>> >>>> MAXIMUMS:
>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>>> >> 17
>>> >>>> MiB
>>> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>>> >>>> 192 GiB
>>> >>>> DB 0 B 762 MiB 0 B 0 B 0 B
>>> >>>> 741 MiB
>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> >> 0 B
>>> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B
>>> >> 0 B
>>> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <
>>> >> ifedotov@xxxxxxx>:
>>> >>>>> Hey Jose,
>>> >>>>>
>>> >>>>> it looks like your WAL volume is out of space which looks weird
>>> given
>>> >>>>> its capacity = 48Gb.
>>> >>>>>
>>> >>>>> Could you please share the output of the following commands:
>>> >>>>>
>>> >>>>> ceph daemon osd.N bluestore bluefs device info
>>> >>>>>
>>> >>>>> ceph daemon osd.N bluefs stats
>>> >>>>>
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>>
>>> >>>>> Igor
>>> >>>>>
>>> >>>>>
>>> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote:
>>> >>>>>> Hello together
>>> >>>>>>
>>> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this
>>> >> messages
>>> >>>>> in
>>> >>>>>> the service logs of the osd daemons.
>>> >>>>>>
>>> >>>>>> we have three osd nodes .. each node has 20osds as bluestore with
>>> >>>>>> nvme/ssd/hdd
>>> >>>>>>
>>> >>>>>> is this a bug or maybe i have some settings wrong?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New
>>> >>>>> memtable
>>> >>>>>> created with log file: #9588. Immutable memtables: 1.
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>> bdev
>>> >> 0,
>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>> >> 0xb40000000,
>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.547575)
>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling
>>> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots
>>> >> available
>>> >>>>> 1,
>>> >>>>>> compaction slots available 1, flush slots scheduled 1, compaction
>>> >> slots
>>> >>>>>> scheduled 0
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709]
>>> Flushing
>>> >>>>>> memtable with next log file: 9587
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709]
>>> Flushing
>>> >>>>>> memtable with next log file: 9588
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850547916,
>>> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2,
>>> >>>>> "num_entries":
>>> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926,
>>> "memory_usage":
>>> >>>>>> 130479920, "flush_reason": "Write Buffer Full"}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] Level-0
>>> >> flush
>>> >>>>>> table #9589: started
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850559292,
>>> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation",
>>> >>>>> "file_number":
>>> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size":
>>> 3247855,
>>> >>>>>> "index_size": 1031, "index_partitions": 0,
>>> "top_level_index_size": 0,
>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>>> >>>>>> "filter_size": 197, "raw_key_size": 1088, "raw_average_key_size":
>>> 16,
>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739,
>>> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32,
>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>>> "format_version":
>>> >> 0,
>>> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>>> >>>>>> "compression": "NoCompression", "compression_options":
>>> >> "window_bits=-14;
>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
>>> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time":
>>> >>>>> 1633511730,
>>> >>>>>> "file_creation_time": 1633511850}}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] Level-0
>>> >> flush
>>> >>>>>> table #9589: 3249934 bytes OK
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.559362)
>>> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.559583)
>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #1
>>> >> done
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.559586)
>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #2
>>> >> done
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.559601)
>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709,
>>> "event":
>>> >>>>>> "flush_finished", "output_compression": "NoCompression",
>>> "lsm_state":
>>> >>>>> [8,
>>> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.559638)
>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary:
>>> files[8 1
>>> >>>>> 0 0
>>> >>>>>> 0 0 0] max score 1.00
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] [L]
>>> [JOB
>>> >>>>> 5710]
>>> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L]
>>> >>>>> Compaction
>>> >>>>>> start summary: Base version 3090 Base level 0, inputs:
>>> [9589(3173KB)
>>> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB)
>>> >>>>> 9570(405KB)
>>> >>>>>> 9567(29KB)], [9564(1115KB)]
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850559956,
>>> >>>>>> "job": 5710, "event": "compaction_started", "compaction_reason":
>>> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576,
>>> 9573,
>>> >>>>> 9570,
>>> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size":
>>> 19542092}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] [L]
>>> [JOB
>>> >>>>> 5710]
>>> >>>>>> Generated table #9590: 36 keys, 3249524 bytes
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850582987,
>>> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation",
>>> >>>>> "file_number":
>>> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size":
>>> 3247449,
>>> >>>>>> "index_size": 1031, "index_partitions": 0,
>>> "top_level_index_size": 0,
>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>>> >>>>>> "filter_size": 197, "raw_key_size": 576, "raw_average_key_size":
>>> 16,
>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173,
>>> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0,
>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>>> "format_version":
>>> >> 0,
>>> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>>> >>>>>> "compression": "NoCompression", "compression_options":
>>> >> "window_bits=-14;
>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
>>> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0,
>>> >>>>>> "file_creation_time": 1633511850}}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] [L]
>>> [JOB
>>> >>>>> 5710]
>>> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.583469)
>>> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0
>>> 0 0 0
>>> >>>>> 0]
>>> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8,
>>> 1)
>>> >>>>> out(1)
>>> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2)
>>> write-amplify(0.2)
>>> >>>>> OK,
>>> >>>>>> records in: 376, records dropped: 340 output_compression:
>>> >> NoCompression
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>>> >> 2021/10/06-09:17:30.583498)
>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710,
>>> "event":
>>> >>>>>> "compaction_finished", "compaction_time_micros": 23098,
>>> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1,
>>> >>>>> "num_output_files":
>>> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376,
>>> >>>>>> "num_output_records": 36, "num_subcompactions": 1,
>>> >> "output_compression":
>>> >>>>>> "NoCompression", "num_single_delete_mismatches": 0,
>>> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0,
>>> 0,
>>> >> 0]}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583615,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9589}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583648,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9586}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583675,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9583}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583709,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9580}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583739,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9576}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583769,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9573}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583804,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9570}
>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>> >> 1633511850583835,
>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9567}
>>> >>>>>> _______________________________________________
>>> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> >>>> --
>>>
>>
>>
--
Mit freundlichen Grüßen,
-
José H. Freidhof
Reyerhütterstrasse 130b
41065 Mönchengladbach
eMail: harald.freidhof@xxxxxxxxx
mobil: +49 (0) 1523 – 717 7801
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
