Good morning,
i checked today the osd.8 and the log shows again the same error
bluefs _allocate unable to allocate 0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid, capacity 0xb40000000, block size
0x100000, free 0xff000, fragmentation 0, allocated 0x0
any idea why that could be?
Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof <
harald.freidhof@xxxxxxxxxxxxxx>:
> Hi Igor,
>
> today i repaired one osd node and all osd´s on the node, creating them new
> again....
> after that i waited for the rebalance/recovery process and the cluster was
> healthy after some hours..
>
> i notices that the osd.2 does not have any more this error in the log.
> but i noticed it now on the same node on osd.8... so i did the test that
> you suggested on osd.8
>
> it took nearly 20minutes to compact those db´s on the bluestore, but it
> helped... the problem on osd.8 is gone...
>
>
> *so the problem that i have with the alloc on the wal device seems to be
> random on different nodes and osd´s and looks like it comes, stay a while
> and disappears after a longer while...*
>
> here are the results that you suggested:
>
> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
> {
> "dev": {
> "device": "BDEV_WAL",
> "total": 48318377984,
> "free": 1044480,
> "bluefs_used": 48317333504
> },
> "dev": {
> "device": "BDEV_DB",
> "total": 187904811008,
> "free": 79842762752,
> "bluefs_used": 108062048256
> },
> "dev": {
> "device": "BDEV_SLOW",
> "total": 6001172414464,
> "free": 5510727389184,
> "bluefs_used": 0,
> "bluefs max available": 5508815847424
> }
> }
> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
> 1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB)
> 2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB)
> RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
> slow_total:5701113793740, db_avail:103884521472
> Usage matrix:
> DEV/LEV WAL DB SLOW * *
> REAL FILES
> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B
> 9.7 MiB 1
> WAL 45 GiB 100 GiB 0 B 0 B 0 B
> 144 GiB 2319
> DB 0 B 276 MiB 0 B 0 B 0 B
> 249 MiB 47
> SLOW 0 B 0 B 0 B 0 B 0 B 0
> B 0
> TOTALS 45 GiB 109 GiB 0 B 0 B 0 B 0
> B 2367
> MAXIMUMS:
> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B 20
> MiB
> WAL 45 GiB 149 GiB 0 B 0 B 0 B
> 192 GiB
> DB 0 B 762 MiB 0 B 0 B 0 B
> 738 MiB
> SLOW 0 B 0 B 0 B 0 B 0 B 0 B
> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B 0 B
>
> ---
>
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb:
> [db_impl/db_impl_write.cc:1668] [L] New memtable created with log file:
> #13656. Immutable memtables: 1.
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
> fragmentation 0, allocated 0x0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time
> 2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198]
> Calling FlushMemTableToOutputFile with column family [L], flush slots
> available 1, compaction slots available 1, flush slots scheduled 1,
> compaction slots scheduled 0
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
> [L] [JOB 8859] Flushing memtable with next log file: 13655
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
> [L] [JOB 8859] Flushing memtable with next log file: 13656
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1633549894998273, "job": 8859, "event": "flush_started",
> "num_memtables": 2, "num_entries": 3662, "num_deletes": 0,
> "total_data_size": 130482337, "memory_usage": 132976224, "flush_reason":
> "Write Buffer Full"}
> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350]
> [L] [JOB 8859] Level-0 flush table #13657: started
> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
> {"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event":
> "table_file_creation", "file_number": 13657, "file_size": 2952537,
> "table_properties": {"data_size": 2951222, "index_size": 267,
> "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
> 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size":
> 1120, "raw_average_key_size": 16, "raw_value_size": 2950151,
> "raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70,
> "num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0,
> "format_version": 0, "fixed_key_len": 0, "filter_policy":
> "rocksdb.BuiltinBloomFilter", "column_family_name": "L",
> "column_family_id": 10, "comparator": "leveldb.BytewiseComparator",
> "merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
> "property_collectors": "[]", "compression": "NoCompression",
> "compression_options": "window_bits=-14; level=32767; strategy=0;
> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
> 1633549854, "oldest_key_time": 1633549854, "file_creation_time":
> 1633549894}}
> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401]
> [L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK
>
> ---
>
> root@cd88-ceph-osdh-01:~# ceph osd set noout
> root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8
> root@cd88-ceph-osdh-01:~# ceph orch ps
> ...
> osd.7 cd133-ceph-osdh-01
> running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd
> 8a98ae61f0eb
> osd.8 cd88-ceph-osdh-01
> stopped 5s ago - - 5528M <unknown> <unknown>
> <unknown>
> osd.9 cd133k-ceph-osdh-01
> running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd
> 0ff7584b1808
> ...
>
> ---
>
> root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact
> 2021-10-06T21:53:50.559+0200 7f87bde3c240 0
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_db_and_around read-only:0 repair:0
> 2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
> discard not supported
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
> rotational discard not supported
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 1 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
> size 175 GiB
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
> discard not supported
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
> 5.5 TiB
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
> non-rotational discard supported
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 0 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
> size 45 GiB
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id 0,
> allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> block size 0x100000
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id 1,
> allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
> block size 0x100000
> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc shared, id
> 2, capacity 0x57541c00000, block size 0x10000
> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount shared_bdev_used
> = 0
> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _prepare_db_environment set db_paths to db,178509578240
> db.slow,5701113793740
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_db opened rocksdb path db options
> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_super_meta old nid_max 167450
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_super_meta old blobid_max 30720
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_super_meta freelist_type bitmap
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_super_meta ondisk_format 4 compat_ondisk_format 3
> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_super_meta min_alloc_size 0x1000
> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init
> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg
> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _init_alloc opening allocation metadata
> 2021-10-06T22:01:41.031+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid,
> capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000,
> fragmentation 2.76445e-05
> 2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount
> 2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
> 2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
> 2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
> rotational discard not supported
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 1 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
> size 175 GiB
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
> discard not supported
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block size
> 5.5 TiB
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
> non-rotational discard supported
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device bdev
> 0 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
> size 45 GiB
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id 0,
> allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> block size 0x100000
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id 1,
> allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
> block size 0x100000
> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc shared, id
> 2, capacity 0x57541c00000, block size 0x10000
> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount shared_bdev_used
> = 0
> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _prepare_db_environment set db_paths to db,178509578240
> db.slow,5701113793740
> 2021-10-06T22:09:36.519+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
> _open_db opened rocksdb path db options
> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
> 2021-10-06T22:09:54.067+0200 7f87bde3c240 1
> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount
> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount
> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
> 2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
> 2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
> 2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown
> 2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800
> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>
> ---
>
> root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8
>
> ---
>
> root@cd88-ceph-osdh-01:/# ceph -s
> cluster:
> id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
> health: HEALTH_OK
>
> services:
> mon: 3 daemons, quorum
> cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h)
> mgr: cd133-ceph-mon-01.mzapob(active, since 15h), standbys:
> cd133k-ceph-mon-01.imikwh
> osd: 60 osds: 60 up (since 2m), 60 in (since 3h)
> rgw: 4 daemons active (2 hosts, 1 zones)
> tcmu-runner: 10 portals active (2 hosts)
>
> data:
> pools: 6 pools, 361 pgs
> objects: 2.46M objects, 8.0 TiB
> usage: 33 TiB used, 304 TiB / 338 TiB avail
> pgs: 361 active+clean
>
> io:
> client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr
>
>
>
> ---
>
> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
> {
> "dev": {
> "device": "BDEV_WAL",
> "total": 48318377984,
>
> * "free": 41354784768,* "bluefs_used": 6963593216
> },
> "dev": {
> "device": "BDEV_DB",
> "total": 187904811008,
> "free": 187302928384,
> "bluefs_used": 601882624
> },
> "dev": {
> "device": "BDEV_SLOW",
> "total": 6001172414464,
> "free": 5507531620352,
> "bluefs_used": 0,
> "bluefs max available": 5505566572544
> }
> }
>
> ---
>
> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
> 0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB)
> 1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB)
> 2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB)
> RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
> slow_total:5701113793740, db_avail:103884521472
> Usage matrix:
> DEV/LEV WAL DB SLOW * *
> REAL FILES
> LOG 12 MiB 18 MiB 0 B 0 B 0 B 10
> MiB 0
> WAL 6.5 GiB 0 B 0 B 0 B 0 B
> 6.4 GiB 102
> DB 0 B 573 MiB 0 B 0 B 0 B
> 557 MiB 22
> SLOW 0 B 0 B 0 B 0 B 0 B 0
> B 0
> TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B 0
> B 125
> MAXIMUMS:
> LOG 12 MiB 18 MiB 0 B 0 B 0 B 17
> MiB
> WAL 45 GiB 101 GiB 0 B 0 B 0 B
> 145 GiB
> DB 0 B 688 MiB 0 B 0 B 0 B
> 670 MiB
> SLOW 0 B 0 B 0 B 0 B 0 B 0 B
> TOTALS 45 GiB 101 GiB 0 B 0 B 0 B 0 B
>
> ----
>
>
> Here are the osd.2... the problem disapeared from alone
> very strange...
>
> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
> 0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB)
> 1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB)
> 2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB)
> RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
> slow_total:5701113793740, db_avail:103884521472
> Usage matrix:
> DEV/LEV WAL DB SLOW * *
> REAL FILES
> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B 10
> MiB 1
> WAL 31 GiB 17 GiB 0 B 0 B 0 B 48
> GiB 765
> DB 0 B 193 MiB 0 B 0 B 0 B
> 175 MiB 30
> SLOW 0 B 0 B 0 B 0 B 0 B 0
> B 0
> TOTALS 32 GiB 21 GiB 0 B 0 B 0 B 0
> B 796
> MAXIMUMS:
> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B 17
> MiB
> WAL 45 GiB 149 GiB 0 B 0 B 0 B
> 192 GiB
> DB 0 B 762 MiB 0 B 0 B 0 B
> 741 MiB
> SLOW 0 B 0 B 0 B 0 B 0 B 0 B
> TOTALS 45 GiB 153 GiB 0 B 0 B 0 B 0 B
> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info
> {
> "dev": {
> "device": "BDEV_WAL",
> "total": 48318377984,
> "free": 15043915776,
> "bluefs_used": 33274462208
> },
> "dev": {
> "device": "BDEV_DB",
> "total": 187904811008,
> "free": 169235963904,
> "bluefs_used": 18668847104
> },
> "dev": {
> "device": "BDEV_SLOW",
> "total": 6001172414464,
> "free": 5602453327872,
> "bluefs_used": 0,
> "bluefs max available": 5600865222656
> }
> }
>
>
>
>
>
>
> Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov <
> igor.fedotov@xxxxxxxx>:
>
>>
>> On 10/6/2021 4:25 PM, José H. Freidhof wrote:
>> > hi,
>> >
>> > no risk no fun 😂 okay
>> > I have reset the settings you mentioned to standard.
>> >
>> > what you exactly mean with taking offline the osd? ceph orch daemon stop
>> > osd.2? or mark down?
>> "daemon stop" is enough. You might want to set noout flag before that
>> though...
>> >
>> > for the command which path i use? you mean:
>> >
>> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
>> yep
>> >
>> >
>> > Igor Fedotov <ifedotov@xxxxxxx> schrieb am Mi., 6. Okt. 2021, 13:33:
>> >
>> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote:
>> >>> Hi Igor,
>> >>>
>> >>> yes i have some osd settings set :-) here are my ceph config dump.
>> those
>> >>> settings are from a redhat document for bluestore devices
>> >>> maybe it is that setting causing this problem? "advanced
>> >>> mon_compact_on_trim false"???
>> >> OMG!!!
>> >>
>> >> No - mon_compact_on_trim has nothing to deal with bluestore.
>> >>
>> >> Highly likely it's bluestore_rocksdb_options which hurts...
>> >> Documentations tend to fall behind the best practices.... I would
>> >> strongly discourage you from using non-default settings unless it's
>> >> absolutely clear why this is necessary.
>> >>
>> >> Even at the first glance the following settings (just a few ones I'm
>> >> completely aware) are suboptimal/non-recommended:
>> >>
>> >> rocksdb_perf
>> >>
>> >> bluefs_sync_write
>> >>
>> >> bluefs_csum_type
>> >>
>> >>
>> >> Not to mention bluestore_rocksdb_options which hasn't got much adoption
>> >> so far and apparently greatly alters rocksdb behavior...
>> >>
>> >>
>> >> So I would suggest to revert rocksdb options back to default, run the
>> >> compaction and if it succeeds monitor the OSD for a while. Then if it
>> >> works fine - apply the same for others
>> >>
>> >>
>> >> Hope this helps,
>> >>
>> >> Igor
>> >>
>> >>
>> >>
>> >>> i will test it this afternoon... at the moment are everything semi
>> >>> prodcuctive and i need to repair one osd node.. because i think of
>> this
>> >>> reason the osds crashed on the node and the osd container crashes
>> with a
>> >>> dump while coming up now.
>> >>> need first to replicate all between all three nodes and then i can
>> take
>> >>> offline the osd.2.and test your command. i will inform you later...
>> >>>
>> >>> root@cd88-ceph-osdh-01:/# ceph config dump
>> >>> WHO MASK
>> >>> LEVEL OPTION VALUE
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> RO
>> >>> global
>> >>> advanced leveldb_max_open_files 131072
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced mon_compact_on_trim false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> dev ms_crc_data false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced osd_deep_scrub_interval
>> 1209600.000000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced osd_max_scrubs 16
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced osd_scrub_load_threshold 0.010000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced osd_scrub_max_interval
>> 1209600.000000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced osd_scrub_min_interval 86400.000000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced perf true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced rbd_readahead_disable_after_bytes 0
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced rbd_readahead_max_bytes 4194304
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced rocksdb_perf true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> global
>> >>> advanced throttler_perf_counter false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> mon
>> >>> advanced auth_allow_insecure_global_id_reclaim false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> mon
>> >>> advanced cluster_network 10.50.50.0/24
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mon
>> >>> advanced mon_osd_down_out_interval 300
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> mon
>> >>> advanced public_network 10.50.50.0/24
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mgr
>> >>> advanced mgr/cephadm/container_init True
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mgr
>> >>> advanced mgr/cephadm/device_enhanced_scan true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mgr
>> >>> advanced mgr/cephadm/migration_current 2
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mgr
>> >>> advanced mgr/cephadm/warn_on_stray_daemons false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> mgr
>> >>> advanced mgr/cephadm/warn_on_stray_hosts false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> osd
>> >>> advanced bluefs_sync_write true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_cache_autotune true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_cache_kv_ratio 0.200000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_cache_meta_ratio 0.800000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_cache_size 2147483648
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_cache_size_hdd 2147483648
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced bluestore_csum_type none
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_extent_map_shard_max_size 200
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_extent_map_shard_min_size 50
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev bluestore_extent_map_shard_target_size 100
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced bluestore_rocksdb_options
>> >>>
>> >>
>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>> >>> *
>> >>> osd
>> >>> advanced mon_osd_cache_size 1024
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev ms_crc_data false
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_map_share_max_epochs 5
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_max_backfills 1
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev osd_max_pg_log_entries 10
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev osd_memory_cache_min 3000000000
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >> host:cd133-ceph-osdh-01
>> >>> basic osd_memory_target 5797322383
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >> host:cd133k-ceph-osdh-01
>> >>> basic osd_memory_target 9402402385
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> host:cd88-ceph-osdh-01
>> >>> basic osd_memory_target 5797322096
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_memory_target_autotune true
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev osd_min_pg_log_entries 10
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_op_num_shards 8
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> osd
>> >>> advanced osd_op_num_threads_per_shard 2
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *
>> >>> osd
>> >>> dev osd_pg_log_dups_tracked 10
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> dev osd_pg_log_trim_min 10
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_recovery_max_active 3
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_recovery_max_single_start 1
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> osd
>> >>> advanced osd_recovery_sleep 0.000000
>> >>>
>> >>>
>> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov <
>> ifedotov@xxxxxxx
>> >>> :
>> >>>
>> >>>> Jose,
>> >>>>
>> >>>> In fact 48GB is a way too much for WAL drive - usually the write
>> ahead
>> >> log
>> >>>> tend to be 2-4 GBs.
>> >>>>
>> >>>> But in your case it's ~150GB, while DB itself is very small
>> (146MB!!!):
>> >>>>
>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>> >>>> 154 GiB 2400
>> >>>>
>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>> >>>> 146 MiB 30
>> >>>>
>> >>>>
>> >>>> which means that there are some issues with RocksDB's WAL processing,
>> >>>> which needs some troubleshooting...
>> >>>>
>> >>>> Curious if other OSDs are suffering from the same and whether you
>> have
>> >> any
>> >>>> custom settings for your OSD(s)?
>> >>>>
>> >>>> Additionally you might want to try the following command to compact
>> this
>> >>>> specific OSD manually and check if this would normalize the DB
>> layout -
>> >> the
>> >>>> majority of data has to be at DB level not WAL. Please share the
>> >> resulting
>> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command) after
>> the
>> >>>> compaction is fulfiled and OSD is restarted.
>> >>>>
>> >>>> The compaction command to be applied on an offline OSD:
>> >> "ceph-kvstore-tool
>> >>>> bluestore-kv <path-to-osd> compact"
>> >>>>
>> >>>> Even if the above works great please refrain from applying that
>> >> compaction
>> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would WAL
>> grow
>> >>>> again or not?
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Igor
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote:
>> >>>>
>> >>>> Hello Igor,
>> >>>>
>> >>>> yes the volume is nvme wal partitions for the bluestore devicegroups
>> >> are
>> >>>> only 48gb each
>> >>>>
>> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb
>> (WAL)
>> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb
>> >> (rock.db)
>> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db)
>> >>>>
>> >>>> each blustore have 1 partition nvme,ssd and hdd like described in the
>> >>>> documentation
>> >>>>
>> >>
>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
>> >>>> is this to small or can i adjust the max allocation on the wal nvme
>> >> device
>> >>>> in the ceph configuration?
>> >>>> i know that the ssd and nvme are to small for those 5.5tb disk...
>> its 1%
>> >>>> only ot the rotation disk.
>> >>>> i am new in ceph and still or always learning, but we are in a little
>> >>>> hurry because our other datastores are old and full.
>> >>>>
>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device
>> >> info
>> >>>> {
>> >>>> "dev": {
>> >>>> "device": "BDEV_WAL",
>> >>>> "total": 48318377984,
>> >>>> "free": 1044480,
>> >>>> "bluefs_used": 48317333504
>> >>>> },
>> >>>> "dev": {
>> >>>> "device": "BDEV_DB",
>> >>>> "total": 187904811008,
>> >>>> "free": 68757217280,
>> >>>> "bluefs_used": 119147593728
>> >>>> },
>> >>>> "dev": {
>> >>>> "device": "BDEV_SLOW",
>> >>>> "total": 6001172414464,
>> >>>> "free": 5624912359424,
>> >>>> "bluefs_used": 0,
>> >>>> "bluefs max available": 5624401231872
>> >>>> }
>> >>>> }
>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
>> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
>> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB)
>> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB)
>> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>> >> db_total:178509578240,
>> >>>> slow_total:5701113793740, db_avail:103884521472
>> >>>> Usage matrix:
>> >>>> DEV/LEV WAL DB SLOW * *
>> >>>> REAL FILES
>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>> >>>> 7.5 MiB 1
>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>> >>>> 154 GiB 2400
>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>> >>>> 146 MiB 30
>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>> >> 0
>> >>>> B 0
>> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B
>> >> 0
>> >>>> B 2431
>> >>>> MAXIMUMS:
>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>> >> 17
>> >>>> MiB
>> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>> >>>> 192 GiB
>> >>>> DB 0 B 762 MiB 0 B 0 B 0 B
>> >>>> 741 MiB
>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>> >> 0 B
>> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B
>> >> 0 B
>> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <
>> >> ifedotov@xxxxxxx>:
>> >>>>> Hey Jose,
>> >>>>>
>> >>>>> it looks like your WAL volume is out of space which looks weird
>> given
>> >>>>> its capacity = 48Gb.
>> >>>>>
>> >>>>> Could you please share the output of the following commands:
>> >>>>>
>> >>>>> ceph daemon osd.N bluestore bluefs device info
>> >>>>>
>> >>>>> ceph daemon osd.N bluefs stats
>> >>>>>
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Igor
>> >>>>>
>> >>>>>
>> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote:
>> >>>>>> Hello together
>> >>>>>>
>> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this
>> >> messages
>> >>>>> in
>> >>>>>> the service logs of the osd daemons.
>> >>>>>>
>> >>>>>> we have three osd nodes .. each node has 20osds as bluestore with
>> >>>>>> nvme/ssd/hdd
>> >>>>>>
>> >>>>>> is this a bug or maybe i have some settings wrong?
>> >>>>>>
>> >>>>>>
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New
>> >>>>> memtable
>> >>>>>> created with log file: #9588. Immutable memtables: 1.
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>> bdev
>> >> 0,
>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>> >> 0xb40000000,
>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.547575)
>> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling
>> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots
>> >> available
>> >>>>> 1,
>> >>>>>> compaction slots available 1, flush slots scheduled 1, compaction
>> >> slots
>> >>>>>> scheduled 0
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
>> >>>>>> memtable with next log file: 9587
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
>> >>>>>> memtable with next log file: 9588
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850547916,
>> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2,
>> >>>>> "num_entries":
>> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926,
>> "memory_usage":
>> >>>>>> 130479920, "flush_reason": "Write Buffer Full"}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] Level-0
>> >> flush
>> >>>>>> table #9589: started
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850559292,
>> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation",
>> >>>>> "file_number":
>> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size":
>> 3247855,
>> >>>>>> "index_size": 1031, "index_partitions": 0, "top_level_index_size":
>> 0,
>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>> >>>>>> "filter_size": 197, "raw_key_size": 1088, "raw_average_key_size":
>> 16,
>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739,
>> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32,
>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>> "format_version":
>> >> 0,
>> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>> >>>>>> "compression": "NoCompression", "compression_options":
>> >> "window_bits=-14;
>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
>> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time":
>> >>>>> 1633511730,
>> >>>>>> "file_creation_time": 1633511850}}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] Level-0
>> >> flush
>> >>>>>> table #9589: 3249934 bytes OK
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.559362)
>> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.559583)
>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #1
>> >> done
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.559586)
>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #2
>> >> done
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.559601)
>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709,
>> "event":
>> >>>>>> "flush_finished", "output_compression": "NoCompression",
>> "lsm_state":
>> >>>>> [8,
>> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.559638)
>> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary:
>> files[8 1
>> >>>>> 0 0
>> >>>>>> 0 0 0] max score 1.00
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] [L]
>> [JOB
>> >>>>> 5710]
>> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L]
>> >>>>> Compaction
>> >>>>>> start summary: Base version 3090 Base level 0, inputs:
>> [9589(3173KB)
>> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB)
>> >>>>> 9570(405KB)
>> >>>>>> 9567(29KB)], [9564(1115KB)]
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850559956,
>> >>>>>> "job": 5710, "event": "compaction_started", "compaction_reason":
>> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, 9573,
>> >>>>> 9570,
>> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size": 19542092}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] [L]
>> [JOB
>> >>>>> 5710]
>> >>>>>> Generated table #9590: 36 keys, 3249524 bytes
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850582987,
>> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation",
>> >>>>> "file_number":
>> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size":
>> 3247449,
>> >>>>>> "index_size": 1031, "index_partitions": 0, "top_level_index_size":
>> 0,
>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>> >>>>>> "filter_size": 197, "raw_key_size": 576, "raw_average_key_size":
>> 16,
>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173,
>> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0,
>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>> "format_version":
>> >> 0,
>> >>>>>> "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>> >>>>>> "compression": "NoCompression", "compression_options":
>> >> "window_bits=-14;
>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
>> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0,
>> >>>>>> "file_creation_time": 1633511850}}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] [L]
>> [JOB
>> >>>>> 5710]
>> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.583469)
>> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0 0
>> 0 0
>> >>>>> 0]
>> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8, 1)
>> >>>>> out(1)
>> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2)
>> write-amplify(0.2)
>> >>>>> OK,
>> >>>>>> records in: 376, records dropped: 340 output_compression:
>> >> NoCompression
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>> >> 2021/10/06-09:17:30.583498)
>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710,
>> "event":
>> >>>>>> "compaction_finished", "compaction_time_micros": 23098,
>> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1,
>> >>>>> "num_output_files":
>> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376,
>> >>>>>> "num_output_records": 36, "num_subcompactions": 1,
>> >> "output_compression":
>> >>>>>> "NoCompression", "num_single_delete_mismatches": 0,
>> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0,
>> >> 0]}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583615,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9589}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583648,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9586}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583675,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9583}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583709,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9580}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583739,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9576}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583769,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9573}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583804,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9570}
>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>> >> 1633511850583835,
>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9567}
>> >>>>>> _______________________________________________
>> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>> --
>>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
