Hello Igor, yes the volume is nvme wal partitions for the bluestore devicegroups are only 48gb each on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb (WAL) on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb (rock.db) on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db) each blustore have 1 partition nvme,ssd and hdd like described in the documentation https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/ is this to small or can i adjust the max allocation on the wal nvme device in the ceph configuration? i know that the ssd and nvme are to small for those 5.5tb disk... its 1% only ot the rotation disk. i am new in ceph and still or always learning, but we are in a little hurry because our other datastores are old and full. root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info { "dev": { "device": "BDEV_WAL", "total": 48318377984, "free": 1044480, "bluefs_used": 48317333504 }, "dev": { "device": "BDEV_DB", "total": 187904811008, "free": 68757217280, "bluefs_used": 119147593728 }, "dev": { "device": "BDEV_SLOW", "total": 6001172414464, "free": 5624912359424, "bluefs_used": 0, "bluefs max available": 5624401231872 } } root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB) 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB) 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB) RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 Usage matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 124 MiB 2.3 GiB 0 B 0 B 0 B 7.5 MiB 1 WAL 45 GiB 111 GiB 0 B 0 B 0 B 154 GiB 2400 DB 0 B 164 MiB 0 B 0 B 0 B 146 MiB 30 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTALS 45 GiB 113 GiB 0 B 0 B 0 B 0 B 2431 MAXIMUMS: LOG 124 MiB 2.3 GiB 0 B 0 B 0 B 17 MiB WAL 45 GiB 149 GiB 0 B 0 B 0 B 192 GiB DB 0 B 762 MiB 0 B 0 B 0 B 741 MiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTALS 45 GiB 150 GiB 0 B 0 B 0 B 0 B Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <ifedotov@xxxxxxx>: > Hey Jose, > > it looks like your WAL volume is out of space which looks weird given > its capacity = 48Gb. > > Could you please share the output of the following commands: > > ceph daemon osd.N bluestore bluefs device info > > ceph daemon osd.N bluefs stats > > > Thanks, > > Igor > > > On 10/6/2021 12:24 PM, José H. Freidhof wrote: > > Hello together > > > > we have a running ceph pacific 16.2.5 cluster and i found this messages > in > > the service logs of the osd daemons. > > > > we have three osd nodes .. each node has 20osds as bluestore with > > nvme/ssd/hdd > > > > is this a bug or maybe i have some settings wrong? > > > > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New memtable > > created with log file: #9588. Immutable memtables: 1. > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on bdev 0, > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.547575) > > [db_impl/db_impl_compaction_flush.cc:2198] Calling > > FlushMemTableToOutputFile with column family [L], flush slots available > 1, > > compaction slots available 1, flush slots scheduled 1, compaction slots > > scheduled 0 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing > > memtable with next log file: 9587 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing > > memtable with next log file: 9588 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850547916, > > "job": 5709, "event": "flush_started", "num_memtables": 2, "num_entries": > > 4146, "num_deletes": 0, "total_data_size": 127203926, "memory_usage": > > 130479920, "flush_reason": "Write Buffer Full"} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 > > 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] Level-0 flush > > table #9589: started > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850559292, > > "cf_name": "L", "job": 5709, "event": "table_file_creation", > "file_number": > > 9589, "file_size": 3249934, "table_properties": {"data_size": 3247855, > > "index_size": 1031, "index_partitions": 0, "top_level_index_size": 0, > > "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, > > "filter_size": 197, "raw_key_size": 1088, "raw_average_key_size": 16, > > "raw_value_size": 3246252, "raw_average_value_size": 47739, > > "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32, > > "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, > > "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", > > "column_family_name": "L", "column_family_id": 10, "comparator": > > "leveldb.BytewiseComparator", "merge_operator": "nullptr", > > "prefix_extractor_name": "nullptr", "property_collectors": "[]", > > "compression": "NoCompression", "compression_options": "window_bits=-14; > > level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; > > enabled=0; ", "creation_time": 1633511730, "oldest_key_time": 1633511730, > > "file_creation_time": 1633511850}} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] Level-0 flush > > table #9589: 3249934 bytes OK > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559362) > > [memtable_list.cc:447] [L] Level-0 commit table #9589 started > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559583) > > [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #1 done > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559586) > > [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #2 done > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559601) > > EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709, "event": > > "flush_finished", "output_compression": "NoCompression", "lsm_state": [8, > > 1, 0, 0, 0, 0, 0], "immutable_memtables": 0} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f3905c02700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559638) > > [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary: files[8 1 0 > 0 > > 0 0 0] max score 1.00 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] [L] [JOB > 5710] > > Compacting 8@0 + 1@1 files to L1, score 1.00 > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L] > Compaction > > start summary: Base version 3090 Base level 0, inputs: [9589(3173KB) > > 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB) > 9570(405KB) > > 9567(29KB)], [9564(1115KB)] > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850559956, > > "job": 5710, "event": "compaction_started", "compaction_reason": > > "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, 9573, 9570, > > 9567], "files_L1": [9564], "score": 1, "input_data_size": 19542092} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] [L] [JOB > 5710] > > Generated table #9590: 36 keys, 3249524 bytes > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850582987, > > "cf_name": "L", "job": 5710, "event": "table_file_creation", > "file_number": > > 9590, "file_size": 3249524, "table_properties": {"data_size": 3247449, > > "index_size": 1031, "index_partitions": 0, "top_level_index_size": 0, > > "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, > > "filter_size": 197, "raw_key_size": 576, "raw_average_key_size": 16, > > "raw_value_size": 3246252, "raw_average_value_size": 90173, > > "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0, > > "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, > > "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", > > "column_family_name": "L", "column_family_id": 10, "comparator": > > "leveldb.BytewiseComparator", "merge_operator": "nullptr", > > "prefix_extractor_name": "nullptr", "property_collectors": "[]", > > "compression": "NoCompression", "compression_options": "window_bits=-14; > > level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; > > enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0, > > "file_creation_time": 1633511850}} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] [L] [JOB > 5710] > > Compacted 8@0 + 1@1 files to L1 => 3249524 bytes > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.583469) > > [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0 0 0 0 0] > > max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8, 1) > out(1) > > MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2) write-amplify(0.2) OK, > > records in: 376, records dropped: 340 output_compression: NoCompression > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: (Original Log Time 2021/10/06-09:17:30.583498) > > EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710, "event": > > "compaction_finished", "compaction_time_micros": 23098, > > "compaction_time_cpu_micros": 20039, "output_level": 1, > "num_output_files": > > 1, "total_output_size": 3249524, "num_input_records": 376, > > "num_output_records": 36, "num_subcompactions": 1, "output_compression": > > "NoCompression", "num_single_delete_mismatches": 0, > > "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583615, > > "job": 5710, "event": "table_file_deletion", "file_number": 9589} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583648, > > "job": 5710, "event": "table_file_deletion", "file_number": 9586} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583675, > > "job": 5710, "event": "table_file_deletion", "file_number": 9583} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583709, > > "job": 5710, "event": "table_file_deletion", "file_number": 9580} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583739, > > "job": 5710, "event": "table_file_deletion", "file_number": 9576} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583769, > > "job": 5710, "event": "table_file_deletion", "file_number": 9573} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583804, > > "job": 5710, "event": "table_file_deletion", "file_number": 9570} > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 > > 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583835, > > "job": 5710, "event": "table_file_deletion", "file_number": 9567} > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Mit freundlichen Grüßen, - José H. Freidhof Reyerhütterstrasse 130b 41065 Mönchengladbach eMail: harald.freidhof@xxxxxxxxx mobil: +49 (0) 1523 – 717 7801 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |