Good morning,
i checked today the osd.8 and the log shows
again the same error
bluefs _allocate unable to allocate 0x100000
on bdev 0, allocator name bluefs-wal,
allocator type hybrid, capacity 0xb40000000,
block size 0x100000, free 0xff000,
fragmentation 0, allocated 0x0
any idea why that could be?
Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José
H. Freidhof <harald.freidhof@xxxxxxxxxxxxxx
<mailto:harald.freidhof@xxxxxxxxxxxxxx>>:
Hi Igor,
today i repaired one osd node and all
osd´s on the node, creating them new again....
after that i waited for the
rebalance/recovery process and the cluster
was healthy after some hours..
i notices that the osd.2 does not have any
more this error in the log.
but i noticed it now on the same node on
osd.8... so i did the test that you
suggested on osd.8
it took nearly 20minutes to compact those
db´s on the bluestore, but it helped...
the problem on osd.8 is gone...
*so the problem that i have with the alloc
on the wal device seems to be random on
different nodes and osd´s and looks like
it comes, stay a while and disappears
after a longer while...
*
here are the results that you suggested:
root@cd88-ceph-osdh-01:/# ceph daemon
osd.8 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 1044480,
"bluefs_used": 48317333504
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 79842762752,
"bluefs_used": 108062048256
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5510727389184,
"bluefs_used": 0,
"bluefs max available": 5508815847424
}
}
root@cd88-ceph-osdh-01:/# ceph daemon
osd.8 bluefs stats
0 : device size 0xb3ffff000 : using
0xb3ff00000(45 GiB)
1 : device size 0x2bbfffe000 : using
0x1931500000(101 GiB)
2 : device size 0x57541c00000 : using
0x7235e3e000(457 GiB)
RocksDBBlueFSVolumeSelector:
wal_total:45902462976,
db_total:178509578240,
slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW
* * REAL FILES
LOG 304 MiB 7.9 GiB 0 B
0 B 0 B 9.7 MiB 1
WAL 45 GiB 100 GiB 0 B
0 B 0 B 144 GiB 2319
DB 0 B 276 MiB 0 B
0 B 0 B 249 MiB 47
SLOW 0 B 0 B 0 B
0 B 0 B 0 B 0
TOTALS 45 GiB 109 GiB 0 B
0 B 0 B 0 B 2367
MAXIMUMS:
LOG 304 MiB 7.9 GiB 0 B 0 B
0 B 20 MiB
WAL 45 GiB 149 GiB 0 B
0 B 0 B 192 GiB
DB 0 B 762 MiB 0 B
0 B 0 B 738 MiB
SLOW 0 B 0 B 0 B
0 B 0 B 0 B
TOTALS 45 GiB 150 GiB 0 B
0 B 0 B 0 B
---
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.464+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x400000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.472+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.480+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.500+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.576+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.624+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.636+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.884+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.968+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700
4 rocksdb:
[db_impl/db_impl_write.cc:1668] [L] New
memtable created with log file: #13656.
Immutable memtables: 1.
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.992+0000 7f4a9483a700
1 bluefs _allocate unable to allocate
0x100000 on bdev 0, allocator name
bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000,
free 0xff000, fragmentation 0, allocated 0x0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700
4 rocksdb: (Original Log Time
2021/10/06-19:51:34.996331)
[db_impl/db_impl_compaction_flush.cc:2198]
Calling FlushMemTableToOutputFile with
column family [L], flush slots available
1, compaction slots available 1, flush
slots scheduled 1, compaction slots
scheduled 0
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700
4 rocksdb: [flush_job.cc:321] [L] [JOB
8859] Flushing memtable with next log
file: 13655
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700
4 rocksdb: [flush_job.cc:321] [L] [JOB
8859] Flushing memtable with next log
file: 13656
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700
4 rocksdb: EVENT_LOG_v1 {"time_micros":
1633549894998273, "job": 8859, "event":
"flush_started", "num_memtables": 2,
"num_entries": 3662, "num_deletes": 0,
"total_data_size": 130482337,
"memory_usage": 132976224, "flush_reason":
"Write Buffer Full"}
Oct 06 21:51:34 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:34.996+0000 7f4aab067700
4 rocksdb: [flush_job.cc:350] [L] [JOB
8859] Level-0 flush table #13657: started
Oct 06 21:51:35 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700
4 rocksdb: EVENT_LOG_v1 {"time_micros":
1633549895008271, "cf_name": "L", "job":
8859, "event": "table_file_creation",
"file_number": 13657, "file_size":
2952537, "table_properties": {"data_size":
2951222, "index_size": 267,
"index_partitions": 0,
"top_level_index_size": 0,
"index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0,
"filter_size": 197, "raw_key_size": 1120,
"raw_average_key_size": 16,
"raw_value_size": 2950151,
"raw_average_value_size": 42145,
"num_data_blocks": 9, "num_entries": 70,
"num_deletions": 61, "num_merge_operands":
0, "num_range_deletions": 0,
"format_version": 0, "fixed_key_len": 0,
"filter_policy":
"rocksdb.BuiltinBloomFilter",
"column_family_name": "L",
"column_family_id": 10, "comparator":
"leveldb.BytewiseComparator",
"merge_operator": "nullptr",
"prefix_extractor_name": "nullptr",
"property_collectors": "[]",
"compression": "NoCompression",
"compression_options": "window_bits=-14;
level=32767; strategy=0; max_dict_bytes=0;
zstd_max_train_bytes=0; enabled=0; ",
"creation_time": 1633549854,
"oldest_key_time": 1633549854,
"file_creation_time": 1633549894}}
Oct 06 21:51:35 cd88-ceph-osdh-01
bash[6328]: debug
2021-10-06T19:51:35.004+0000 7f4aab067700
4 rocksdb: [flush_job.cc:401] [L] [JOB
8859] Level-0 flush table #13657: 2952537
bytes OK
---
root@cd88-ceph-osdh-01:~# ceph osd set noout
root@cd88-ceph-osdh-01:~# ceph orch daemon
stop osd.8
root@cd88-ceph-osdh-01:~# ceph orch ps
...
osd.7 cd133-ceph-osdh-01 running (4h)
44s ago - 2738M 5528M 16.2.5
6933c2a0b7dd 8a98ae61f0eb
osd.8 cd88-ceph-osdh-01 stopped
5s ago - - 5528M <unknown>
<unknown> <unknown>
osd.9 cd133k-ceph-osdh-01 running (3d)
5m ago - 4673M 8966M 16.2.5
6933c2a0b7dd 0ff7584b1808
...
---
root@cd88-ceph-osdh-01:~#
ceph-kvstore-tool bluestore-kv
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/
compact
2021-10-06T21:53:50.559+0200 7f87bde3c240
0
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db_and_around read-only:0 repair:0
2021-10-06T21:53:50.559+0200 7f87bde3c240
1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000,
5.5 TiB) block_size 4096 (4 KiB)
rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_set_cache_sizes cache_size 1073741824
meta 0.45 kv 0.45 data 0.06
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open size 187904819200 (0x2bc0000000, 175
GiB) block_size 4096 (4 KiB) rotational
discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs add_block_device bdev 1 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000,
5.5 TiB) block_size 4096 (4 KiB)
rotational discard not supported
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs add_block_device bdev 2 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
size 5.5 TiB
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open size 48318382080 (0xb40000000, 45
GiB) block_size 4096 (4 KiB)
non-rotational discard supported
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs add_block_device bdev 0 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs mount
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs _init_alloc new, id 0, allocator
name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs _init_alloc new, id 1, allocator
name bluefs-db, allocator type hybrid,
capacity 0x2bc0000000, block size 0x100000
2021-10-06T21:53:50.563+0200 7f87bde3c240
1 bluefs _init_alloc shared, id 2,
capacity 0x57541c00000, block size 0x10000
2021-10-06T21:53:50.655+0200 7f87bde3c240
1 bluefs mount shared_bdev_used = 0
2021-10-06T21:53:50.655+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to
db,178509578240 db.slow,5701113793740
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old nid_max 167450
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta old blobid_max 30720
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta freelist_type bitmap
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta ondisk_format 4
compat_ondisk_format 3
2021-10-06T22:01:32.715+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_super_meta min_alloc_size 0x1000
2021-10-06T22:01:33.347+0200 7f87bde3c240
1 freelist init
2021-10-06T22:01:33.347+0200 7f87bde3c240
1 freelist _read_cfg
2021-10-06T22:01:33.347+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc opening allocation metadata
2021-10-06T22:01:41.031+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_init_alloc loaded 5.0 TiB in 37191
extents, allocator type hybrid, capacity
0x57541c00000, block size 0x1000, free
0x502f8f9a000, fragmentation 2.76445e-05
2021-10-06T22:01:41.039+0200 7f87bde3c240
1 bluefs umount
2021-10-06T22:01:41.043+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
close
2021-10-06T22:01:43.623+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
close
2021-10-06T22:01:54.727+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
open size 187904819200 (0x2bc0000000, 175
GiB) block_size 4096 (4 KiB) rotational
discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs add_block_device bdev 1 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
size 175 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
open size 6001172414464 (0x57541c00000,
5.5 TiB) block_size 4096 (4 KiB)
rotational discard not supported
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs add_block_device bdev 2 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
size 5.5 TiB
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
open size 48318382080 (0xb40000000, 45
GiB) block_size 4096 (4 KiB)
non-rotational discard supported
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs add_block_device bdev 0 path
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
size 45 GiB
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs mount
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs _init_alloc new, id 0, allocator
name bluefs-wal, allocator type hybrid,
capacity 0xb40000000, block size 0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs _init_alloc new, id 1, allocator
name bluefs-db, allocator type hybrid,
capacity 0x2bc0000000, block size 0x100000
2021-10-06T22:01:54.995+0200 7f87bde3c240
1 bluefs _init_alloc shared, id 2,
capacity 0x57541c00000, block size 0x10000
2021-10-06T22:01:55.079+0200 7f87bde3c240
1 bluefs mount shared_bdev_used = 0
2021-10-06T22:01:55.079+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_prepare_db_environment set db_paths to
db,178509578240 db.slow,5701113793740
2021-10-06T22:09:36.519+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
_open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
2021-10-06T22:09:54.067+0200 7f87bde3c240
1
bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
umount
2021-10-06T22:09:54.079+0200 7f87bde3c240
1 bluefs umount
2021-10-06T22:09:54.079+0200 7f87bde3c240
1 bdev(0x5644f056d400
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal)
close
2021-10-06T22:09:56.612+0200 7f87bde3c240
1 bdev(0x5644f056d000
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db)
close
2021-10-06T22:10:07.520+0200 7f87bde3c240
1 bdev(0x5644f056cc00
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
2021-10-06T22:10:07.688+0200 7f87bde3c240
1 freelist shutdown
2021-10-06T22:10:07.692+0200 7f87bde3c240
1 bdev(0x5644f056c800
/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block)
close
---
root@cd88-ceph-osdh-01:~# ceph orch daemon
start osd.8
---
root@cd88-ceph-osdh-01:/# ceph -s
cluster:
id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
health: HEALTH_OK
services:
mon: 3 daemons, quorum
cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01
(age 15h)
mgr: cd133-ceph-mon-01.mzapob(active,
since 15h), standbys:
cd133k-ceph-mon-01.imikwh
osd: 60 osds: 60 up (since 2m), 60
in (since 3h)
rgw: 4 daemons active (2 hosts, 1 zones)
tcmu-runner: 10 portals active (2 hosts)
data:
pools: 6 pools, 361 pgs
objects: 2.46M objects, 8.0 TiB
usage: 33 TiB used, 304 TiB / 338 TiB
avail
pgs: 361 active+clean
io:
client: 45 MiB/s rd, 50 MiB/s wr, 921
op/s rd, 674 op/s wr
---
root@cd88-ceph-osdh-01:/# ceph daemon
osd.8 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
* "free": 41354784768,
* "bluefs_used": 6963593216
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 187302928384,
"bluefs_used": 601882624
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5507531620352,
"bluefs_used": 0,
"bluefs max available": 5505566572544
}
}
---
root@cd88-ceph-osdh-01:/# ceph daemon
osd.8 bluefs stats
0 : device size 0xb3ffff000 : using
0x1a0c00000(6.5 GiB)
1 : device size 0x2bbfffe000 : using
0x23e00000(574 MiB)
2 : device size 0x57541c00000 : using
0x72f0803000(460 GiB)
RocksDBBlueFSVolumeSelector:
wal_total:45902462976,
db_total:178509578240,
slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW
* * REAL FILES
LOG 12 MiB 18 MiB 0 B 0 B
0 B 10 MiB 0
WAL 6.5 GiB 0 B 0 B 0
B 0 B 6.4 GiB 102
DB 0 B 573 MiB 0 B
0 B 0 B 557 MiB 22
SLOW 0 B 0 B 0 B
0 B 0 B 0 B 0
TOTALS 6.5 GiB 591 MiB 0 B 0
B 0 B 0 B 125
MAXIMUMS:
LOG 12 MiB 18 MiB 0 B 0 B
0 B 17 MiB
WAL 45 GiB 101 GiB 0 B
0 B 0 B 145 GiB
DB 0 B 688 MiB 0 B
0 B 0 B 670 MiB
SLOW 0 B 0 B 0 B
0 B 0 B 0 B
TOTALS 45 GiB 101 GiB 0 B
0 B 0 B 0 B
----
Here are the osd.2... the problem
disapeared from alone
very strange...
root@cd88-ceph-osdh-01:/# ceph daemon
osd.2 bluefs stats
0 : device size 0xb3ffff000 : using
0x7bcc00000(31 GiB)
1 : device size 0x2bbfffe000 : using
0x458c00000(17 GiB)
2 : device size 0x57541c00000 : using
0x5cd3665000(371 GiB)
RocksDBBlueFSVolumeSelector:
wal_total:45902462976,
db_total:178509578240,
slow_total:5701113793740,
db_avail:103884521472
Usage matrix:
DEV/LEV WAL DB SLOW
* * REAL FILES
LOG 920 MiB 4.0 GiB 0 B 0 B
0 B 10 MiB 1
WAL 31 GiB 17 GiB 0 B
0 B 0 B 48 GiB 765
DB 0 B 193 MiB 0 B
0 B 0 B 175 MiB 30
SLOW 0 B 0 B 0 B
0 B 0 B 0 B 0
TOTALS 32 GiB 21 GiB 0 B
0 B 0 B 0 B 796
MAXIMUMS:
LOG 920 MiB 4.0 GiB 0 B 0 B
0 B 17 MiB
WAL 45 GiB 149 GiB 0 B
0 B 0 B 192 GiB
DB 0 B 762 MiB 0 B
0 B 0 B 741 MiB
SLOW 0 B 0 B 0 B
0 B 0 B 0 B
TOTALS 45 GiB 153 GiB 0 B
0 B 0 B 0 B
root@cd88-ceph-osdh-01:/# ceph daemon
osd.2 bluestore bluefs device info
{
"dev": {
"device": "BDEV_WAL",
"total": 48318377984,
"free": 15043915776,
"bluefs_used": 33274462208
},
"dev": {
"device": "BDEV_DB",
"total": 187904811008,
"free": 169235963904,
"bluefs_used": 18668847104
},
"dev": {
"device": "BDEV_SLOW",
"total": 6001172414464,
"free": 5602453327872,
"bluefs_used": 0,
"bluefs max available": 5600865222656
}
}
Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb
Igor Fedotov <igor.fedotov@xxxxxxxx
<mailto:igor.fedotov@xxxxxxxx>>:
On 10/6/2021 4:25 PM, José H. Freidhof
wrote:
> hi,
>
> no risk no fun 😂 okay
> I have reset the settings you
mentioned to standard.
>
> what you exactly mean with taking
offline the osd? ceph orch daemon stop
> osd.2? or mark down?
"daemon stop" is enough. You might
want to set noout flag before that
though...
>
> for the command which path i use?
you mean:
>
> bluestore-kv
/var/lib/ceph/$fsid/osd.2 compact???
yep
>
>
> Igor Fedotov <ifedotov@xxxxxxx
<mailto:ifedotov@xxxxxxx>> schrieb am
Mi., 6. Okt. 2021, 13:33:
>
>> On 10/6/2021 2:16 PM, José H.
Freidhof wrote:
>>> Hi Igor,
>>>
>>> yes i have some osd settings set
:-) here are my ceph config dump. those
>>> settings are from a redhat
document for bluestore devices
>>> maybe it is that setting causing
this problem? "advanced
>>> mon_compact_on_trim false"???
>> OMG!!!
>>
>> No - mon_compact_on_trim has
nothing to deal with bluestore.
>>
>> Highly likely it's
bluestore_rocksdb_options which hurts...
>> Documentations tend to fall behind
the best practices.... I would
>> strongly discourage you from using
non-default settings unless it's
>> absolutely clear why this is necessary.
>>
>> Even at the first glance the
following settings (just a few ones I'm
>> completely aware) are
suboptimal/non-recommended:
>>
>> rocksdb_perf
>>
>> bluefs_sync_write
>>
>> bluefs_csum_type
>>
>>
>> Not to mention
bluestore_rocksdb_options which hasn't
got much adoption
>> so far and apparently greatly
alters rocksdb behavior...
>>
>>
>> So I would suggest to revert
rocksdb options back to default, run the
>> compaction and if it succeeds
monitor the OSD for a while. Then if it
>> works fine - apply the same for others
>>
>>
>> Hope this helps,
>>
>> Igor
>>
>>
>>
>>> i will test it this afternoon...
at the moment are everything semi
>>> prodcuctive and i need to repair
one osd node.. because i think of this
>>> reason the osds crashed on the
node and the osd container crashes with a
>>> dump while coming up now.
>>> need first to replicate all
between all three nodes and then i can
take
>>> offline the osd.2.and test your
command. i will inform you later...
>>>
>>> root@cd88-ceph-osdh-01:/# ceph
config dump
>>> WHO MASK
>>> LEVEL OPTION VALUE
>>>
>>>
>>>
>>>
>>>
>>> RO
>>> global
>>> advanced leveldb_max_open_files
131072
>>>
>>>
>>>
>>>
>>> global
>>> advanced mon_compact_on_trim false
>>>
>>>
>>>
>>>
>>> global
>>> dev ms_crc_data false
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_deep_scrub_interval
1209600.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_max_scrubs 16
>>>
>>>
>>>
>>>
>>> global
>>> advanced
osd_scrub_load_threshold 0.010000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_scrub_max_interval
1209600.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced osd_scrub_min_interval
86400.000000
>>>
>>>
>>>
>>>
>>> global
>>> advanced perf true
>>>
>>>
>>>
>>>
>>> global
>>> advanced
rbd_readahead_disable_after_bytes
0
>>>
>>>
>>>
>>>
>>> global
>>> advanced rbd_readahead_max_bytes
4194304
>>>
>>>
>>>
>>>
>>> global
>>> advanced rocksdb_perf true
>>>
>>>
>>>
>>>
>>> global
>>> advanced throttler_perf_counter
false
>>>
>>>
>>>
>>>
>>> mon
>>> advanced
auth_allow_insecure_global_id_reclaim
false
>>>
>>>
>>>
>>>
>>> mon
>>> advanced cluster_network
10.50.50.0/24 <http://10.50.50.0/24>
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mon
>>> advanced mon_osd_down_out_interval
300
>>>
>>>
>>>
>>>
>>> mon
>>> advanced public_network
10.50.50.0/24 <http://10.50.50.0/24>
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced
mgr/cephadm/container_init True
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced
mgr/cephadm/device_enhanced_scan true
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced
mgr/cephadm/migration_current 2
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced
mgr/cephadm/warn_on_stray_daemons false
>>>
>>>
>>>
>>>
>>>
>>> *
>>> mgr
>>> advanced
mgr/cephadm/warn_on_stray_hosts false
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> advanced bluefs_sync_write true
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_autotune
true
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_kv_ratio
0.200000
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_meta_ratio
0.800000
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_size 2147483648
>>>
>>>
>>>
>>>
>>> osd
>>> dev bluestore_cache_size_hdd
2147483648
>>>
>>>
>>>
>>>
>>> osd
>>> advanced bluestore_csum_type
none
>>>
>>>
>>>
>>>
>>> osd
>>> dev
bluestore_extent_map_shard_max_size
200
>>>
>>>
>>>
>>>
>>> osd
>>> dev
bluestore_extent_map_shard_min_size
50
>>>
>>>
>>>
>>>
>>> osd
>>> dev
bluestore_extent_map_shard_target_size
100
>>>
>>>
>>>
>>>
>>> osd
>>> advanced bluestore_rocksdb_options
>>>
>>
compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>>> *
>>> osd
>>> advanced mon_osd_cache_size 1024
>>>
>>>
>>>
>>>
>>> osd
>>> dev ms_crc_data false
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_map_share_max_epochs
5
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_max_backfills
1
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_max_pg_log_entries 10
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_memory_cache_min 3000000000
>>>
>>>
>>>
>>>
>>> osd
>> host:cd133-ceph-osdh-01
>>> basic osd_memory_target 5797322383
>>>
>>>
>>>
>>>
>>> osd
>> host:cd133k-ceph-osdh-01
>>> basic osd_memory_target 9402402385
>>>
>>>
>>>
>>>
>>> osd host:cd88-ceph-osdh-01
>>> basic osd_memory_target 5797322096
>>>
>>>
>>>
>>>
>>> osd
>>> advanced
osd_memory_target_autotune true
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_min_pg_log_entries 10
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_op_num_shards
8
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> advanced
osd_op_num_threads_per_shard 2
>>>
>>>
>>>
>>>
>>>
>>> *
>>> osd
>>> dev osd_pg_log_dups_tracked 10
>>>
>>>
>>>
>>>
>>> osd
>>> dev osd_pg_log_trim_min 10
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_recovery_max_active
3
>>>
>>>
>>>
>>>
>>> osd
>>> advanced
osd_recovery_max_single_start 1
>>>
>>>
>>>
>>>
>>> osd
>>> advanced osd_recovery_sleep 0.000000
>>>
>>>
>>> Am Mi., 6. Okt. 2021 um 12:55 Uhr
schrieb Igor Fedotov <ifedotov@xxxxxxx
<mailto:ifedotov@xxxxxxx>
>>> :
>>>
>>>> Jose,
>>>>
>>>> In fact 48GB is a way too much
for WAL drive - usually the write ahead
>> log
>>>> tend to be 2-4 GBs.
>>>>
>>>> But in your case it's ~150GB,
while DB itself is very small (146MB!!!):
>>>>
>>>> WAL 45 GiB 111 GiB
0 B 0 B 0 B
>>>> 154 GiB 2400
>>>>
>>>> DB 0 B 164 MiB
0 B 0 B 0 B
>>>> 146 MiB 30
>>>>
>>>>
>>>> which means that there are some
issues with RocksDB's WAL processing,
>>>> which needs some troubleshooting...
>>>>
>>>> Curious if other OSDs are
suffering from the same and whether
you have
>> any
>>>> custom settings for your OSD(s)?
>>>>
>>>> Additionally you might want to
try the following command to compact this
>>>> specific OSD manually and check
if this would normalize the DB layout -
>> the
>>>> majority of data has to be at DB
level not WAL. Please share the
>> resulting
>>>> layout (reported by "ceph daemon
osd.2 bluefs stats" command) after the
>>>> compaction is fulfiled and OSD is
restarted.
>>>>
>>>> The compaction command to be
applied on an offline OSD:
>> "ceph-kvstore-tool
>>>> bluestore-kv <path-to-osd> compact"
>>>>
>>>> Even if the above works great
please refrain from applying that
>> compaction
>>>> to every OSD - let's see how that
"compacted" OSD evolves.Would WAL grow
>>>> again or not?
>>>>
>>>> Thanks,
>>>>
>>>> Igor
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/6/2021 1:35 PM, José H.
Freidhof wrote:
>>>>
>>>> Hello Igor,
>>>>
>>>> yes the volume is nvme wal
partitions for the bluestore devicegroups
>> are
>>>> only 48gb each
>>>>
>>>> on each osd node are 1 nvme with
1tb splitted in 20 lvs with 48gb (WAL)
>>>> on each osd node are 4 ssd with
1tb splitted in 5 lvs with 175gb
>> (rock.db)
>>>> on each osd node are 20 hdd with
5.5tb with 1 lvs (block.db)
>>>>
>>>> each blustore have 1 partition
nvme,ssd and hdd like described in the
>>>> documentation
>>>>
>>
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
<https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/>
>>>> is this to small or can i adjust
the max allocation on the wal nvme
>> device
>>>> in the ceph configuration?
>>>> i know that the ssd and nvme are
to small for those 5.5tb disk... its 1%
>>>> only ot the rotation disk.
>>>> i am new in ceph and still or
always learning, but we are in a little
>>>> hurry because our other
datastores are old and full.
>>>>
>>>> root@cd88-ceph-osdh-01:/# ceph
daemon osd.2 bluestore bluefs device
>> info
>>>> {
>>>> "dev": {
>>>> "device": "BDEV_WAL",
>>>> "total": 48318377984,
>>>> "free": 1044480,
>>>> "bluefs_used": 48317333504
>>>> },
>>>> "dev": {
>>>> "device": "BDEV_DB",
>>>> "total": 187904811008,
>>>> "free": 68757217280,
>>>> "bluefs_used": 119147593728
>>>> },
>>>> "dev": {
>>>> "device": "BDEV_SLOW",
>>>> "total": 6001172414464,
>>>> "free": 5624912359424,
>>>> "bluefs_used": 0,
>>>> "bluefs max available":
5624401231872
>>>> }
>>>> }
>>>> root@cd88-ceph-osdh-01:/# ceph
daemon osd.2 bluefs stats
>>>> 0 : device size 0xb3ffff000 :
using 0xb3ff00000(45 GiB)
>>>> 1 : device size 0x2bbfffe000 :
using 0x1bbeb00000(111 GiB)
>>>> 2 : device size 0x57541c00000 :
using 0x579b592000(350 GiB)
>>>> RocksDBBlueFSVolumeSelector:
wal_total:45902462976,
>> db_total:178509578240,
>>>> slow_total:5701113793740,
db_avail:103884521472
>>>> Usage matrix:
>>>> DEV/LEV WAL DB
SLOW * *
>>>> REAL FILES
>>>> LOG 124 MiB 2.3 GiB
0 B 0 B 0 B
>>>> 7.5 MiB 1
>>>> WAL 45 GiB 111 GiB
0 B 0 B 0 B
>>>> 154 GiB 2400
>>>> DB 0 B 164 MiB
0 B 0 B 0 B
>>>> 146 MiB 30
>>>> SLOW 0 B 0 B
0 B 0 B 0 B
>> 0
>>>> B 0
>>>> TOTALS 45 GiB 113 GiB
0 B 0 B 0 B
>> 0
>>>> B 2431
>>>> MAXIMUMS:
>>>> LOG 124 MiB 2.3 GiB
0 B 0 B 0 B
>> 17
>>>> MiB
>>>> WAL 45 GiB 149 GiB
0 B 0 B 0 B
>>>> 192 GiB
>>>> DB 0 B 762 MiB
0 B 0 B 0 B
>>>> 741 MiB
>>>> SLOW 0 B 0 B
0 B 0 B 0 B
>> 0 B
>>>> TOTALS 45 GiB 150 GiB
0 B 0 B 0 B
>> 0 B
>>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr
schrieb Igor Fedotov <
>> ifedotov@xxxxxxx
<mailto:ifedotov@xxxxxxx>>:
>>>>> Hey Jose,
>>>>>
>>>>> it looks like your WAL volume is
out of space which looks weird given
>>>>> its capacity = 48Gb.
>>>>>
>>>>> Could you please share the
output of the following commands:
>>>>>
>>>>> ceph daemon osd.N bluestore
bluefs device info
>>>>>
>>>>> ceph daemon osd.N bluefs stats
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Igor
>>>>>
>>>>>
>>>>> On 10/6/2021 12:24 PM, José H.
Freidhof wrote:
>>>>>> Hello together
>>>>>>
>>>>>> we have a running ceph pacific
16.2.5 cluster and i found this
>> messages
>>>>> in
>>>>>> the service logs of the osd
daemons.
>>>>>>
>>>>>> we have three osd nodes .. each
node has 20osds as bluestore with
>>>>>> nvme/ssd/hdd
>>>>>>
>>>>>> is this a bug or maybe i have
some settings wrong?
>>>>>>
>>>>>>
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:25.821+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:29.857+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.073+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x400000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.405+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.465+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.529+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f38eebd4700 4 rocksdb:
[db_impl/db_impl_write.cc:1668] [L] New
>>>>> memtable
>>>>>> created with log file: #9588.
Immutable memtables: 1.
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f38eebd4700 1 bluefs
_allocate unable to allocate 0x100000
on bdev
>> 0,
>>>>>> allocator name bluefs-wal,
allocator type hybrid, capacity
>> 0xb40000000,
>>>>>> block size 0x100000, free
0xff000, fragmentation 0, allocated 0x0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.547575)
>>>>>>
[db_impl/db_impl_compaction_flush.cc:2198]
Calling
>>>>>> FlushMemTableToOutputFile with
column family [L], flush slots
>> available
>>>>> 1,
>>>>>> compaction slots available 1,
flush slots scheduled 1, compaction
>> slots
>>>>>> scheduled 0
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb:
[flush_job.cc:321] [L] [JOB 5709] Flushing
>>>>>> memtable with next log file: 9587
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb:
[flush_job.cc:321] [L] [JOB 5709] Flushing
>>>>>> memtable with next log file: 9588
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850547916,
>>>>>> "job": 5709, "event":
"flush_started", "num_memtables": 2,
>>>>> "num_entries":
>>>>>> 4146, "num_deletes": 0,
"total_data_size": 127203926,
"memory_usage":
>>>>>> 130479920, "flush_reason":
"Write Buffer Full"}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.545+0000
>>>>>> 7f3905c02700 4 rocksdb:
[flush_job.cc:350] [L] [JOB 5709] Level-0
>> flush
>>>>>> table #9589: started
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850559292,
>>>>>> "cf_name": "L", "job": 5709,
"event": "table_file_creation",
>>>>> "file_number":
>>>>>> 9589, "file_size": 3249934,
"table_properties": {"data_size": 3247855,
>>>>>> "index_size": 1031,
"index_partitions": 0,
"top_level_index_size": 0,
>>>>>> "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0,
>>>>>> "filter_size": 197,
"raw_key_size": 1088,
"raw_average_key_size": 16,
>>>>>> "raw_value_size": 3246252,
"raw_average_value_size": 47739,
>>>>>> "num_data_blocks": 36,
"num_entries": 68, "num_deletions": 32,
>>>>>> "num_merge_operands": 0,
"num_range_deletions": 0,
"format_version":
>> 0,
>>>>>> "fixed_key_len": 0,
"filter_policy":
"rocksdb.BuiltinBloomFilter",
>>>>>> "column_family_name": "L",
"column_family_id": 10, "comparator":
>>>>>> "leveldb.BytewiseComparator",
"merge_operator": "nullptr",
>>>>>> "prefix_extractor_name":
"nullptr", "property_collectors": "[]",
>>>>>> "compression": "NoCompression",
"compression_options":
>> "window_bits=-14;
>>>>>> level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0;
>>>>>> enabled=0; ", "creation_time":
1633511730, "oldest_key_time":
>>>>> 1633511730,
>>>>>> "file_creation_time": 1633511850}}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
[flush_job.cc:401] [L] [JOB 5709] Level-0
>> flush
>>>>>> table #9589: 3249934 bytes OK
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.559362)
>>>>>> [memtable_list.cc:447] [L]
Level-0 commit table #9589 started
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.559583)
>>>>>> [memtable_list.cc:503] [L]
Level-0 commit table #9589: memtable #1
>> done
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.559586)
>>>>>> [memtable_list.cc:503] [L]
Level-0 commit table #9589: memtable #2
>> done
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.559601)
>>>>>> EVENT_LOG_v1 {"time_micros":
1633511850559593, "job": 5709, "event":
>>>>>> "flush_finished",
"output_compression": "NoCompression",
"lsm_state":
>>>>> [8,
>>>>>> 1, 0, 0, 0, 0, 0],
"immutable_memtables": 0}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f3905c02700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.559638)
>>>>>>
[db_impl/db_impl_compaction_flush.cc:205]
[L] Level summary: files[8 1
>>>>> 0 0
>>>>>> 0 0 0] max score 1.00
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1676]
[L] [JOB
>>>>> 5710]
>>>>>> Compacting 8@0 + 1@1 files to
L1, score 1.00
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1680] [L]
>>>>> Compaction
>>>>>> start summary: Base version
3090 Base level 0, inputs: [9589(3173KB)
>>>>>> 9586(4793KB) 9583(1876KB)
9580(194KB) 9576(6417KB) 9573(1078KB)
>>>>> 9570(405KB)
>>>>>> 9567(29KB)], [9564(1115KB)]
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.557+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850559956,
>>>>>> "job": 5710, "event":
"compaction_started", "compaction_reason":
>>>>>> "LevelL0FilesNum", "files_L0":
[9589, 9586, 9583, 9580, 9576, 9573,
>>>>> 9570,
>>>>>> 9567], "files_L1": [9564],
"score": 1, "input_data_size": 19542092}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1349]
[L] [JOB
>>>>> 5710]
>>>>>> Generated table #9590: 36 keys,
3249524 bytes
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850582987,
>>>>>> "cf_name": "L", "job": 5710,
"event": "table_file_creation",
>>>>> "file_number":
>>>>>> 9590, "file_size": 3249524,
"table_properties": {"data_size": 3247449,
>>>>>> "index_size": 1031,
"index_partitions": 0,
"top_level_index_size": 0,
>>>>>> "index_key_is_user_key": 0,
"index_value_is_delta_encoded": 0,
>>>>>> "filter_size": 197,
"raw_key_size": 576,
"raw_average_key_size": 16,
>>>>>> "raw_value_size": 3246252,
"raw_average_value_size": 90173,
>>>>>> "num_data_blocks": 36,
"num_entries": 36, "num_deletions": 0,
>>>>>> "num_merge_operands": 0,
"num_range_deletions": 0,
"format_version":
>> 0,
>>>>>> "fixed_key_len": 0,
"filter_policy":
"rocksdb.BuiltinBloomFilter",
>>>>>> "column_family_name": "L",
"column_family_id": 10, "comparator":
>>>>>> "leveldb.BytewiseComparator",
"merge_operator": "nullptr",
>>>>>> "prefix_extractor_name":
"nullptr", "property_collectors": "[]",
>>>>>> "compression": "NoCompression",
"compression_options":
>> "window_bits=-14;
>>>>>> level=32767; strategy=0;
max_dict_bytes=0; zstd_max_train_bytes=0;
>>>>>> enabled=0; ", "creation_time":
1633471854, "oldest_key_time": 0,
>>>>>> "file_creation_time": 1633511850}}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
[compaction/compaction_job.cc:1415]
[L] [JOB
>>>>> 5710]
>>>>>> Compacted 8@0 + 1@1 files to L1
=> 3249524 bytes
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.583469)
>>>>>>
[compaction/compaction_job.cc:760] [L]
compacted to: files[0 1 0 0 0 0
>>>>> 0]
>>>>>> max score 0.01, MB/sec: 846.1
rd, 140.7 wr, level 1, files in(8, 1)
>>>>> out(1)
>>>>>> MB in(17.5, 1.1) out(3.1),
read-write-amplify(1.2) write-amplify(0.2)
>>>>> OK,
>>>>>> records in: 376, records
dropped: 340 output_compression:
>> NoCompression
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
(Original Log Time
>> 2021/10/06-09:17:30.583498)
>>>>>> EVENT_LOG_v1 {"time_micros":
1633511850583485, "job": 5710, "event":
>>>>>> "compaction_finished",
"compaction_time_micros": 23098,
>>>>>> "compaction_time_cpu_micros":
20039, "output_level": 1,
>>>>> "num_output_files":
>>>>>> 1, "total_output_size":
3249524, "num_input_records": 376,
>>>>>> "num_output_records": 36,
"num_subcompactions": 1,
>> "output_compression":
>>>>>> "NoCompression",
"num_single_delete_mismatches": 0,
>>>>>>
"num_single_delete_fallthrough": 0,
"lsm_state": [0, 1, 0, 0, 0, 0,
>> 0]}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583615,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9589}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583648,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9586}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583675,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9583}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583709,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9580}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583739,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9576}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583769,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9573}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583804,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9570}
>>>>>> cd88-ceph-osdh-01 bash[6283]:
debug 2021-10-06T09:17:30.581+0000
>>>>>> 7f38fb3ed700 4 rocksdb:
EVENT_LOG_v1 {"time_micros":
>> 1633511850583835,
>>>>>> "job": 5710, "event":
"table_file_deletion", "file_number":
9567}
>>>>>>
_______________________________________________
>>>>>> ceph-users mailing list --
ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
>>>>>> To unsubscribe send an email to
ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
>>>> --
