Hello Igor "Does single OSD startup (after if's experiencing "unable to allocate) takes 20 mins as well?" A: YES Here the example log of the startup and recovery of a problematic osd. https://paste.ubuntu.com/p/2WVJbg7cBy/ Here the example log of a problematic osd https://paste.ubuntu.com/p/qbB6y7663f/ I found this post about a similar error and a bug in 16.2.4... we are running 16.2.5...maybe the bug is not really fixed??? https://tracker.ceph.com/issues/50656 https://forum.proxmox.com/threads/ceph-16-2-pacific-cluster-crash.92367/ Am Mo., 11. Okt. 2021 um 11:53 Uhr schrieb Igor Fedotov < igor.fedotov@xxxxxxxx>: > hmm... so it looks like RocksDB still doesn't perform WAL cleanup during > regular operation but applies it on OSD startup.... > > Does single OSD startup (after if's experiencing "unable to allocate) > takes 20 mins as well? > > Could you please share OSD log containing both that long startup and > following (e.g. 1+ hour) regular operation? > > Preferable for OSD.2 (or whatever one which has been using default > settings from the deployment). > > > Thanks, > > Igor > > > On 10/9/2021 12:18 AM, José H. Freidhof wrote: > > Hi Igor, > > "And was osd.2 redeployed AFTER settings had been reset to defaults ?" > A: YES > > "Anything particular about current cluster use cases?" > A: we are using it temporary as a iscsi target for a vmware esxi cluster > with 6 hosts. We created two 10tb iscsi images/luns for vmware, because the > other datastore are at 90%. > We plan in the future, after ceph is working right, and stable to install > openstack and kvm and we want to convert all vms into rbd images. > Like i told you is a three osd nodes cluster with 32 cores and 256gb ram > and two 10g bond network cards on a 10g network > > "E.g. is it a sort of regular usage (with load flukes and peak) or may be > some permanently running stress load testing. The latter might tend to hold > the resources and e.g. prevent from internal house keeping... > A: Its a SAN for vmware and there are running 43 VMs at the moment... at > the daytime is more stress on the disks because the people are working and > in the afternoon the iops goes down because the users are at home > noting speculative... > > There is something else that i noticed... if i reboot one osd with > 20osds then it takes 20min to come up... if i tail the logs of the osd i > can see a lot of " recovery log mode 2" on all osd > after the 20min the osd comes one after one up and the waldb are small and > no error in the logs about bluefs _allocate unable to allocate... > > it seems that the problem is rocking up after a longer time (12h) > > > Am Fr., 8. Okt. 2021 um 15:24 Uhr schrieb Igor Fedotov < > igor.fedotov@xxxxxxxx>: > >> And was osd.2 redeployed AFTER settings had been reset to defaults ? >> >> Anything particular about current cluster use cases? >> >> E.g. is it a sort of regular usage (with load flukes and peak) or may be >> some permanently running stress load testing. The latter might tend to hold >> the resources and e.g. prevent from internal house keeping... >> >> Igor >> >> >> On 10/8/2021 12:16 AM, José H. Freidhof wrote: >> >> Hi Igor, >> >> yes the same problem is on osd.2 >> >> we have 3 OSD Nodes... Each Node has 20 Bluestore OSDs ... in total we >> have 60 OSDs >> i checked right now one node... and 15 of 20 OSDs have this problem and >> error in the log. >> >> the settings that you have complained some emails ago .. i have reverted >> them to default. >> >> ceph.conf file: >> >> [global] >> fsid = 462c44b4-eed6-11eb-8b2c-a1ad45f88a97 >> mon_host = [v2:10.50.50.21:3300/0,v1:10.50.50.21:6789/0] [v2: >> 10.50.50.22:3300/0,v1:10.50.50.22:6789/0] [v2: >> 10.50.50.20:3300/0,v1:10.50.50.20:6789/0] >> log file = /var/log/ceph/$cluster-$type-$id.log >> max open files = 131072 >> mon compact on trim = False >> osd deep scrub interval = 137438953472 >> osd max scrubs = 16 >> osd objectstore = bluestore >> osd op threads = 2 >> osd scrub load threshold = 0.01 >> osd scrub max interval = 137438953472 >> osd scrub min interval = 137438953472 >> perf = True >> rbd readahead disable after bytes = 0 >> rbd readahead max bytes = 4194304 >> throttler perf counter = False >> >> [client] >> rbd cache = False >> >> >> [mon] >> mon health preluminous compat = True >> mon osd down out interval = 300 >> >> [osd] >> bluestore cache autotune = 0 >> bluestore cache kv ratio = 0.2 >> bluestore cache meta ratio = 0.8 >> bluestore extent map shard max size = 200 >> bluestore extent map shard min size = 50 >> bluestore extent map shard target size = 100 >> bluestore rocksdb options = >> compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB >> osd map share max epochs = 100 >> osd max backfills = 5 >> osd op num shards = 8 >> osd op num threads per shard = 2 >> osd min pg log entries = 10 >> osd max pg log entries = 10 >> osd pg log dups tracked = 10 >> osd pg log trim min = 10 >> >> >> >> root@cd133-ceph-osdh-01:~# ceph config dump >> WHO MASK >> LEVEL OPTION VALUE >> >> >> >> >> >> RO >> global >> basic container_image >> docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb >> >> >> >> >> * >> global >> advanced leveldb_max_open_files 131072 >> >> >> >> >> global >> advanced mon_compact_on_trim false >> >> >> >> >> global >> dev ms_crc_data false >> >> >> >> >> global >> advanced osd_deep_scrub_interval 1209600.000000 >> >> >> >> >> global >> advanced osd_max_scrubs 16 >> >> >> >> >> global >> advanced osd_scrub_load_threshold 0.010000 >> >> >> >> >> global >> advanced osd_scrub_max_interval 1209600.000000 >> >> >> >> >> global >> advanced osd_scrub_min_interval 86400.000000 >> >> >> >> >> global >> advanced perf true >> >> >> >> >> global >> advanced rbd_readahead_disable_after_bytes 0 >> >> >> >> >> global >> advanced rbd_readahead_max_bytes 4194304 >> >> >> >> >> global >> advanced throttler_perf_counter false >> >> >> >> >> mon >> advanced auth_allow_insecure_global_id_reclaim false >> >> >> >> >> mon >> advanced cluster_network 10.50.50.0/24 >> >> >> >> >> >> * >> mon >> advanced mon_osd_down_out_interval 300 >> >> >> >> >> mon >> advanced public_network 10.50.50.0/24 >> >> >> >> >> >> * >> mgr >> advanced mgr/cephadm/container_init True >> >> >> >> >> >> * >> mgr >> advanced mgr/cephadm/device_enhanced_scan true >> >> >> >> >> >> * >> mgr >> advanced mgr/cephadm/migration_current 2 >> >> >> >> >> >> * >> mgr >> advanced mgr/cephadm/warn_on_stray_daemons false >> >> >> >> >> >> * >> mgr >> advanced mgr/cephadm/warn_on_stray_hosts false >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/10.50.50.21/server_addr >> >> >> >> >> >> * >> >> >> >> * >> mgr >> advanced mgr/dashboard/camdatadash/ssl_server_port 8443 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/cd133-ceph-mon-01/server_addr >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/dasboard/server_port 80 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/dashboard/server_addr 10.251.133.161 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/dashboard/ssl_server_port 8443 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/server_addr 0.0.0.0 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/server_port 8080 >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/ssl false >> >> >> >> >> >> * >> mgr >> advanced mgr/dashboard/ssl_server_port 8443 >> >> >> >> >> >> * >> mgr >> advanced mgr/orchestrator/orchestrator cephadm >> >> >> >> >> mgr >> advanced mgr/prometheus/server_addr 0.0.0.0 >> >> >> >> >> >> * >> mgr >> advanced mgr/telemetry/channel_ident true >> >> >> >> >> >> * >> mgr >> advanced mgr/telemetry/enabled true >> >> >> >> >> >> * >> mgr >> advanced mgr/telemetry/last_opt_revision 3 >> >> >> >> >> >> * >> osd >> dev bluestore_cache_autotune true >> >> >> >> >> osd >> dev bluestore_cache_kv_ratio 0.200000 >> >> >> >> >> osd >> dev bluestore_cache_meta_ratio 0.800000 >> >> >> >> >> osd >> dev bluestore_cache_size 2147483648 >> >> >> >> >> osd >> dev bluestore_cache_size_hdd 2147483648 >> >> >> >> >> osd >> dev bluestore_extent_map_shard_max_size 200 >> >> >> >> >> osd >> dev bluestore_extent_map_shard_min_size 50 >> >> >> >> >> osd >> dev bluestore_extent_map_shard_target_size 100 >> >> >> >> >> osd >> advanced bluestore_rocksdb_options >> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >> * >> osd >> advanced mon_osd_cache_size 1024 >> >> >> >> >> osd >> dev ms_crc_data false >> >> >> >> >> osd >> advanced osd_map_share_max_epochs 5 >> >> >> >> >> osd >> advanced osd_max_backfills 1 >> >> >> >> >> osd >> dev osd_max_pg_log_entries 10 >> >> >> >> >> osd >> dev osd_memory_cache_min 3000000000 >> >> >> >> >> osd host:cd133-ceph-osdh-01 >> basic osd_memory_target 5797322096 >> >> >> >> >> osd >> host:cd133k-ceph-osdh-01 basic osd_memory_target >> 9402402385 >> >> >> >> osd host:cd88-ceph-osdh-01 >> basic osd_memory_target 5797322096 >> >> >> >> >> osd >> advanced osd_memory_target_autotune true >> >> >> >> >> osd >> dev osd_min_pg_log_entries 10 >> >> >> >> >> osd >> advanced osd_op_num_shards 8 >> >> >> >> >> >> * >> osd >> advanced osd_op_num_threads_per_shard 2 >> >> >> >> >> >> * >> osd >> dev osd_pg_log_dups_tracked 10 >> >> >> >> >> osd >> dev osd_pg_log_trim_min 10 >> >> >> >> >> osd >> advanced osd_recovery_max_active 3 >> >> >> >> >> osd >> advanced osd_recovery_max_single_start 1 >> >> >> >> >> osd >> advanced osd_recovery_sleep 0.000000 >> >> >> >> >> client >> advanced rbd_cache false >> >> >> >> >> >> >> >> Am Do., 7. Okt. 2021 um 19:27 Uhr schrieb Igor Fedotov < >> igor.fedotov@xxxxxxxx>: >> >>> And does redeployed osd.2 expose the same issue (or at least DB/WAL >>> disbalance) again? Were settings reverted to defaults for it as well? >>> >>> >>> Thanks >>> >>> Igor >>> On 10/7/2021 12:46 PM, José H. Freidhof wrote: >>> >>> Good morning, >>> >>> i checked today the osd.8 and the log shows again the same error >>> bluefs _allocate unable to allocate 0x100000 on bdev 0, allocator name >>> bluefs-wal, allocator type hybrid, capacity 0xb40000000, block size >>> 0x100000, free 0xff000, fragmentation 0, allocated 0x0 >>> >>> any idea why that could be? >>> >>> Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof < >>> harald.freidhof@xxxxxxxxxxxxxx>: >>> >>>> Hi Igor, >>>> >>>> today i repaired one osd node and all osd´s on the node, creating them >>>> new again.... >>>> after that i waited for the rebalance/recovery process and the cluster >>>> was healthy after some hours.. >>>> >>>> i notices that the osd.2 does not have any more this error in the log. >>>> but i noticed it now on the same node on osd.8... so i did the test >>>> that you suggested on osd.8 >>>> >>>> it took nearly 20minutes to compact those db´s on the bluestore, but it >>>> helped... the problem on osd.8 is gone... >>>> >>>> >>>> *so the problem that i have with the alloc on the wal device seems to >>>> be random on different nodes and osd´s and looks like it comes, stay >>>> a while and disappears after a longer while... * >>>> >>>> here are the results that you suggested: >>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device >>>> info >>>> { >>>> "dev": { >>>> "device": "BDEV_WAL", >>>> "total": 48318377984, >>>> "free": 1044480, >>>> "bluefs_used": 48317333504 >>>> }, >>>> "dev": { >>>> "device": "BDEV_DB", >>>> "total": 187904811008, >>>> "free": 79842762752, >>>> "bluefs_used": 108062048256 >>>> }, >>>> "dev": { >>>> "device": "BDEV_SLOW", >>>> "total": 6001172414464, >>>> "free": 5510727389184, >>>> "bluefs_used": 0, >>>> "bluefs max available": 5508815847424 >>>> } >>>> } >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB) >>>> 1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB) >>>> 2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB) >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>> Usage matrix: >>>> DEV/LEV WAL DB SLOW * * >>>> REAL FILES >>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B >>>> 9.7 MiB 1 >>>> WAL 45 GiB 100 GiB 0 B 0 B 0 B >>>> 144 GiB 2319 >>>> DB 0 B 276 MiB 0 B 0 B 0 B >>>> 249 MiB 47 >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B 0 >>>> TOTALS 45 GiB 109 GiB 0 B 0 B 0 B >>>> 0 B 2367 >>>> MAXIMUMS: >>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B >>>> 20 MiB >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>> 192 GiB >>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>> 738 MiB >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B >>>> 0 B >>>> >>>> --- >>>> >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb: >>>> [db_impl/db_impl_write.cc:1668] [L] New memtable created with log file: >>>> #13656. Immutable memtables: 1. >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to >>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>> fragmentation 0, allocated 0x0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time >>>> 2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198] >>>> Calling FlushMemTableToOutputFile with column family [L], flush slots >>>> available 1, compaction slots available 1, flush slots scheduled 1, >>>> compaction slots scheduled 0 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321] >>>> [L] [JOB 8859] Flushing memtable with next log file: 13655 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321] >>>> [L] [JOB 8859] Flushing memtable with next log file: 13656 >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1 >>>> {"time_micros": 1633549894998273, "job": 8859, "event": "flush_started", >>>> "num_memtables": 2, "num_entries": 3662, "num_deletes": 0, >>>> "total_data_size": 130482337, "memory_usage": 132976224, "flush_reason": >>>> "Write Buffer Full"} >>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350] >>>> [L] [JOB 8859] Level-0 flush table #13657: started >>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1 >>>> {"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event": >>>> "table_file_creation", "file_number": 13657, "file_size": 2952537, >>>> "table_properties": {"data_size": 2951222, "index_size": 267, >>>> "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": >>>> 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": >>>> 1120, "raw_average_key_size": 16, "raw_value_size": 2950151, >>>> "raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70, >>>> "num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0, >>>> "format_version": 0, "fixed_key_len": 0, "filter_policy": >>>> "rocksdb.BuiltinBloomFilter", "column_family_name": "L", >>>> "column_family_id": 10, "comparator": "leveldb.BytewiseComparator", >>>> "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", >>>> "property_collectors": "[]", "compression": "NoCompression", >>>> "compression_options": "window_bits=-14; level=32767; strategy=0; >>>> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": >>>> 1633549854, "oldest_key_time": 1633549854, "file_creation_time": >>>> 1633549894}} >>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug >>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401] >>>> [L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:~# ceph osd set noout >>>> root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8 >>>> root@cd88-ceph-osdh-01:~# ceph orch ps >>>> ... >>>> osd.7 cd133-ceph-osdh-01 >>>> running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd >>>> 8a98ae61f0eb >>>> osd.8 cd88-ceph-osdh-01 >>>> stopped 5s ago - - 5528M <unknown> <unknown> >>>> <unknown> >>>> osd.9 cd133k-ceph-osdh-01 >>>> running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd >>>> 0ff7584b1808 >>>> ... >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact >>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 0 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_db_and_around read-only:0 repair:0 >>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>> discard not supported >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB) >>>> rotational discard not supported >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 1 path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175 >>>> GiB >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>> discard not supported >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>> size 5.5 TiB >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB) >>>> non-rotational discard supported >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 0 path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45 >>>> GiB >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id >>>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, >>>> block size 0x100000 >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id >>>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000, >>>> block size 0x100000 >>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc shared, >>>> id 2, capacity 0x57541c00000, block size 0x10000 >>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount >>>> shared_bdev_used = 0 >>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _prepare_db_environment set db_paths to db,178509578240 >>>> db.slow,5701113793740 >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_db opened rocksdb path db options >>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_super_meta old nid_max 167450 >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_super_meta old blobid_max 30720 >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_super_meta freelist_type bitmap >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_super_meta ondisk_format 4 compat_ondisk_format 3 >>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_super_meta min_alloc_size 0x1000 >>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init >>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg >>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _init_alloc opening allocation metadata >>>> 2021-10-06T22:01:41.031+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid, >>>> capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000, >>>> fragmentation 2.76445e-05 >>>> 2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount >>>> 2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close >>>> 2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close >>>> 2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB) >>>> rotational discard not supported >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 1 path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175 >>>> GiB >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>> discard not supported >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>> size 5.5 TiB >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB) >>>> non-rotational discard supported >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>> bdev 0 path >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45 >>>> GiB >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id >>>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000, >>>> block size 0x100000 >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id >>>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000, >>>> block size 0x100000 >>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc shared, >>>> id 2, capacity 0x57541c00000, block size 0x10000 >>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount >>>> shared_bdev_used = 0 >>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _prepare_db_environment set db_paths to db,178509578240 >>>> db.slow,5701113793740 >>>> 2021-10-06T22:09:36.519+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>> _open_db opened rocksdb path db options >>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 >>>> 2021-10-06T22:09:54.067+0200 7f87bde3c240 1 >>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount >>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount >>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close >>>> 2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close >>>> 2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>> 2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown >>>> 2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8 >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:/# ceph -s >>>> cluster: >>>> id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97 >>>> health: HEALTH_OK >>>> >>>> services: >>>> mon: 3 daemons, quorum >>>> cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h) >>>> mgr: cd133-ceph-mon-01.mzapob(active, since 15h), standbys: >>>> cd133k-ceph-mon-01.imikwh >>>> osd: 60 osds: 60 up (since 2m), 60 in (since 3h) >>>> rgw: 4 daemons active (2 hosts, 1 zones) >>>> tcmu-runner: 10 portals active (2 hosts) >>>> >>>> data: >>>> pools: 6 pools, 361 pgs >>>> objects: 2.46M objects, 8.0 TiB >>>> usage: 33 TiB used, 304 TiB / 338 TiB avail >>>> pgs: 361 active+clean >>>> >>>> io: >>>> client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr >>>> >>>> >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device >>>> info >>>> { >>>> "dev": { >>>> "device": "BDEV_WAL", >>>> "total": 48318377984, >>>> >>>> * "free": 41354784768, * "bluefs_used": 6963593216 >>>> }, >>>> "dev": { >>>> "device": "BDEV_DB", >>>> "total": 187904811008, >>>> "free": 187302928384, >>>> "bluefs_used": 601882624 >>>> }, >>>> "dev": { >>>> "device": "BDEV_SLOW", >>>> "total": 6001172414464, >>>> "free": 5507531620352, >>>> "bluefs_used": 0, >>>> "bluefs max available": 5505566572544 >>>> } >>>> } >>>> >>>> --- >>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats >>>> 0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB) >>>> 1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB) >>>> 2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB) >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>> Usage matrix: >>>> DEV/LEV WAL DB SLOW * * >>>> REAL FILES >>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B >>>> 10 MiB 0 >>>> WAL 6.5 GiB 0 B 0 B 0 B 0 B >>>> 6.4 GiB 102 >>>> DB 0 B 573 MiB 0 B 0 B 0 B >>>> 557 MiB 22 >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B 0 >>>> TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B >>>> 0 B 125 >>>> MAXIMUMS: >>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B >>>> 17 MiB >>>> WAL 45 GiB 101 GiB 0 B 0 B 0 B >>>> 145 GiB >>>> DB 0 B 688 MiB 0 B 0 B 0 B >>>> 670 MiB >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B >>>> TOTALS 45 GiB 101 GiB 0 B 0 B 0 B >>>> 0 B >>>> >>>> ---- >>>> >>>> >>>> Here are the osd.2... the problem disapeared from alone >>>> very strange... >>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats >>>> 0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB) >>>> 1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB) >>>> 2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB) >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>> Usage matrix: >>>> DEV/LEV WAL DB SLOW * * >>>> REAL FILES >>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B >>>> 10 MiB 1 >>>> WAL 31 GiB 17 GiB 0 B 0 B 0 B >>>> 48 GiB 765 >>>> DB 0 B 193 MiB 0 B 0 B 0 B >>>> 175 MiB 30 >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B 0 >>>> TOTALS 32 GiB 21 GiB 0 B 0 B 0 B >>>> 0 B 796 >>>> MAXIMUMS: >>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B >>>> 17 MiB >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>> 192 GiB >>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>> 741 MiB >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>> 0 B >>>> TOTALS 45 GiB 153 GiB 0 B 0 B 0 B >>>> 0 B >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device >>>> info >>>> { >>>> "dev": { >>>> "device": "BDEV_WAL", >>>> "total": 48318377984, >>>> "free": 15043915776, >>>> "bluefs_used": 33274462208 >>>> }, >>>> "dev": { >>>> "device": "BDEV_DB", >>>> "total": 187904811008, >>>> "free": 169235963904, >>>> "bluefs_used": 18668847104 >>>> }, >>>> "dev": { >>>> "device": "BDEV_SLOW", >>>> "total": 6001172414464, >>>> "free": 5602453327872, >>>> "bluefs_used": 0, >>>> "bluefs max available": 5600865222656 >>>> } >>>> } >>>> >>>> >>>> >>>> >>>> >>>> >>>> Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov < >>>> igor.fedotov@xxxxxxxx>: >>>> >>>>> >>>>> On 10/6/2021 4:25 PM, José H. Freidhof wrote: >>>>> > hi, >>>>> > >>>>> > no risk no fun 😂 okay >>>>> > I have reset the settings you mentioned to standard. >>>>> > >>>>> > what you exactly mean with taking offline the osd? ceph orch daemon >>>>> stop >>>>> > osd.2? or mark down? >>>>> "daemon stop" is enough. You might want to set noout flag before that >>>>> though... >>>>> > >>>>> > for the command which path i use? you mean: >>>>> > >>>>> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact??? >>>>> yep >>>>> > >>>>> > >>>>> > Igor Fedotov <ifedotov@xxxxxxx> schrieb am Mi., 6. Okt. 2021, 13:33: >>>>> > >>>>> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote: >>>>> >>> Hi Igor, >>>>> >>> >>>>> >>> yes i have some osd settings set :-) here are my ceph config dump. >>>>> those >>>>> >>> settings are from a redhat document for bluestore devices >>>>> >>> maybe it is that setting causing this problem? "advanced >>>>> >>> mon_compact_on_trim false"??? >>>>> >> OMG!!! >>>>> >> >>>>> >> No - mon_compact_on_trim has nothing to deal with bluestore. >>>>> >> >>>>> >> Highly likely it's bluestore_rocksdb_options which hurts... >>>>> >> Documentations tend to fall behind the best practices.... I would >>>>> >> strongly discourage you from using non-default settings unless it's >>>>> >> absolutely clear why this is necessary. >>>>> >> >>>>> >> Even at the first glance the following settings (just a few ones I'm >>>>> >> completely aware) are suboptimal/non-recommended: >>>>> >> >>>>> >> rocksdb_perf >>>>> >> >>>>> >> bluefs_sync_write >>>>> >> >>>>> >> bluefs_csum_type >>>>> >> >>>>> >> >>>>> >> Not to mention bluestore_rocksdb_options which hasn't got much >>>>> adoption >>>>> >> so far and apparently greatly alters rocksdb behavior... >>>>> >> >>>>> >> >>>>> >> So I would suggest to revert rocksdb options back to default, run >>>>> the >>>>> >> compaction and if it succeeds monitor the OSD for a while. Then if >>>>> it >>>>> >> works fine - apply the same for others >>>>> >> >>>>> >> >>>>> >> Hope this helps, >>>>> >> >>>>> >> Igor >>>>> >> >>>>> >> >>>>> >> >>>>> >>> i will test it this afternoon... at the moment are everything semi >>>>> >>> prodcuctive and i need to repair one osd node.. because i think of >>>>> this >>>>> >>> reason the osds crashed on the node and the osd container crashes >>>>> with a >>>>> >>> dump while coming up now. >>>>> >>> need first to replicate all between all three nodes and then i can >>>>> take >>>>> >>> offline the osd.2.and test your command. i will inform you later... >>>>> >>> >>>>> >>> root@cd88-ceph-osdh-01:/# ceph config dump >>>>> >>> WHO MASK >>>>> >>> LEVEL OPTION VALUE >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> RO >>>>> >>> global >>>>> >>> advanced leveldb_max_open_files 131072 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced mon_compact_on_trim false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> dev ms_crc_data false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced osd_deep_scrub_interval >>>>> 1209600.000000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced osd_max_scrubs 16 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced osd_scrub_load_threshold 0.010000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced osd_scrub_max_interval >>>>> 1209600.000000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced osd_scrub_min_interval >>>>> 86400.000000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced perf true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced rbd_readahead_disable_after_bytes 0 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced rbd_readahead_max_bytes 4194304 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced rocksdb_perf true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> global >>>>> >>> advanced throttler_perf_counter false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> mon >>>>> >>> advanced auth_allow_insecure_global_id_reclaim false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> mon >>>>> >>> advanced cluster_network >>>>> 10.50.50.0/24 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mon >>>>> >>> advanced mon_osd_down_out_interval 300 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> mon >>>>> >>> advanced public_network >>>>> 10.50.50.0/24 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mgr >>>>> >>> advanced mgr/cephadm/container_init True >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mgr >>>>> >>> advanced mgr/cephadm/device_enhanced_scan true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mgr >>>>> >>> advanced mgr/cephadm/migration_current 2 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mgr >>>>> >>> advanced mgr/cephadm/warn_on_stray_daemons false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> mgr >>>>> >>> advanced mgr/cephadm/warn_on_stray_hosts false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> osd >>>>> >>> advanced bluefs_sync_write true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_cache_autotune true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_cache_kv_ratio 0.200000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_cache_meta_ratio 0.800000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_cache_size 2147483648 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_cache_size_hdd 2147483648 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced bluestore_csum_type none >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_extent_map_shard_max_size 200 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_extent_map_shard_min_size 50 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev bluestore_extent_map_shard_target_size 100 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced bluestore_rocksdb_options >>>>> >>> >>>>> >> >>>>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >>>>> >>> * >>>>> >>> osd >>>>> >>> advanced mon_osd_cache_size 1024 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev ms_crc_data false >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_map_share_max_epochs 5 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_max_backfills 1 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev osd_max_pg_log_entries 10 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev osd_memory_cache_min 3000000000 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >> host:cd133-ceph-osdh-01 >>>>> >>> basic osd_memory_target 5797322383 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >> host:cd133k-ceph-osdh-01 >>>>> >>> basic osd_memory_target >>>>> 9402402385 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> host:cd88-ceph-osdh-01 >>>>> >>> basic osd_memory_target >>>>> 5797322096 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_memory_target_autotune true >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev osd_min_pg_log_entries 10 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_op_num_shards 8 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> osd >>>>> >>> advanced osd_op_num_threads_per_shard 2 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> * >>>>> >>> osd >>>>> >>> dev osd_pg_log_dups_tracked 10 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> dev osd_pg_log_trim_min 10 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_recovery_max_active 3 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_recovery_max_single_start 1 >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> osd >>>>> >>> advanced osd_recovery_sleep 0.000000 >>>>> >>> >>>>> >>> >>>>> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov < >>>>> ifedotov@xxxxxxx >>>>> >>> : >>>>> >>> >>>>> >>>> Jose, >>>>> >>>> >>>>> >>>> In fact 48GB is a way too much for WAL drive - usually the write >>>>> ahead >>>>> >> log >>>>> >>>> tend to be 2-4 GBs. >>>>> >>>> >>>>> >>>> But in your case it's ~150GB, while DB itself is very small >>>>> (146MB!!!): >>>>> >>>> >>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B >>>>> >>>> 154 GiB 2400 >>>>> >>>> >>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B >>>>> >>>> 146 MiB 30 >>>>> >>>> >>>>> >>>> >>>>> >>>> which means that there are some issues with RocksDB's WAL >>>>> processing, >>>>> >>>> which needs some troubleshooting... >>>>> >>>> >>>>> >>>> Curious if other OSDs are suffering from the same and whether you >>>>> have >>>>> >> any >>>>> >>>> custom settings for your OSD(s)? >>>>> >>>> >>>>> >>>> Additionally you might want to try the following command to >>>>> compact this >>>>> >>>> specific OSD manually and check if this would normalize the DB >>>>> layout - >>>>> >> the >>>>> >>>> majority of data has to be at DB level not WAL. Please share the >>>>> >> resulting >>>>> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command) >>>>> after the >>>>> >>>> compaction is fulfiled and OSD is restarted. >>>>> >>>> >>>>> >>>> The compaction command to be applied on an offline OSD: >>>>> >> "ceph-kvstore-tool >>>>> >>>> bluestore-kv <path-to-osd> compact" >>>>> >>>> >>>>> >>>> Even if the above works great please refrain from applying that >>>>> >> compaction >>>>> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would >>>>> WAL grow >>>>> >>>> again or not? >>>>> >>>> >>>>> >>>> Thanks, >>>>> >>>> >>>>> >>>> Igor >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote: >>>>> >>>> >>>>> >>>> Hello Igor, >>>>> >>>> >>>>> >>>> yes the volume is nvme wal partitions for the bluestore >>>>> devicegroups >>>>> >> are >>>>> >>>> only 48gb each >>>>> >>>> >>>>> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb >>>>> (WAL) >>>>> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb >>>>> >> (rock.db) >>>>> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db) >>>>> >>>> >>>>> >>>> each blustore have 1 partition nvme,ssd and hdd like described in >>>>> the >>>>> >>>> documentation >>>>> >>>> >>>>> >> >>>>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/ >>>>> >>>> is this to small or can i adjust the max allocation on the wal >>>>> nvme >>>>> >> device >>>>> >>>> in the ceph configuration? >>>>> >>>> i know that the ssd and nvme are to small for those 5.5tb disk... >>>>> its 1% >>>>> >>>> only ot the rotation disk. >>>>> >>>> i am new in ceph and still or always learning, but we are in a >>>>> little >>>>> >>>> hurry because our other datastores are old and full. >>>>> >>>> >>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs >>>>> device >>>>> >> info >>>>> >>>> { >>>>> >>>> "dev": { >>>>> >>>> "device": "BDEV_WAL", >>>>> >>>> "total": 48318377984, >>>>> >>>> "free": 1044480, >>>>> >>>> "bluefs_used": 48317333504 >>>>> >>>> }, >>>>> >>>> "dev": { >>>>> >>>> "device": "BDEV_DB", >>>>> >>>> "total": 187904811008, >>>>> >>>> "free": 68757217280, >>>>> >>>> "bluefs_used": 119147593728 >>>>> >>>> }, >>>>> >>>> "dev": { >>>>> >>>> "device": "BDEV_SLOW", >>>>> >>>> "total": 6001172414464, >>>>> >>>> "free": 5624912359424, >>>>> >>>> "bluefs_used": 0, >>>>> >>>> "bluefs max available": 5624401231872 >>>>> >>>> } >>>>> >>>> } >>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats >>>>> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB) >>>>> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB) >>>>> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB) >>>>> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>>> >> db_total:178509578240, >>>>> >>>> slow_total:5701113793740, db_avail:103884521472 >>>>> >>>> Usage matrix: >>>>> >>>> DEV/LEV WAL DB SLOW * * >>>>> >>>> REAL FILES >>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B >>>>> >>>> 7.5 MiB 1 >>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B >>>>> >>>> 154 GiB 2400 >>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B >>>>> >>>> 146 MiB 30 >>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> >> 0 >>>>> >>>> B 0 >>>>> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B >>>>> >> 0 >>>>> >>>> B 2431 >>>>> >>>> MAXIMUMS: >>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B >>>>> >> 17 >>>>> >>>> MiB >>>>> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>>> >>>> 192 GiB >>>>> >>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>>> >>>> 741 MiB >>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> >> 0 B >>>>> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B >>>>> >> 0 B >>>>> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov < >>>>> >> ifedotov@xxxxxxx>: >>>>> >>>>> Hey Jose, >>>>> >>>>> >>>>> >>>>> it looks like your WAL volume is out of space which looks weird >>>>> given >>>>> >>>>> its capacity = 48Gb. >>>>> >>>>> >>>>> >>>>> Could you please share the output of the following commands: >>>>> >>>>> >>>>> >>>>> ceph daemon osd.N bluestore bluefs device info >>>>> >>>>> >>>>> >>>>> ceph daemon osd.N bluefs stats >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> Igor >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote: >>>>> >>>>>> Hello together >>>>> >>>>>> >>>>> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this >>>>> >> messages >>>>> >>>>> in >>>>> >>>>>> the service logs of the osd daemons. >>>>> >>>>>> >>>>> >>>>>> we have three osd nodes .. each node has 20osds as bluestore >>>>> with >>>>> >>>>>> nvme/ssd/hdd >>>>> >>>>>> >>>>> >>>>>> is this a bug or maybe i have some settings wrong? >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New >>>>> >>>>> memtable >>>>> >>>>>> created with log file: #9588. Immutable memtables: 1. >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on >>>>> bdev >>>>> >> 0, >>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>> >> 0xb40000000, >>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>> 0x0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.547575) >>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling >>>>> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots >>>>> >> available >>>>> >>>>> 1, >>>>> >>>>>> compaction slots available 1, flush slots scheduled 1, >>>>> compaction >>>>> >> slots >>>>> >>>>>> scheduled 0 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] >>>>> Flushing >>>>> >>>>>> memtable with next log file: 9587 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] >>>>> Flushing >>>>> >>>>>> memtable with next log file: 9588 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850547916, >>>>> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2, >>>>> >>>>> "num_entries": >>>>> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926, >>>>> "memory_usage": >>>>> >>>>>> 130479920, "flush_reason": "Write Buffer Full"} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] >>>>> Level-0 >>>>> >> flush >>>>> >>>>>> table #9589: started >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850559292, >>>>> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation", >>>>> >>>>> "file_number": >>>>> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size": >>>>> 3247855, >>>>> >>>>>> "index_size": 1031, "index_partitions": 0, >>>>> "top_level_index_size": 0, >>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, >>>>> >>>>>> "filter_size": 197, "raw_key_size": 1088, >>>>> "raw_average_key_size": 16, >>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739, >>>>> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32, >>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, >>>>> "format_version": >>>>> >> 0, >>>>> >>>>>> "fixed_key_len": 0, "filter_policy": >>>>> "rocksdb.BuiltinBloomFilter", >>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator": >>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr", >>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]", >>>>> >>>>>> "compression": "NoCompression", "compression_options": >>>>> >> "window_bits=-14; >>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; >>>>> zstd_max_train_bytes=0; >>>>> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time": >>>>> >>>>> 1633511730, >>>>> >>>>>> "file_creation_time": 1633511850}} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] >>>>> Level-0 >>>>> >> flush >>>>> >>>>>> table #9589: 3249934 bytes OK >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.559362) >>>>> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.559583) >>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable >>>>> #1 >>>>> >> done >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.559586) >>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable >>>>> #2 >>>>> >> done >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.559601) >>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709, >>>>> "event": >>>>> >>>>>> "flush_finished", "output_compression": "NoCompression", >>>>> "lsm_state": >>>>> >>>>> [8, >>>>> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.559638) >>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary: >>>>> files[8 1 >>>>> >>>>> 0 0 >>>>> >>>>>> 0 0 0] max score 1.00 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] >>>>> [L] [JOB >>>>> >>>>> 5710] >>>>> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00 >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L] >>>>> >>>>> Compaction >>>>> >>>>>> start summary: Base version 3090 Base level 0, inputs: >>>>> [9589(3173KB) >>>>> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB) >>>>> >>>>> 9570(405KB) >>>>> >>>>>> 9567(29KB)], [9564(1115KB)] >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850559956, >>>>> >>>>>> "job": 5710, "event": "compaction_started", "compaction_reason": >>>>> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, >>>>> 9573, >>>>> >>>>> 9570, >>>>> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size": >>>>> 19542092} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] >>>>> [L] [JOB >>>>> >>>>> 5710] >>>>> >>>>>> Generated table #9590: 36 keys, 3249524 bytes >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850582987, >>>>> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation", >>>>> >>>>> "file_number": >>>>> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size": >>>>> 3247449, >>>>> >>>>>> "index_size": 1031, "index_partitions": 0, >>>>> "top_level_index_size": 0, >>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, >>>>> >>>>>> "filter_size": 197, "raw_key_size": 576, >>>>> "raw_average_key_size": 16, >>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173, >>>>> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0, >>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, >>>>> "format_version": >>>>> >> 0, >>>>> >>>>>> "fixed_key_len": 0, "filter_policy": >>>>> "rocksdb.BuiltinBloomFilter", >>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator": >>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr", >>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]", >>>>> >>>>>> "compression": "NoCompression", "compression_options": >>>>> >> "window_bits=-14; >>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; >>>>> zstd_max_train_bytes=0; >>>>> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0, >>>>> >>>>>> "file_creation_time": 1633511850}} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] >>>>> [L] [JOB >>>>> >>>>> 5710] >>>>> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.583469) >>>>> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 >>>>> 0 0 0 0 >>>>> >>>>> 0] >>>>> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files >>>>> in(8, 1) >>>>> >>>>> out(1) >>>>> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2) >>>>> write-amplify(0.2) >>>>> >>>>> OK, >>>>> >>>>>> records in: 376, records dropped: 340 output_compression: >>>>> >> NoCompression >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time >>>>> >> 2021/10/06-09:17:30.583498) >>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710, >>>>> "event": >>>>> >>>>>> "compaction_finished", "compaction_time_micros": 23098, >>>>> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1, >>>>> >>>>> "num_output_files": >>>>> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376, >>>>> >>>>>> "num_output_records": 36, "num_subcompactions": 1, >>>>> >> "output_compression": >>>>> >>>>>> "NoCompression", "num_single_delete_mismatches": 0, >>>>> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, >>>>> 0, 0, >>>>> >> 0]} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583615, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9589} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583648, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9586} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583675, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9583} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583709, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9580} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583739, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9576} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583769, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9573} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583804, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9570} >>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000 >>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>> >> 1633511850583835, >>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>> 9567} >>>>> >>>>>> _______________________________________________ >>>>> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>> >>>> -- >>>>> >>>> >>>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |