Hi Igor, Thx for checking the logs.. but what the hell is going on here? :-) Yes its true i tested the and created the osd´s with three different rockdb options. I can not understand why the osd dont have the same rockdb option, because i have created ALL OSDs new after set and test those settings. Maybe i do something wrong with the re-deployment of the osds? What i do: ceph osd out osd.x ceph osd down osd.x systemctl stop ceph-osd@x ceph osd rm osd.x ceph osd crush rm osd.x ceph auth del osd.x ceph-volume lvm zap --destroy /dev/ceph-block-0/block-0 (lvm hdd partition) ceph-volume lvm zap --destroy /dev/ceph-db-0/db-0 (lvm ssd partition) ceph-volume lvm zap --destroy /dev/ceph-wal-0/wal-db-0 (lvm nvme partition) ... Later i recreate the osds with: cephadm shell -m /var/lib/ceph ceph auth export client.bootstrap-osd vi /var/lib/ceph/bootstrap-osd/ceph.keyring ceph-volume lvm prepare --no-systemd --bluestore --data ceph-block-4/block-4 --block.wal ceph-wal-0/waldb-4 --block.db ceph-db-0/db-4 cp -r /var/lib/ceph/osd /mnt/ceph/ Exit the shell in the container. cephadm --image ceph/ceph:v16.2.5 adopt --style legacy --name osd.X systemctl start ceph-462c44b4-eed6-11eb-8b2c-a1ad45f88a97@xxxxxx.service Igor one question: is there actually an easier way to recreate the osd? maybe over the dashboard? can you recommend something? i have no problem to create the osd on the nodes again, but i need to be sure that no old setting stays on the osd. Am Di., 12. Okt. 2021 um 12:03 Uhr schrieb Igor Fedotov < igor.fedotov@xxxxxxxx>: > Hey Jose, > > your rocksdb settings are still different from the default ones. > > These are options you shared originally: > > > compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB > > These are ones I could find in osd.5 startup log, note e.g. > max_write_buffer_number: > > Oct 12 09:09:30 cd88-ceph-osdh-01 bash[1572206]: debug > 2021-10-12T07:09:30.686+0000 7f16d24a0080 1 > bluestore(/var/lib/ceph/osd/ceph-5) _open_db opened rocksdb path db options > compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB > > And here are the ones I'd expect as defaults - again please note > max_write_buffer_number: > > > compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 > > > And here is the source code for v16.2.5 where the expected default line > comes from: > > > https://github.com/ceph/ceph/blob/0883bdea7337b95e4b611c768c0279868462204a/src/common/options.cc#L4644 > > > Not that I'm absolutely sure this is the actual root cause but I'd > suggest to revert back to the baseline prior to proceeding with the > troubleshooting... > > So please adjust properly and restart OSDs!!! Hopefully it wouldn't need a > redeployment... > > > As for https://tracker.ceph.com/issues/50656 - it's irrelevant to your > case. It was unexpected ENOSPC result from an allocator which still had > enough free space. But in your case bluefs allocator doesn't have free > space at all as the latter is totally wasted by tons of WAL files. > > > Thanks, > > Igor > > > > On 10/12/2021 10:51 AM, José H. Freidhof wrote: > > Hello Igor > > "Does single OSD startup (after if's experiencing "unable to allocate) > takes 20 mins as well?" > A: YES > > Here the example log of the startup and recovery of a problematic osd. > https://paste.ubuntu.com/p/2WVJbg7cBy/ > > Here the example log of a problematic osd > https://paste.ubuntu.com/p/qbB6y7663f/ > > I found this post about a similar error and a bug in 16.2.4... we are > running 16.2.5...maybe the bug is not really fixed??? > https://tracker.ceph.com/issues/50656 > https://forum.proxmox.com/threads/ceph-16-2-pacific-cluster-crash.92367/ > > > > Am Mo., 11. Okt. 2021 um 11:53 Uhr schrieb Igor Fedotov < > igor.fedotov@xxxxxxxx>: > >> hmm... so it looks like RocksDB still doesn't perform WAL cleanup during >> regular operation but applies it on OSD startup.... >> >> Does single OSD startup (after if's experiencing "unable to allocate) >> takes 20 mins as well? >> >> Could you please share OSD log containing both that long startup and >> following (e.g. 1+ hour) regular operation? >> >> Preferable for OSD.2 (or whatever one which has been using default >> settings from the deployment). >> >> >> Thanks, >> >> Igor >> >> >> On 10/9/2021 12:18 AM, José H. Freidhof wrote: >> >> Hi Igor, >> >> "And was osd.2 redeployed AFTER settings had been reset to defaults ?" >> A: YES >> >> "Anything particular about current cluster use cases?" >> A: we are using it temporary as a iscsi target for a vmware esxi cluster >> with 6 hosts. We created two 10tb iscsi images/luns for vmware, because the >> other datastore are at 90%. >> We plan in the future, after ceph is working right, and stable to install >> openstack and kvm and we want to convert all vms into rbd images. >> Like i told you is a three osd nodes cluster with 32 cores and 256gb ram >> and two 10g bond network cards on a 10g network >> >> "E.g. is it a sort of regular usage (with load flukes and peak) or may be >> some permanently running stress load testing. The latter might tend to hold >> the resources and e.g. prevent from internal house keeping... >> A: Its a SAN for vmware and there are running 43 VMs at the moment... at >> the daytime is more stress on the disks because the people are working and >> in the afternoon the iops goes down because the users are at home >> noting speculative... >> >> There is something else that i noticed... if i reboot one osd with >> 20osds then it takes 20min to come up... if i tail the logs of the osd i >> can see a lot of " recovery log mode 2" on all osd >> after the 20min the osd comes one after one up and the waldb are small >> and no error in the logs about bluefs _allocate unable to allocate... >> >> it seems that the problem is rocking up after a longer time (12h) >> >> >> Am Fr., 8. Okt. 2021 um 15:24 Uhr schrieb Igor Fedotov < >> igor.fedotov@xxxxxxxx>: >> >>> And was osd.2 redeployed AFTER settings had been reset to defaults ? >>> >>> Anything particular about current cluster use cases? >>> >>> E.g. is it a sort of regular usage (with load flukes and peak) or may be >>> some permanently running stress load testing. The latter might tend to hold >>> the resources and e.g. prevent from internal house keeping... >>> >>> Igor >>> >>> >>> On 10/8/2021 12:16 AM, José H. Freidhof wrote: >>> >>> Hi Igor, >>> >>> yes the same problem is on osd.2 >>> >>> we have 3 OSD Nodes... Each Node has 20 Bluestore OSDs ... in total we >>> have 60 OSDs >>> i checked right now one node... and 15 of 20 OSDs have this problem and >>> error in the log. >>> >>> the settings that you have complained some emails ago .. i have reverted >>> them to default. >>> >>> ceph.conf file: >>> >>> [global] >>> fsid = 462c44b4-eed6-11eb-8b2c-a1ad45f88a97 >>> mon_host = [v2:10.50.50.21:3300/0,v1:10.50.50.21:6789/0] [v2: >>> 10.50.50.22:3300/0,v1:10.50.50.22:6789/0] [v2: >>> 10.50.50.20:3300/0,v1:10.50.50.20:6789/0] >>> log file = /var/log/ceph/$cluster-$type-$id.log >>> max open files = 131072 >>> mon compact on trim = False >>> osd deep scrub interval = 137438953472 >>> osd max scrubs = 16 >>> osd objectstore = bluestore >>> osd op threads = 2 >>> osd scrub load threshold = 0.01 >>> osd scrub max interval = 137438953472 >>> osd scrub min interval = 137438953472 >>> perf = True >>> rbd readahead disable after bytes = 0 >>> rbd readahead max bytes = 4194304 >>> throttler perf counter = False >>> >>> [client] >>> rbd cache = False >>> >>> >>> [mon] >>> mon health preluminous compat = True >>> mon osd down out interval = 300 >>> >>> [osd] >>> bluestore cache autotune = 0 >>> bluestore cache kv ratio = 0.2 >>> bluestore cache meta ratio = 0.8 >>> bluestore extent map shard max size = 200 >>> bluestore extent map shard min size = 50 >>> bluestore extent map shard target size = 100 >>> bluestore rocksdb options = >>> compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB >>> osd map share max epochs = 100 >>> osd max backfills = 5 >>> osd op num shards = 8 >>> osd op num threads per shard = 2 >>> osd min pg log entries = 10 >>> osd max pg log entries = 10 >>> osd pg log dups tracked = 10 >>> osd pg log trim min = 10 >>> >>> >>> >>> root@cd133-ceph-osdh-01:~# ceph config dump >>> WHO MASK >>> LEVEL OPTION VALUE >>> >>> >>> >>> >>> >>> RO >>> global >>> basic container_image >>> docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb >>> >>> >>> >>> >>> * >>> global >>> advanced leveldb_max_open_files 131072 >>> >>> >>> >>> >>> global >>> advanced mon_compact_on_trim false >>> >>> >>> >>> >>> global >>> dev ms_crc_data false >>> >>> >>> >>> >>> global >>> advanced osd_deep_scrub_interval 1209600.000000 >>> >>> >>> >>> >>> global >>> advanced osd_max_scrubs 16 >>> >>> >>> >>> >>> global >>> advanced osd_scrub_load_threshold 0.010000 >>> >>> >>> >>> >>> global >>> advanced osd_scrub_max_interval 1209600.000000 >>> >>> >>> >>> >>> global >>> advanced osd_scrub_min_interval 86400.000000 >>> >>> >>> >>> >>> global >>> advanced perf true >>> >>> >>> >>> >>> global >>> advanced rbd_readahead_disable_after_bytes 0 >>> >>> >>> >>> >>> global >>> advanced rbd_readahead_max_bytes 4194304 >>> >>> >>> >>> >>> global >>> advanced throttler_perf_counter false >>> >>> >>> >>> >>> mon >>> advanced auth_allow_insecure_global_id_reclaim false >>> >>> >>> >>> >>> mon >>> advanced cluster_network 10.50.50.0/24 >>> >>> >>> >>> >>> >>> * >>> mon >>> advanced mon_osd_down_out_interval 300 >>> >>> >>> >>> >>> mon >>> advanced public_network 10.50.50.0/24 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/cephadm/container_init True >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/cephadm/device_enhanced_scan true >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/cephadm/migration_current 2 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/cephadm/warn_on_stray_daemons false >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/cephadm/warn_on_stray_hosts false >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/10.50.50.21/server_addr >>> >>> >>> >>> >>> >>> * >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/camdatadash/ssl_server_port 8443 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/cd133-ceph-mon-01/server_addr >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/dasboard/server_port 80 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/dashboard/server_addr 10.251.133.161 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/dashboard/ssl_server_port 8443 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/server_addr 0.0.0.0 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/server_port 8080 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/ssl false >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/dashboard/ssl_server_port 8443 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/orchestrator/orchestrator cephadm >>> >>> >>> >>> >>> mgr >>> advanced mgr/prometheus/server_addr 0.0.0.0 >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/telemetry/channel_ident true >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/telemetry/enabled true >>> >>> >>> >>> >>> >>> * >>> mgr >>> advanced mgr/telemetry/last_opt_revision 3 >>> >>> >>> >>> >>> >>> * >>> osd >>> dev bluestore_cache_autotune true >>> >>> >>> >>> >>> osd >>> dev bluestore_cache_kv_ratio 0.200000 >>> >>> >>> >>> >>> osd >>> dev bluestore_cache_meta_ratio 0.800000 >>> >>> >>> >>> >>> osd >>> dev bluestore_cache_size 2147483648 >>> >>> >>> >>> >>> osd >>> dev bluestore_cache_size_hdd 2147483648 >>> >>> >>> >>> >>> osd >>> dev bluestore_extent_map_shard_max_size 200 >>> >>> >>> >>> >>> osd >>> dev bluestore_extent_map_shard_min_size 50 >>> >>> >>> >>> >>> osd >>> dev bluestore_extent_map_shard_target_size 100 >>> >>> >>> >>> >>> osd >>> advanced bluestore_rocksdb_options >>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >>> * >>> osd >>> advanced mon_osd_cache_size 1024 >>> >>> >>> >>> >>> osd >>> dev ms_crc_data false >>> >>> >>> >>> >>> osd >>> advanced osd_map_share_max_epochs 5 >>> >>> >>> >>> >>> osd >>> advanced osd_max_backfills 1 >>> >>> >>> >>> >>> osd >>> dev osd_max_pg_log_entries 10 >>> >>> >>> >>> >>> osd >>> dev osd_memory_cache_min 3000000000 >>> >>> >>> >>> >>> osd >>> host:cd133-ceph-osdh-01 basic osd_memory_target >>> 5797322096 >>> >>> >>> >>> osd >>> host:cd133k-ceph-osdh-01 basic osd_memory_target >>> 9402402385 >>> >>> >>> >>> osd host:cd88-ceph-osdh-01 >>> basic osd_memory_target 5797322096 >>> >>> >>> >>> >>> osd >>> advanced osd_memory_target_autotune true >>> >>> >>> >>> >>> osd >>> dev osd_min_pg_log_entries 10 >>> >>> >>> >>> >>> osd >>> advanced osd_op_num_shards 8 >>> >>> >>> >>> >>> >>> * >>> osd >>> advanced osd_op_num_threads_per_shard 2 >>> >>> >>> >>> >>> >>> * >>> osd >>> dev osd_pg_log_dups_tracked 10 >>> >>> >>> >>> >>> osd >>> dev osd_pg_log_trim_min 10 >>> >>> >>> >>> >>> osd >>> advanced osd_recovery_max_active 3 >>> >>> >>> >>> >>> osd >>> advanced osd_recovery_max_single_start 1 >>> >>> >>> >>> >>> osd >>> advanced osd_recovery_sleep 0.000000 >>> >>> >>> >>> >>> client >>> advanced rbd_cache false >>> >>> >>> >>> >>> >>> >>> >>> Am Do., 7. Okt. 2021 um 19:27 Uhr schrieb Igor Fedotov < >>> igor.fedotov@xxxxxxxx>: >>> >>>> And does redeployed osd.2 expose the same issue (or at least DB/WAL >>>> disbalance) again? Were settings reverted to defaults for it as well? >>>> >>>> >>>> Thanks >>>> >>>> Igor >>>> On 10/7/2021 12:46 PM, José H. Freidhof wrote: >>>> >>>> Good morning, >>>> >>>> i checked today the osd.8 and the log shows again the same error >>>> bluefs _allocate unable to allocate 0x100000 on bdev 0, allocator name >>>> bluefs-wal, allocator type hybrid, capacity 0xb40000000, block size >>>> 0x100000, free 0xff000, fragmentation 0, allocated 0x0 >>>> >>>> any idea why that could be? >>>> >>>> Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof < >>>> harald.freidhof@xxxxxxxxxxxxxx>: >>>> >>>>> Hi Igor, >>>>> >>>>> today i repaired one osd node and all osd´s on the node, creating them >>>>> new again.... >>>>> after that i waited for the rebalance/recovery process and the cluster >>>>> was healthy after some hours.. >>>>> >>>>> i notices that the osd.2 does not have any more this error in the log. >>>>> but i noticed it now on the same node on osd.8... so i did the test >>>>> that you suggested on osd.8 >>>>> >>>>> it took nearly 20minutes to compact those db´s on the bluestore, but >>>>> it helped... the problem on osd.8 is gone... >>>>> >>>>> >>>>> *so the problem that i have with the alloc on the wal device seems to >>>>> be random on different nodes and osd´s and looks like it comes, stay >>>>> a while and disappears after a longer while... * >>>>> >>>>> here are the results that you suggested: >>>>> >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device >>>>> info >>>>> { >>>>> "dev": { >>>>> "device": "BDEV_WAL", >>>>> "total": 48318377984, >>>>> "free": 1044480, >>>>> "bluefs_used": 48317333504 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_DB", >>>>> "total": 187904811008, >>>>> "free": 79842762752, >>>>> "bluefs_used": 108062048256 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_SLOW", >>>>> "total": 6001172414464, >>>>> "free": 5510727389184, >>>>> "bluefs_used": 0, >>>>> "bluefs max available": 5508815847424 >>>>> } >>>>> } >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats >>>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB) >>>>> 1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB) >>>>> 2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB) >>>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>>> Usage matrix: >>>>> DEV/LEV WAL DB SLOW * * >>>>> REAL FILES >>>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B >>>>> 9.7 MiB 1 >>>>> WAL 45 GiB 100 GiB 0 B 0 B 0 B >>>>> 144 GiB 2319 >>>>> DB 0 B 276 MiB 0 B 0 B 0 B >>>>> 249 MiB 47 >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B 0 >>>>> TOTALS 45 GiB 109 GiB 0 B 0 B 0 B >>>>> 0 B 2367 >>>>> MAXIMUMS: >>>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B >>>>> 20 MiB >>>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>>> 192 GiB >>>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>>> 738 MiB >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B >>>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B >>>>> 0 B >>>>> >>>>> --- >>>>> >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb: >>>>> [db_impl/db_impl_write.cc:1668] [L] New memtable created with log file: >>>>> #13656. Immutable memtables: 1. >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to >>>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type >>>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000, >>>>> fragmentation 0, allocated 0x0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time >>>>> 2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198] >>>>> Calling FlushMemTableToOutputFile with column family [L], flush slots >>>>> available 1, compaction slots available 1, flush slots scheduled 1, >>>>> compaction slots scheduled 0 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321] >>>>> [L] [JOB 8859] Flushing memtable with next log file: 13655 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321] >>>>> [L] [JOB 8859] Flushing memtable with next log file: 13656 >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1 >>>>> {"time_micros": 1633549894998273, "job": 8859, "event": "flush_started", >>>>> "num_memtables": 2, "num_entries": 3662, "num_deletes": 0, >>>>> "total_data_size": 130482337, "memory_usage": 132976224, "flush_reason": >>>>> "Write Buffer Full"} >>>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350] >>>>> [L] [JOB 8859] Level-0 flush table #13657: started >>>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1 >>>>> {"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event": >>>>> "table_file_creation", "file_number": 13657, "file_size": 2952537, >>>>> "table_properties": {"data_size": 2951222, "index_size": 267, >>>>> "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": >>>>> 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size": >>>>> 1120, "raw_average_key_size": 16, "raw_value_size": 2950151, >>>>> "raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70, >>>>> "num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0, >>>>> "format_version": 0, "fixed_key_len": 0, "filter_policy": >>>>> "rocksdb.BuiltinBloomFilter", "column_family_name": "L", >>>>> "column_family_id": 10, "comparator": "leveldb.BytewiseComparator", >>>>> "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", >>>>> "property_collectors": "[]", "compression": "NoCompression", >>>>> "compression_options": "window_bits=-14; level=32767; strategy=0; >>>>> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": >>>>> 1633549854, "oldest_key_time": 1633549854, "file_creation_time": >>>>> 1633549894}} >>>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug >>>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401] >>>>> [L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:~# ceph osd set noout >>>>> root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8 >>>>> root@cd88-ceph-osdh-01:~# ceph orch ps >>>>> ... >>>>> osd.7 cd133-ceph-osdh-01 >>>>> running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd >>>>> 8a98ae61f0eb >>>>> osd.8 cd88-ceph-osdh-01 >>>>> stopped 5s ago - - 5528M <unknown> <unknown> >>>>> <unknown> >>>>> osd.9 cd133k-ceph-osdh-01 >>>>> running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd >>>>> 0ff7584b1808 >>>>> ... >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact >>>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 0 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_db_and_around read-only:0 repair:0 >>>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>>> discard not supported >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB) >>>>> rotational discard not supported >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 1 path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175 >>>>> GiB >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>>> discard not supported >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>>> size 5.5 TiB >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB) >>>>> non-rotational discard supported >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 0 path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45 >>>>> GiB >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, >>>>> id 0, allocator name bluefs-wal, allocator type hybrid, capacity >>>>> 0xb40000000, block size 0x100000 >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, >>>>> id 1, allocator name bluefs-db, allocator type hybrid, capacity >>>>> 0x2bc0000000, block size 0x100000 >>>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc >>>>> shared, id 2, capacity 0x57541c00000, block size 0x10000 >>>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount >>>>> shared_bdev_used = 0 >>>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _prepare_db_environment set db_paths to db,178509578240 >>>>> db.slow,5701113793740 >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_db opened rocksdb path db options >>>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_super_meta old nid_max 167450 >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_super_meta old blobid_max 30720 >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_super_meta freelist_type bitmap >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_super_meta ondisk_format 4 compat_ondisk_format 3 >>>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_super_meta min_alloc_size 0x1000 >>>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init >>>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg >>>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _init_alloc opening allocation metadata >>>>> 2021-10-06T22:01:41.031+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid, >>>>> capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000, >>>>> fragmentation 2.76445e-05 >>>>> 2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount >>>>> 2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close >>>>> 2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close >>>>> 2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open >>>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB) >>>>> rotational discard not supported >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 1 path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175 >>>>> GiB >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size >>>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational >>>>> discard not supported >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block >>>>> size 5.5 TiB >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open >>>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB) >>>>> non-rotational discard supported >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device >>>>> bdev 0 path >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45 >>>>> GiB >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, >>>>> id 0, allocator name bluefs-wal, allocator type hybrid, capacity >>>>> 0xb40000000, block size 0x100000 >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, >>>>> id 1, allocator name bluefs-db, allocator type hybrid, capacity >>>>> 0x2bc0000000, block size 0x100000 >>>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc >>>>> shared, id 2, capacity 0x57541c00000, block size 0x10000 >>>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount >>>>> shared_bdev_used = 0 >>>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _prepare_db_environment set db_paths to db,178509578240 >>>>> db.slow,5701113793740 >>>>> 2021-10-06T22:09:36.519+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) >>>>> _open_db opened rocksdb path db options >>>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 >>>>> 2021-10-06T22:09:54.067+0200 7f87bde3c240 1 >>>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount >>>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount >>>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close >>>>> 2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close >>>>> 2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>>> 2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown >>>>> 2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800 >>>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8 >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:/# ceph -s >>>>> cluster: >>>>> id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97 >>>>> health: HEALTH_OK >>>>> >>>>> services: >>>>> mon: 3 daemons, quorum >>>>> cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h) >>>>> mgr: cd133-ceph-mon-01.mzapob(active, since 15h), >>>>> standbys: cd133k-ceph-mon-01.imikwh >>>>> osd: 60 osds: 60 up (since 2m), 60 in (since 3h) >>>>> rgw: 4 daemons active (2 hosts, 1 zones) >>>>> tcmu-runner: 10 portals active (2 hosts) >>>>> >>>>> data: >>>>> pools: 6 pools, 361 pgs >>>>> objects: 2.46M objects, 8.0 TiB >>>>> usage: 33 TiB used, 304 TiB / 338 TiB avail >>>>> pgs: 361 active+clean >>>>> >>>>> io: >>>>> client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr >>>>> >>>>> >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device >>>>> info >>>>> { >>>>> "dev": { >>>>> "device": "BDEV_WAL", >>>>> "total": 48318377984, >>>>> >>>>> * "free": 41354784768, * "bluefs_used": 6963593216 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_DB", >>>>> "total": 187904811008, >>>>> "free": 187302928384, >>>>> "bluefs_used": 601882624 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_SLOW", >>>>> "total": 6001172414464, >>>>> "free": 5507531620352, >>>>> "bluefs_used": 0, >>>>> "bluefs max available": 5505566572544 >>>>> } >>>>> } >>>>> >>>>> --- >>>>> >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats >>>>> 0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB) >>>>> 1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB) >>>>> 2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB) >>>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>>> Usage matrix: >>>>> DEV/LEV WAL DB SLOW * * >>>>> REAL FILES >>>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B >>>>> 10 MiB 0 >>>>> WAL 6.5 GiB 0 B 0 B 0 B 0 B >>>>> 6.4 GiB 102 >>>>> DB 0 B 573 MiB 0 B 0 B 0 B >>>>> 557 MiB 22 >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B 0 >>>>> TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B >>>>> 0 B 125 >>>>> MAXIMUMS: >>>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B >>>>> 17 MiB >>>>> WAL 45 GiB 101 GiB 0 B 0 B 0 B >>>>> 145 GiB >>>>> DB 0 B 688 MiB 0 B 0 B 0 B >>>>> 670 MiB >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B >>>>> TOTALS 45 GiB 101 GiB 0 B 0 B 0 B >>>>> 0 B >>>>> >>>>> ---- >>>>> >>>>> >>>>> Here are the osd.2... the problem disapeared from alone >>>>> very strange... >>>>> >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats >>>>> 0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB) >>>>> 1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB) >>>>> 2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB) >>>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472 >>>>> Usage matrix: >>>>> DEV/LEV WAL DB SLOW * * >>>>> REAL FILES >>>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B >>>>> 10 MiB 1 >>>>> WAL 31 GiB 17 GiB 0 B 0 B 0 B >>>>> 48 GiB 765 >>>>> DB 0 B 193 MiB 0 B 0 B 0 B >>>>> 175 MiB 30 >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B 0 >>>>> TOTALS 32 GiB 21 GiB 0 B 0 B 0 B >>>>> 0 B 796 >>>>> MAXIMUMS: >>>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B >>>>> 17 MiB >>>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>>> 192 GiB >>>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>>> 741 MiB >>>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>> 0 B >>>>> TOTALS 45 GiB 153 GiB 0 B 0 B 0 B >>>>> 0 B >>>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device >>>>> info >>>>> { >>>>> "dev": { >>>>> "device": "BDEV_WAL", >>>>> "total": 48318377984, >>>>> "free": 15043915776, >>>>> "bluefs_used": 33274462208 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_DB", >>>>> "total": 187904811008, >>>>> "free": 169235963904, >>>>> "bluefs_used": 18668847104 >>>>> }, >>>>> "dev": { >>>>> "device": "BDEV_SLOW", >>>>> "total": 6001172414464, >>>>> "free": 5602453327872, >>>>> "bluefs_used": 0, >>>>> "bluefs max available": 5600865222656 >>>>> } >>>>> } >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov < >>>>> igor.fedotov@xxxxxxxx>: >>>>> >>>>>> >>>>>> On 10/6/2021 4:25 PM, José H. Freidhof wrote: >>>>>> > hi, >>>>>> > >>>>>> > no risk no fun 😂 okay >>>>>> > I have reset the settings you mentioned to standard. >>>>>> > >>>>>> > what you exactly mean with taking offline the osd? ceph orch daemon >>>>>> stop >>>>>> > osd.2? or mark down? >>>>>> "daemon stop" is enough. You might want to set noout flag before >>>>>> that >>>>>> though... >>>>>> > >>>>>> > for the command which path i use? you mean: >>>>>> > >>>>>> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact??? >>>>>> yep >>>>>> > >>>>>> > >>>>>> > Igor Fedotov <ifedotov@xxxxxxx> schrieb am Mi., 6. Okt. 2021, >>>>>> 13:33: >>>>>> > >>>>>> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote: >>>>>> >>> Hi Igor, >>>>>> >>> >>>>>> >>> yes i have some osd settings set :-) here are my ceph config >>>>>> dump. those >>>>>> >>> settings are from a redhat document for bluestore devices >>>>>> >>> maybe it is that setting causing this problem? "advanced >>>>>> >>> mon_compact_on_trim false"??? >>>>>> >> OMG!!! >>>>>> >> >>>>>> >> No - mon_compact_on_trim has nothing to deal with bluestore. >>>>>> >> >>>>>> >> Highly likely it's bluestore_rocksdb_options which hurts... >>>>>> >> Documentations tend to fall behind the best practices.... I would >>>>>> >> strongly discourage you from using non-default settings unless it's >>>>>> >> absolutely clear why this is necessary. >>>>>> >> >>>>>> >> Even at the first glance the following settings (just a few ones >>>>>> I'm >>>>>> >> completely aware) are suboptimal/non-recommended: >>>>>> >> >>>>>> >> rocksdb_perf >>>>>> >> >>>>>> >> bluefs_sync_write >>>>>> >> >>>>>> >> bluefs_csum_type >>>>>> >> >>>>>> >> >>>>>> >> Not to mention bluestore_rocksdb_options which hasn't got much >>>>>> adoption >>>>>> >> so far and apparently greatly alters rocksdb behavior... >>>>>> >> >>>>>> >> >>>>>> >> So I would suggest to revert rocksdb options back to default, run >>>>>> the >>>>>> >> compaction and if it succeeds monitor the OSD for a while. Then if >>>>>> it >>>>>> >> works fine - apply the same for others >>>>>> >> >>>>>> >> >>>>>> >> Hope this helps, >>>>>> >> >>>>>> >> Igor >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >>> i will test it this afternoon... at the moment are everything semi >>>>>> >>> prodcuctive and i need to repair one osd node.. because i think >>>>>> of this >>>>>> >>> reason the osds crashed on the node and the osd container crashes >>>>>> with a >>>>>> >>> dump while coming up now. >>>>>> >>> need first to replicate all between all three nodes and then i >>>>>> can take >>>>>> >>> offline the osd.2.and test your command. i will inform you >>>>>> later... >>>>>> >>> >>>>>> >>> root@cd88-ceph-osdh-01:/# ceph config dump >>>>>> >>> WHO MASK >>>>>> >>> LEVEL OPTION VALUE >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> RO >>>>>> >>> global >>>>>> >>> advanced leveldb_max_open_files 131072 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced mon_compact_on_trim false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> dev ms_crc_data false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced osd_deep_scrub_interval >>>>>> 1209600.000000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced osd_max_scrubs 16 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced osd_scrub_load_threshold 0.010000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced osd_scrub_max_interval >>>>>> 1209600.000000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced osd_scrub_min_interval >>>>>> 86400.000000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced perf true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced rbd_readahead_disable_after_bytes 0 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced rbd_readahead_max_bytes 4194304 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced rocksdb_perf true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> global >>>>>> >>> advanced throttler_perf_counter false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> mon >>>>>> >>> advanced auth_allow_insecure_global_id_reclaim false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> mon >>>>>> >>> advanced cluster_network >>>>>> 10.50.50.0/24 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mon >>>>>> >>> advanced mon_osd_down_out_interval 300 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> mon >>>>>> >>> advanced public_network >>>>>> 10.50.50.0/24 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mgr >>>>>> >>> advanced mgr/cephadm/container_init True >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mgr >>>>>> >>> advanced mgr/cephadm/device_enhanced_scan true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mgr >>>>>> >>> advanced mgr/cephadm/migration_current 2 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mgr >>>>>> >>> advanced mgr/cephadm/warn_on_stray_daemons false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> mgr >>>>>> >>> advanced mgr/cephadm/warn_on_stray_hosts false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> osd >>>>>> >>> advanced bluefs_sync_write true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_cache_autotune true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_cache_kv_ratio 0.200000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_cache_meta_ratio 0.800000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_cache_size 2147483648 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_cache_size_hdd 2147483648 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced bluestore_csum_type none >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_extent_map_shard_max_size 200 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_extent_map_shard_min_size 50 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev bluestore_extent_map_shard_target_size 100 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced bluestore_rocksdb_options >>>>>> >>> >>>>>> >> >>>>>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB >>>>>> >>> * >>>>>> >>> osd >>>>>> >>> advanced mon_osd_cache_size 1024 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev ms_crc_data false >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_map_share_max_epochs 5 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_max_backfills 1 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev osd_max_pg_log_entries 10 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev osd_memory_cache_min 3000000000 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >> host:cd133-ceph-osdh-01 >>>>>> >>> basic osd_memory_target 5797322383 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >> host:cd133k-ceph-osdh-01 >>>>>> >>> basic osd_memory_target >>>>>> 9402402385 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> host:cd88-ceph-osdh-01 >>>>>> >>> basic osd_memory_target >>>>>> 5797322096 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_memory_target_autotune true >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev osd_min_pg_log_entries 10 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_op_num_shards 8 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> osd >>>>>> >>> advanced osd_op_num_threads_per_shard 2 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> * >>>>>> >>> osd >>>>>> >>> dev osd_pg_log_dups_tracked 10 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> dev osd_pg_log_trim_min 10 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_recovery_max_active 3 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_recovery_max_single_start 1 >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> osd >>>>>> >>> advanced osd_recovery_sleep 0.000000 >>>>>> >>> >>>>>> >>> >>>>>> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov < >>>>>> ifedotov@xxxxxxx >>>>>> >>> : >>>>>> >>> >>>>>> >>>> Jose, >>>>>> >>>> >>>>>> >>>> In fact 48GB is a way too much for WAL drive - usually the write >>>>>> ahead >>>>>> >> log >>>>>> >>>> tend to be 2-4 GBs. >>>>>> >>>> >>>>>> >>>> But in your case it's ~150GB, while DB itself is very small >>>>>> (146MB!!!): >>>>>> >>>> >>>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B >>>>>> >>>> 154 GiB 2400 >>>>>> >>>> >>>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B >>>>>> >>>> 146 MiB 30 >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> which means that there are some issues with RocksDB's WAL >>>>>> processing, >>>>>> >>>> which needs some troubleshooting... >>>>>> >>>> >>>>>> >>>> Curious if other OSDs are suffering from the same and whether >>>>>> you have >>>>>> >> any >>>>>> >>>> custom settings for your OSD(s)? >>>>>> >>>> >>>>>> >>>> Additionally you might want to try the following command to >>>>>> compact this >>>>>> >>>> specific OSD manually and check if this would normalize the DB >>>>>> layout - >>>>>> >> the >>>>>> >>>> majority of data has to be at DB level not WAL. Please share the >>>>>> >> resulting >>>>>> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command) >>>>>> after the >>>>>> >>>> compaction is fulfiled and OSD is restarted. >>>>>> >>>> >>>>>> >>>> The compaction command to be applied on an offline OSD: >>>>>> >> "ceph-kvstore-tool >>>>>> >>>> bluestore-kv <path-to-osd> compact" >>>>>> >>>> >>>>>> >>>> Even if the above works great please refrain from applying that >>>>>> >> compaction >>>>>> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would >>>>>> WAL grow >>>>>> >>>> again or not? >>>>>> >>>> >>>>>> >>>> Thanks, >>>>>> >>>> >>>>>> >>>> Igor >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote: >>>>>> >>>> >>>>>> >>>> Hello Igor, >>>>>> >>>> >>>>>> >>>> yes the volume is nvme wal partitions for the bluestore >>>>>> devicegroups >>>>>> >> are >>>>>> >>>> only 48gb each >>>>>> >>>> >>>>>> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with >>>>>> 48gb (WAL) >>>>>> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb >>>>>> >> (rock.db) >>>>>> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db) >>>>>> >>>> >>>>>> >>>> each blustore have 1 partition nvme,ssd and hdd like described >>>>>> in the >>>>>> >>>> documentation >>>>>> >>>> >>>>>> >> >>>>>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/ >>>>>> >>>> is this to small or can i adjust the max allocation on the wal >>>>>> nvme >>>>>> >> device >>>>>> >>>> in the ceph configuration? >>>>>> >>>> i know that the ssd and nvme are to small for those 5.5tb >>>>>> disk... its 1% >>>>>> >>>> only ot the rotation disk. >>>>>> >>>> i am new in ceph and still or always learning, but we are in a >>>>>> little >>>>>> >>>> hurry because our other datastores are old and full. >>>>>> >>>> >>>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs >>>>>> device >>>>>> >> info >>>>>> >>>> { >>>>>> >>>> "dev": { >>>>>> >>>> "device": "BDEV_WAL", >>>>>> >>>> "total": 48318377984, >>>>>> >>>> "free": 1044480, >>>>>> >>>> "bluefs_used": 48317333504 >>>>>> >>>> }, >>>>>> >>>> "dev": { >>>>>> >>>> "device": "BDEV_DB", >>>>>> >>>> "total": 187904811008, >>>>>> >>>> "free": 68757217280, >>>>>> >>>> "bluefs_used": 119147593728 >>>>>> >>>> }, >>>>>> >>>> "dev": { >>>>>> >>>> "device": "BDEV_SLOW", >>>>>> >>>> "total": 6001172414464, >>>>>> >>>> "free": 5624912359424, >>>>>> >>>> "bluefs_used": 0, >>>>>> >>>> "bluefs max available": 5624401231872 >>>>>> >>>> } >>>>>> >>>> } >>>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats >>>>>> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB) >>>>>> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB) >>>>>> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB) >>>>>> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976, >>>>>> >> db_total:178509578240, >>>>>> >>>> slow_total:5701113793740, db_avail:103884521472 >>>>>> >>>> Usage matrix: >>>>>> >>>> DEV/LEV WAL DB SLOW * * >>>>>> >>>> REAL FILES >>>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B >>>>>> >>>> 7.5 MiB 1 >>>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B >>>>>> >>>> 154 GiB 2400 >>>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B >>>>>> >>>> 146 MiB 30 >>>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>>> >> 0 >>>>>> >>>> B 0 >>>>>> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B >>>>>> >> 0 >>>>>> >>>> B 2431 >>>>>> >>>> MAXIMUMS: >>>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B >>>>>> >> 17 >>>>>> >>>> MiB >>>>>> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B >>>>>> >>>> 192 GiB >>>>>> >>>> DB 0 B 762 MiB 0 B 0 B 0 B >>>>>> >>>> 741 MiB >>>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B >>>>>> >> 0 B >>>>>> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B >>>>>> >> 0 B >>>>>> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov < >>>>>> >> ifedotov@xxxxxxx>: >>>>>> >>>>> Hey Jose, >>>>>> >>>>> >>>>>> >>>>> it looks like your WAL volume is out of space which looks weird >>>>>> given >>>>>> >>>>> its capacity = 48Gb. >>>>>> >>>>> >>>>>> >>>>> Could you please share the output of the following commands: >>>>>> >>>>> >>>>>> >>>>> ceph daemon osd.N bluestore bluefs device info >>>>>> >>>>> >>>>>> >>>>> ceph daemon osd.N bluefs stats >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> Thanks, >>>>>> >>>>> >>>>>> >>>>> Igor >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote: >>>>>> >>>>>> Hello together >>>>>> >>>>>> >>>>>> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this >>>>>> >> messages >>>>>> >>>>> in >>>>>> >>>>>> the service logs of the osd daemons. >>>>>> >>>>>> >>>>>> >>>>>> we have three osd nodes .. each node has 20osds as bluestore >>>>>> with >>>>>> >>>>>> nvme/ssd/hdd >>>>>> >>>>>> >>>>>> >>>>>> is this a bug or maybe i have some settings wrong? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:25.821+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:29.857+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.073+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.405+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.465+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.529+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] >>>>>> New >>>>>> >>>>> memtable >>>>>> >>>>>> created with log file: #9588. Immutable memtables: 1. >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 >>>>>> on bdev >>>>>> >> 0, >>>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity >>>>>> >> 0xb40000000, >>>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated >>>>>> 0x0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.547575) >>>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling >>>>>> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots >>>>>> >> available >>>>>> >>>>> 1, >>>>>> >>>>>> compaction slots available 1, flush slots scheduled 1, >>>>>> compaction >>>>>> >> slots >>>>>> >>>>>> scheduled 0 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] >>>>>> Flushing >>>>>> >>>>>> memtable with next log file: 9587 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] >>>>>> Flushing >>>>>> >>>>>> memtable with next log file: 9588 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850547916, >>>>>> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2, >>>>>> >>>>> "num_entries": >>>>>> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926, >>>>>> "memory_usage": >>>>>> >>>>>> 130479920, "flush_reason": "Write Buffer Full"} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.545+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] >>>>>> Level-0 >>>>>> >> flush >>>>>> >>>>>> table #9589: started >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850559292, >>>>>> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation", >>>>>> >>>>> "file_number": >>>>>> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size": >>>>>> 3247855, >>>>>> >>>>>> "index_size": 1031, "index_partitions": 0, >>>>>> "top_level_index_size": 0, >>>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, >>>>>> >>>>>> "filter_size": 197, "raw_key_size": 1088, >>>>>> "raw_average_key_size": 16, >>>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739, >>>>>> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32, >>>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, >>>>>> "format_version": >>>>>> >> 0, >>>>>> >>>>>> "fixed_key_len": 0, "filter_policy": >>>>>> "rocksdb.BuiltinBloomFilter", >>>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, >>>>>> "comparator": >>>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr", >>>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": >>>>>> "[]", >>>>>> >>>>>> "compression": "NoCompression", "compression_options": >>>>>> >> "window_bits=-14; >>>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; >>>>>> zstd_max_train_bytes=0; >>>>>> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time": >>>>>> >>>>> 1633511730, >>>>>> >>>>>> "file_creation_time": 1633511850}} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] >>>>>> Level-0 >>>>>> >> flush >>>>>> >>>>>> table #9589: 3249934 bytes OK >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.559362) >>>>>> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.559583) >>>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: >>>>>> memtable #1 >>>>>> >> done >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.559586) >>>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: >>>>>> memtable #2 >>>>>> >> done >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.559601) >>>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709, >>>>>> "event": >>>>>> >>>>>> "flush_finished", "output_compression": "NoCompression", >>>>>> "lsm_state": >>>>>> >>>>> [8, >>>>>> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.559638) >>>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary: >>>>>> files[8 1 >>>>>> >>>>> 0 0 >>>>>> >>>>>> 0 0 0] max score 1.00 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] >>>>>> [L] [JOB >>>>>> >>>>> 5710] >>>>>> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00 >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] >>>>>> [L] >>>>>> >>>>> Compaction >>>>>> >>>>>> start summary: Base version 3090 Base level 0, inputs: >>>>>> [9589(3173KB) >>>>>> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB) >>>>>> >>>>> 9570(405KB) >>>>>> >>>>>> 9567(29KB)], [9564(1115KB)] >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.557+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850559956, >>>>>> >>>>>> "job": 5710, "event": "compaction_started", >>>>>> "compaction_reason": >>>>>> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, >>>>>> 9573, >>>>>> >>>>> 9570, >>>>>> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size": >>>>>> 19542092} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] >>>>>> [L] [JOB >>>>>> >>>>> 5710] >>>>>> >>>>>> Generated table #9590: 36 keys, 3249524 bytes >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850582987, >>>>>> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation", >>>>>> >>>>> "file_number": >>>>>> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size": >>>>>> 3247449, >>>>>> >>>>>> "index_size": 1031, "index_partitions": 0, >>>>>> "top_level_index_size": 0, >>>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, >>>>>> >>>>>> "filter_size": 197, "raw_key_size": 576, >>>>>> "raw_average_key_size": 16, >>>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173, >>>>>> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0, >>>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0, >>>>>> "format_version": >>>>>> >> 0, >>>>>> >>>>>> "fixed_key_len": 0, "filter_policy": >>>>>> "rocksdb.BuiltinBloomFilter", >>>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, >>>>>> "comparator": >>>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr", >>>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": >>>>>> "[]", >>>>>> >>>>>> "compression": "NoCompression", "compression_options": >>>>>> >> "window_bits=-14; >>>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0; >>>>>> zstd_max_train_bytes=0; >>>>>> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": >>>>>> 0, >>>>>> >>>>>> "file_creation_time": 1633511850}} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] >>>>>> [L] [JOB >>>>>> >>>>> 5710] >>>>>> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.583469) >>>>>> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 >>>>>> 0 0 0 0 >>>>>> >>>>> 0] >>>>>> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files >>>>>> in(8, 1) >>>>>> >>>>> out(1) >>>>>> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2) >>>>>> write-amplify(0.2) >>>>>> >>>>> OK, >>>>>> >>>>>> records in: 376, records dropped: 340 output_compression: >>>>>> >> NoCompression >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time >>>>>> >> 2021/10/06-09:17:30.583498) >>>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710, >>>>>> "event": >>>>>> >>>>>> "compaction_finished", "compaction_time_micros": 23098, >>>>>> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1, >>>>>> >>>>> "num_output_files": >>>>>> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376, >>>>>> >>>>>> "num_output_records": 36, "num_subcompactions": 1, >>>>>> >> "output_compression": >>>>>> >>>>>> "NoCompression", "num_single_delete_mismatches": 0, >>>>>> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, >>>>>> 0, 0, >>>>>> >> 0]} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583615, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9589} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583648, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9586} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583675, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9583} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583709, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9580} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583739, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9576} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583769, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9573} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583804, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9570} >>>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug >>>>>> 2021-10-06T09:17:30.581+0000 >>>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": >>>>>> >> 1633511850583835, >>>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": >>>>>> 9567} >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>>> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>>> >>>> -- >>>>>> >>>>> >>>>> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |