Hi Robert,
Thanks for pointing that out. This issue stems from version differences between our cluster environments:
Our current Pacific cluster (migrating to Reef next week) uses the default configuration:
bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P'
While Reef and newer versions use:
bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32'
According to documentation, the bluestore_rocksdb_cfs parameter is only applied during OSD creation (when using --mkfs). Since there was no mention of needing to include L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 in the reshard command, I didn't include these options when resharding the OSDs. Did you?
Additionally, min_write_buffer_number_to_merge appears to be a RocksDB option (bluestore_rocksdb_options) with a default value of 1 in Pacific and 6 in Reef. It's unclear whether the effective value is determined by bluestore_rocksdb_options or by the value specified in bluestore_rocksdb_cfs during OSD creation.
This probably deserves clarification to prevent inconsistent behavior within the same cluster due to OSDs having different min_write_buffer_number_to_merge values from their creation time.
Hopefully Igor can provide some insight on this issue.
Regards,
Frédéric
----- Le 11 Mar 25, à 12:05, Robert Sander r.sander@xxxxxxxxxxxxxxxxxxx a écrit :
> Hi,
>
> Am 3/11/25 um 11:33 schrieb Frédéric Nass:
>
>> $ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path
>> /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12)
>> O(3,0-13)=block_cache={type=binned_lru} L P" reshard
>>
>> As a consequence, the 'Object' key/value pairs were stored in the default column
>> family instead of the intended 'O' column family. Resharding again with the
>> correct capital 'O' resolved this issue for the affected overspilling OSDs.
>>
>> Be aware that there remains an error in both the Quincy and Reef documentation,
>> where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using
>> lowercase 'l' and 'p' wrecks the OSD entirely.
>> I've created this tracker [1] in hopes of getting the Quincy and Reef
>> documentation fixed.
>
> The latest docs seems to have it correct:
>
> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding
>
> Is there a reason why you omitted "=min_write_buffer_number_to_merge=32" after L
> and P?
> These are the default values that I also see on my reef cluster.
>
> Regards
> --
> Robert Sander
> Linux Consultant
>
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: +49 30 405051 - 0
> Fax: +49 30 405051 - 19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
