Hi Frank, Thanks for your insight on this. I had done a bunch of testing on this over a year ago and found improvements with these settings. I then applied them all at once to our production cluster and confirmed the 3x reduction in latency, however I did not test the settings individually. It could well be, as you say, that the settings cannot be changed at runtime and that in fact only the other settings such as op queue and throttle cost are making the difference. I'll attempt to test the settings again this week and see which ones are actually affecting latency during runtime setting. > I'm not sure why with your OSD creation procedure the data part is created with the correct HDD parameters. I believe that at prepare time my osds get all SSD parameters. That's why I manually change the class and these runtime settings. Rich On Sat, 9 Apr 2022 at 00:22, Frank Schilder <frans@xxxxxx> wrote: > > Hi Richard, > > thanks for the additional info, now I understand the whole scenario and what might be different when using lvm and dm_cache. > > > In my process, bcache is added before osd creation as bcache creates a > > disk device called /dev/bcache0 for example. This is used for the data > > This is an important detail. As far as I know, dm_cache is transparent. It can be added/removed at run time and doesn't create a new device. However, I don't know if it changes the rotational attribute of the LVM device. > > I'm not sure why with your OSD creation procedure the data part is created with the correct HDD parameters. I believe at least these if not more parameters are used at prepare time only and cannot be changed after the OSD is created: > > bluestore_prefer_deferred_size = 32768 > bluestore_compression_max_blob_size = 524288 > bluestore_max_blob_size = 524288 > bluestore_min_alloc_size = 65536 > > If you set these for osd/class:hdd they should *not* be used if the initial device class is ssd. If I understood you correctly, you create an OSD with class=ssd and then change its class to class=hdd. At this point, however, it is too late, the hard-coded ssd options should persist. I wonder if using a command like > > ceph-volume lvm batch --crush-device-class hdd ... > > will select the right parameters irrespective of the rotational flag. How did you do it? I believe the only way to get the burned-in bluestore values was to start an OSD with high debug logging. The "config show" commands will show what is in the config DB and not what is burned onto disk (and actually used). > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Richard Bade <hitrich@xxxxxxxxx> > Sent: 08 April 2022 00:08 > To: Frank Schilder > Cc: Igor Fedotov; Ceph Users > Subject: [Warning Possible spam] Re: [Warning Possible spam] Re: [Warning Possible spam] Re: Ceph Bluestore tweaks for Bcache > > Hi Frank, > Yes, I think you have got to the crux of the issue. > > - some_config_value_hdd is used for "rotational=0" devices and > > - osd/class:hdd values are used for "device_class=hdd" OSDs, > > The class is something that is user defined and you can actually > define your own class names. By default the class is set to ssd for > rotational=0 and hdd for rotational=1. I override this so my osds end > up in the right pools as my pools are class based. I also have another > class called nvme for all nvme storage. > So the rotational=0 and the class=ssd are actually disconnected and > used for two different purposes. > > > Or are you observing that an HDD+bcache OSD comes up in device class hdd *but* bluestore thinks it is an ssd and applies SSD defaults (some_config_value_ssd) *unless* you explicitly set the config option for device class hdd? > > Yes, this is what I am observing, because I am manually changing the > device class to HDD. > > > - OSD is prepared on HDD and put into device class hdd (with correct persistent prepare-time options) > > - bcache is added *after* OSD creation (???) > > - after this, on (re-)start the OSD comes up in device class hdd but bluestore thinks now its an SSD and uses some incorrect run-time config option defaults > > - to fix the incorrect run-time options, you explicitly copy some hdd-defaults to the config data base with filter "osd/class:hdd" > > In my process, bcache is added before osd creation as bcache creates a > disk device called /dev/bcache0 for example. This is used for the data > disk. As you have surmised bluestore thinks my disks are ssd and > applies settings as such. I set the class to HDD and then I correct > runtime settings based on the class. > > > There is actually an interesting follow up on this. With bcache/dm_cache large enough it should make sense to use SSD rocks-DB settings, because the data base will fit into the cache. Are there any recommendations for tweaking the prepare-time config options, in particular, the rocks-db options for such hybrid drives? > > In my case, this doesn't apply as I have used volumes on the ssd > specifically for the db. This means I know the db will always be on > the fast storage. > But yes, a larger cache size may change the performance and make it > closer to what ceph expects from an ssd. In my experience the ssd > settings made performance considerably worse than the hdd settings (3x > average latency) on bcache. > > Regards, > Rich > > On Fri, 8 Apr 2022 at 02:03, Frank Schilder <frans@xxxxxx> wrote: > > > > Hi Richard, > > > > so you are tweaking run-time config values, not OSD prepare-time config values. There is something I don't understand here: > > > > > What I do for my settings is to set them for the hdd class (ceph config set osd/class:hdd bluestore_setting_blah=blahblah. > > > I think that's the correct syntax, but I'm not currently at a computer) in the config database. > > > > If the OSD comes up as class=hdd, then the hdd defaults should be applied any way and there is no point setting these values explicitly to their defaults. How do you make the OSD come up in class hdd, wasn't it your original problem that the OSDs came up in class ssd? Or are you observing that an HDD+bcache OSD comes up in device class hdd *but* bluestore thinks it is an ssd and applies SSD defaults (some_config_value_ssd) *unless* you explicitly set the config option for device class hdd? > > > > I think I am confused about the OSD device class, the drive type detected by bluestore and what options are used if there is a mis-match - if there is any. If I understand you correctly, it seems you observe that: > > > > - OSD is prepared on HDD and put into device class hdd (with correct persistent prepare-time options) > > - bcache is added *after* OSD creation (???) > > - after this, on (re-)start the OSD comes up in device class hdd but bluestore thinks now its an SSD and uses some incorrect run-time config option defaults > > - to fix the incorrect run-time options, you explicitly copy some hdd-defaults to the config data base with filter "osd/class:hdd" > > > > If this is correct, then I believe the underlying issue is that: > > > > - some_config_value_hdd is used for "rotational=0" devices and > > - osd/class:hdd values are used for "device_class=hdd" OSDs, > > > > which is not the same despite the string "hdd" indicating that it is. > > > > There is actually an interesting follow up on this. With bcache/dm_cache large enough it should make sense to use SSD rocks-DB settings, because the data base will fit into the cache. Are there any recommendations for tweaking the prepare-time config options, in particular, the rocks-db options for such hybrid drives? > > > > Best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |