Hi Frank, I can't speak for the bluestore debug enforce settings as I don't have this setting but I would guess it's the same. What I do for my settings is to set them for the hdd class (ceph config set osd/class:hdd bluestore_setting_blah=blahblah. I think that's the correct syntax, but I'm not currently at a computer) in the config database. The class is a permanent setting on the osd so when it starts or server reboots it automatically applies these setting based on the osd class. That way any new osds also get them as soon as the class is defined for the osd. Hopefully that helps. Rich On Thu, 7 Apr 2022, 23:40 Frank Schilder, <frans@xxxxxx> wrote: > Hi Richard and Igor, > > are these tweaks required at build-time (osd prepare) only or are they > required for every restart? > > Is this setting "bluestore debug enforce settings=hdd" in the ceph config > data base or set somewhere else? How does this work if deploying HDD- and > SSD-OSDs at the same time? > > Ideally, all these tweaks should be applicable and settable at creation > time only without affecting generic settings (that is, at the ceph-volume > command line and not via config side effects). Otherwise it becomes really > tedious to manage these. > > For example, would the following work-flow apply the correct settings > *permanently* across restarts: > > 1) Prepare OSD on fresh HDD with ceph-volume lvm batch --prepare ... > 2) Assign dm_cache to logical OSD volume created in step 1 > 3) Start OSD, restart OSDs, boot server ... > > I would assume that the HDD settings are burned into the OSD in step 1 and > will be used in all future (re-)starts without the need to do anything > despite the device being detected as non-rotational after step 2. Is this > assumption correct? > > Thanks and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Richard Bade <hitrich@xxxxxxxxx> > Sent: 06 April 2022 00:43:48 > To: Igor Fedotov > Cc: Ceph Users > Subject: [Warning Possible spam] Re: Ceph Bluestore tweaks > for Bcache > > Just for completeness for anyone that is following this thread. Igor > added that setting in Octopus, so unfortunately I am unable to use it > as I am still on Nautilus. > > Thanks, > Rich > > On Wed, 6 Apr 2022 at 10:01, Richard Bade <hitrich@xxxxxxxxx> wrote: > > > > Thanks Igor for the tip. I'll see if I can use this to reduce the > > number of tweaks I need. > > > > Rich > > > > On Tue, 5 Apr 2022 at 21:26, Igor Fedotov <igor.fedotov@xxxxxxxx> wrote: > > > > > > Hi Richard, > > > > > > just FYI: one can use "bluestore debug enforce settings=hdd" config > > > parameter to manually enforce HDD-related settings for a BlueStore > > > > > > > > > Thanks, > > > > > > Igor > > > > > > On 4/5/2022 1:07 AM, Richard Bade wrote: > > > > Hi Everyone, > > > > I just wanted to share a discovery I made about running bluestore on > > > > top of Bcache in case anyone else is doing this or considering it. > > > > We've run Bcache under Filestore for a long time with good results > but > > > > recently rebuilt all the osds on bluestore. This caused some > > > > degradation in performance that I couldn't quite put my finger on. > > > > Bluestore osds have some smarts where they detect the disk type. > > > > Unfortunately in the case of Bcache it detects as SSD, when in fact > > > > the HDD parameters are better suited. > > > > I changed the following parameters to match the HDD default values > and > > > > immediately saw my average osd latency during normal workload drop > > > > from 6ms to 2ms. Peak performance didn't change really, but a test > > > > machine that I have running a constant iops workload was much more > > > > stable as was the average latency. > > > > Performance has returned to Filestore or better levels. > > > > Here are the parameters. > > > > > > > > ; Make sure that we use values appropriate for HDD not SSD - Bcache > > > > gets detected as SSD > > > > bluestore_prefer_deferred_size = 32768 > > > > bluestore_compression_max_blob_size = 524288 > > > > bluestore_deferred_batch_ops = 64 > > > > bluestore_max_blob_size = 524288 > > > > bluestore_min_alloc_size = 65536 > > > > bluestore_throttle_cost_per_io = 670000 > > > > > > > > ; Try to improve responsiveness when some disks are fully utilised > > > > osd_op_queue = wpq > > > > osd_op_queue_cut_off = high > > > > > > > > Hopefully someone else finds this useful. > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > -- > > > Igor Fedotov > > > Ceph Lead Developer > > > > > > Looking for help with your Ceph cluster? Contact us at > https://croit.io > > > > > > croit GmbH, Freseniusstr. 31h, 81247 Munich > > > CEO: Martin Verges - VAT-ID: DE310638492 > > > Com. register: Amtsgericht Munich HRB 231263 > > > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |