Hi Anthony,
Thank you for posting that link. I can see there that the description for that option is:'Inject an expensive sleep during deep scrub IO to make it easier to induce preemption'
Does this mean that it is supposed to be used in conjunction with the 'osd scrub max pre-emption' option? I am also still unclear if this option is strictly meant for developers to debug things or if it's ok to run on a production cluster.
> Are your OSDs HDDs? Using EC?
I am using bcache devices for OSDs so each OSD is a HDD that is fronted by a cache SSD. All pools are replicated, no EC.
> How many deep scrubs do you have running in parallel? Assuming more than one, you could increase osd_deep_scrub_interval to spread them out over time.
I have only 1 deep scrub running per osd ( osd max scrubs = 1). I have already increased the deep scrub interval which does help to spread them out, but when they do eventually happen there is still a noticeable performance impact.
On Tuesday, 7 February 2023 at 14:54:53 GMT, Anthony D'Atri <aad@xxxxxxxxxxxxxx> wrote:
Documented here:
https://github.com/ceph/ceph/blob/9754cafc029e1da83f5ddd4332b69066fe6b3ffb/src/common/options/global.yaml.in#L3202
Introduced back here with a bunch of other scrub tweaks:
https://github.com/ceph/ceph/pull/18971/files
Are your OSDs HDDs? Using EC?
How many deep scrubs do you have running in parallel? Assuming more than one, you could increase osd_deep_scrub_interval to spread them out over time.
> On Feb 7, 2023, at 05:58, Broccoli Bob <brockolibob@xxxxxxxxx> wrote:
>
> I have been running a ceph cluster for a while and one of the main things that impacts performance is deep-scrubbing. I would like to limit this as much as possible and have tried the below options to do this:
>
> osd scrub sleep = 1 # Time to sleep before scrubbing next group of chunks
> osd scrub chunk max = 1 # Maximum number of chunks to scrub during a single operation
> osd scrub chunk min = 1 # Minimum number of chunks to scrub during a single operation
> osd scrub max pre-emption = 30 # Maximum number of times ceph pre-empts a deep scrub due to a client operation blocking it
> osd client op priority = 63 # The priority of client operations
> osd requested scrub priority = 1 # The priority of administrator requested scrubs
> osd scrub priority = 1 # The priority of scheduled scrubs
>
> These options combined do slow down scrubbing and make it easier on client IO, but there is still a performance impact when they happen. One other option I have found is:
> ‘osd debug deep scrub sleep’
>
> Setting this option to even 0.1 will have an immediate and large effect (I have seen nodes that were scrubbing at 50MB/s to down to <5MB/s). The effect is larger than all the above options combined and is exactly what I want, however there is no documentation about this option and I don’t know the full impact of configuring it. It also spams the logs with messages saying that the deep scrub is sleeping for x seconds. So I would like to know:
> Is it safe to have the debug deep scrub option turned on? If so is there a way to stop it from spamming the logs?
> If it is not safe are there any other options available to us to limit the impact of deep scrubbing?
> Thank you and Regards
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
