On 23/08/2023 08:27, Sridhar Seshasayee wrote:
This also leads me to agree with you there's 'something wrong' with
the mclock scheduler. I was almost starting to suspect hardware issues
or something like that, I was at my wit's end.
Could you update this thread with the exact quincy version by running:
$ ceph versions
and
$ ceph config show-with-defaults osd.N | grep osd_mclock
Please replace N with any valid OSD id.
I suspect that the quincy version you are running on doesn't
have the latest changes we made to the Reef upstream release.
Recent changes introduced significant improvements to the
mClock profiles and address slow recovery/backfill rates. The
improvements to the mClock profiles should also help throttle
snaptrim operations.
Snaptrim operation with mClock currently uses a static cost as
defined by osd_snap_trim_cost. There are improvements planned
around this soon. For e.g., the cost must be dynamic and reflect
the size of the object being trimmed.
-Sridhar
Here's the requested info, even though I'm going to stay on wpq for a
while.
# ceph versions
{
"mon": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
quincy (stable)": 3
},
"mgr": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
quincy (stable)": 3
},
"osd": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
quincy (stable)": 117
},
"mds": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
quincy (stable)": 3
},
"overall": {
"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
quincy (stable)": 126
}
}
I'm using the docker image
registry.<company>.local/quay-proxy/ceph/ceph@sha256:673b48521fd53e1b4bc7dda96335505c4d4b2e13d7bb92bf2e7782e2083094c9.
# ceph config show-with-defaults osd.0 | grep osd_mclock
osd_mclock_cost_per_byte_usec 0.000000
default
osd_mclock_cost_per_byte_usec_hdd 2.600000
default
osd_mclock_cost_per_byte_usec_ssd 0.011000
default
osd_mclock_cost_per_io_usec 0.000000
default
osd_mclock_cost_per_io_usec_hdd 11400.000000
default
osd_mclock_cost_per_io_usec_ssd 50.000000
default
osd_mclock_force_run_benchmark_on_init false
default
osd_mclock_iops_capacity_threshold_hdd 500.000000
default
osd_mclock_iops_capacity_threshold_ssd 80000.000000
default
osd_mclock_max_capacity_iops_hdd 250.000000
mon
osd_mclock_max_capacity_iops_ssd 21500.000000
default
osd_mclock_override_recovery_settings true
mon
osd_mclock_profile
high_client_ops
mon
osd_mclock_scheduler_anticipation_timeout 0.000000
default
osd_mclock_scheduler_background_best_effort_lim 999999
default
osd_mclock_scheduler_background_best_effort_res 1
default
osd_mclock_scheduler_background_best_effort_wgt 1
default
osd_mclock_scheduler_background_recovery_lim 999999
default
osd_mclock_scheduler_background_recovery_res 1
default
osd_mclock_scheduler_background_recovery_wgt 1
default
osd_mclock_scheduler_client_lim 999999
default
osd_mclock_scheduler_client_res 1
default
osd_mclock_scheduler_client_wgt 1
default
osd_mclock_skip_benchmark false
default
# ceph config show-with-defaults osd.0 | grep trim_cost
osd_snap_trim_cost 1048576
default
Angelo.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
