Hi Nick,
That's expected since jewel, which moved the scrub IOs out of the disk
thread and into the ?op? thread. They can now be prioritized using
osd_scrub_priority, and you can experiment with osd_op_queue =
prio/wpq to see if scrubs can be made more transparent with the
latter, newer, queuing implementation.
I don't recall what's still left in the disk thread -- perhaps snap
trimming or PG removal.
Cheers, Dan
On Tue, Nov 8, 2016 at 9:16 AM, Nick Fisk <nick@xxxxxxxxxx> wrote:
> Hi,
>
> I have all the normal options set in ceph.conf (priority and class for disk threads) however scrubs look like they are running as
> the standard BE/4 class in iotop. Running 10.2.3.
>
> Eg
>
> PG Dump (Shows that OSD 1 will be scrubbing)
> pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up
> up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp
> 1.8fc 3314 0 0 0 0 13899923456 3043 3043 active+clean+scrubbing+deep 2016-11-08
> 07:51:25.708169 1468442'173537 1468443:139543 [15,32,1] 15 [15,32,1] 15 1435301'169725 2016-11-06
> 21:46:26.537632 1338333'158008 2016-10-30 23:59:50.794774
>
> sudo iotop -o --batch --iter 1 | grep ceph
> 3958453 be/4 ceph 92.17 M/s 0.00 B/s 0.00 % 66.59 % ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
> [tp_osd_tp]
> 3956201 be/4 ceph 12.47 K/s 0.00 B/s 0.00 % 4.72 % ceph-osd -f --cluster ceph --id 11 --setuser ceph --setgroup ceph
> [tp_osd_tp]
> 3954970 be/4 ceph 361.68 K/s 0.00 B/s 0.00 % 1.98 % ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
> [tp_osd_tp]
> 3956653 be/4 ceph 12.47 K/s 0.00 B/s 0.00 % 1.26 % ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
> [tp_osd_tp]
> 3957297 be/4 ceph 12.47 K/s 0.00 B/s 0.00 % 0.60 % ceph-osd -f --cluster ceph --id 7 --setuser ceph --setgroup ceph
> [tp_osd_tp]
> 3956648 be/4 ceph 12.47 K/s 0.00 B/s 0.00 % 0.14 % ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
> [tp_osd_tp]
>
> Only OSD1 is doing IO, the rest are doing nothing in comparison, so this must be the scrubbing, but why is it not running as idle?
>
> There are idle threads though, they just don't seem to do anything
>
> iotop --batch --iter 1 | grep ceph | grep idle
> 3954974 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3956209 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 11 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3956655 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3957253 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3957306 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 7 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3957350 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 10 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3958464 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3958505 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3958832 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3960488 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3960851 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph
> [tp_osd_disk]
> 3959437 idle ceph 0.00 B/s 0.00 B/s 0.00 % 0.00 % ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
> [tp_osd_disk]
>
> ceph daemon osd.1 config get osd_disk_thread_ioprio_class
> {
> "osd_disk_thread_ioprio_class": "idle"
> }
> ceph daemon osd.1 config get osd_disk_thread_ioprio_priority
> {
> "osd_disk_thread_ioprio_priority": "7"
> }
>
> cat /sys/block/sd*/queue/scheduler
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
> noop deadline [cfq]
>
> Any Ideas or have I misunderstood something?
> Nick
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
