Re: Limit deep scrub

Karun Josy <karunjosy1@xxxxxxxxx> · Sun, 28 Jan 2018 21:35:51 +0530

Hi,

I used these settings and there are no more slow requests in the cluster.

---------
ceph tell osd.* injectargs '--osd_scrub_sleep 0.1'
ceph tell osd.* injectargs '--osd_scrub_load_threshold 0.3'
ceph tell osd.* injectargs '--osd_scrub_chunk_max 6' 
----------

Yes, scrubbing is slower now, but there has been no osd flapping and slow requests!

Thanks for all your help!

Karun Josy

On Sun, Jan 28, 2018 at 9:25 PM, David Turner <drakonstein@xxxxxxxxx> wrote:
Use a get with the second syntax to see the currently running config.

On Sun, Jan 28, 2018, 3:41 AM Karun Josy <karunjosy1@xxxxxxxxx> wrote:
Hello,
Sorry for bringing this up again.

What is the proper way to adjust the scrub settings ?
Can I use injectargs ?
-------
ceph tell osd.* injectargs '--osd_scrub_sleep .1'
-------

Or do I have to use set manually in each osd daemons ?
---
ceph daemon osd.21 set osd_scrub_sleep .1
----

While using both it shows (not observed, change may require restart)
So is it not set ?

Karun Josy

On Mon, Jan 15, 2018 at 7:16 AM, shadow_lin <shadow_lin@xxxxxxx> wrote:

hi,
you can try to adjusting 
osd_scrub_chunk_min,osd_scrub_chunk_max and 
osd_scrub_sleep.       

  osd scrub sleep

  Description: Time to sleep before 
  scrubbing next group of chunks. Increasing this value will slow down whole 
  scrub operation while client operations will be less impacted.
Type: 
  Float
Default: 0

  osd scrub chunk min

  Description: The minimal number of object store chunks to scrub during 
  single operation. Ceph blocks writes to single chunk during scrub.
Type: 
  32-bit Integer
Default: 5

2018-01-15 

lin.yunfan

  发件人：Karun Josy <karunjosy1@xxxxxxxxx>
  发送时间：2018-01-15 06:53
  主题： Limit deep scrub
  收件人："ceph-users"<ceph-users@lists.ceph.com>
  抄送：

  Hello,

  It appears that cluster is having many slow requests while it is 
  scrubbing and deep scrubbing. Also sometimes we can see 
  osds flapping.

  So we have put the flags : noscrub,nodeep-scrub 

  When we unset it, 5 PGs start to scrub.
  Is there a way to limit it to one at a time?

  # ceph daemon osd.35 config show | grep scrub
      "mds_max_scrub_ops_in_progress": "5",
      "mon_scrub_inject_crc_mismatch": "0.000000",
      "mon_scrub_inject_missing_keys": "0.000000",
      "mon_scrub_interval": "86400",
      "mon_scrub_max_keys": "100",
      "mon_scrub_timeout": "300",
      "mon_warn_not_deep_scrubbed": "0",
      "mon_warn_not_scrubbed": "0",
      "osd_debug_scrub_chance_rewrite_digest": "0",
      "osd_deep_scrub_interval": "604800.000000",
      "osd_deep_scrub_randomize_ratio": "0.150000",
      "osd_deep_scrub_stride": "524288",
      "osd_deep_scrub_update_digest_min_age": "7200",
      "osd_max_scrubs": "1",
      "osd_op_queue_mclock_scrub_lim": "0.001000",
      "osd_op_queue_mclock_scrub_res": "0.000000",
      "osd_op_queue_mclock_scrub_wgt": "1.000000",
      "osd_requested_scrub_priority": "120",
      "osd_scrub_auto_repair": "false",
      "osd_scrub_auto_repair_num_errors": "5",
      "osd_scrub_backoff_ratio": "0.660000",
      "osd_scrub_begin_hour": "0",
      "osd_scrub_chunk_max": "25",
      "osd_scrub_chunk_min": "5",
      "osd_scrub_cost": "52428800",
      "osd_scrub_during_recovery": "false",
      "osd_scrub_end_hour": "24",
      "osd_scrub_interval_randomize_ratio": "0.500000",
      "osd_scrub_invalid_stats": "true",
      "osd_scrub_load_threshold": "0.500000",
      "osd_scrub_max_interval": "604800.000000",
      "osd_scrub_min_interval": "86400.000000",
      "osd_scrub_priority": "5",
      "osd_scrub_sleep": "0.000000",

  Karun 

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com