On Mon, 16 Nov 2015, Dan van der Ster wrote: > On Mon, Nov 16, 2015 at 4:58 PM, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > > On Mon, Nov 16, 2015 at 4:32 PM, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > >> On Mon, Nov 16, 2015 at 4:20 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > >>> On Mon, 16 Nov 2015, Dan van der Ster wrote: > >>>> Instead of keeping a 24hr loadavg, how about we allow scrubs whenever > >>>> the loadavg is decreasing (or below the threshold)? As long as the > >>>> 1min loadavg is less than the 15min loadavg, we should be ok to allow > >>>> new scrubs. If you agree I'll add the patch below to my PR. > >>> > >>> I like the simplicity of that, I'm afraid its going to just trigger a > >>> feedback loop and oscillations on the host. I.e., as soo as we see *any* > >>> decrease, all osds on the host will start to scrub, which will push the > >>> load up. Once that round of PGs finish, the load will start to drop > >>> again, triggering another round. This'll happen regardless of whether > >>> we're in the peak hours or not, and the high-level goal (IMO at least) is > >>> to do scrubbing in non-peak hours. > >> > >> We checked our OSDs' 24hr loadavg plots today and found that the > >> original idea of 0.8 * 24hr loadavg wouldn't leave many chances for > >> scrubs to run. So maybe if we used 0.9 or 1.0 it would be doable. > >> > >> BTW, I realized there was a silly error in that earlier patch, and we > >> anyway need an upper bound, say # cpus. So until your response came I > >> was working with this idea: > >> https://stikked.web.cern.ch/stikked/view/raw/5586a912 > > > > Sorry for SSO. Here: > > > > https://gist.github.com/dvanders/f3b08373af0f5957f589 > > Hi again. Here's a first shot at a daily loadavg heuristic: > https://github.com/ceph/ceph/commit/15474124a183c7e92f457f836f7008a2813aa672 > I had to guess where it would be best to store the daily_loadavg > member and where to initialize it... please advise. > > I took the conservative approach of triggering scrubs when either: > 1m loadavg < osd_scrub_load_threshold, or > 1m loadavg < 24hr loadavg && 1m loadavg < 15m loadavg > > The whole PR would become this: > https://github.com/ceph/ceph/compare/master...cernceph:wip-deepscrub-daily Looks reasonable to me! I'm still a bit worried that the 1m < 15m thing will mean that on the completion of every scrub we have to wait ~1m before the next scrub starts. Maybe that's okay, though... I'd say let's try this and adjust that later if it seems problematic (conservative == better). sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
