On Tue, 9 May 2023, Adriano Silva wrote: > I got the parameters with this script, although I also checked / sys, doing the math everything is right. > > https://gist.github.com/damoxc/6267899 Thanks. prio_stats gives me what I'm looking for. More below. > Em segunda-feira, 8 de maio de 2023 às 21:42:26 BRT, Eric Wheeler <bcache@xxxxxxxxxxxxxxxxxx> escreveu: > On Thu, 4 May 2023, Coly Li wrote: > > > 2023年5月3日 04:34，Eric Wheeler <bcache@xxxxxxxxxxxxxxxxxx> 写道： > > > > > > On Thu, 20 Apr 2023, Adriano Silva wrote: > > >> I continue to investigate the situation. There is actually a performance > > >> gain when the bcache device is only half filled versus full. There is a > > >> reduction and greater stability in the latency of direct writes and this > > >> improves my scenario. > > > > > > Hi Coly, have you been able to look at this? > > > > > > This sounds like a great optimization and Adriano is in a place to test > > > this now and report his findings. > > > > > > I think you said this should be a simple hack to add early reclaim, so > > > maybe you can throw a quick patch together (even a rough first-pass with > > > hard-coded reclaim values) > > > > > > If we can get back to Adriano quickly then he can test while he has an > > > easy-to-reproduce environment. Indeed, this could benefit all bcache > > > users. > > > > My current to-do list on hand is a little bit long. Yes I’d like and > > plan to do it, but the response time cannot be estimated. > > I understand. Maybe I can put something together if you can provide some > pointers since you are _the_ expert on bcache these days. Here are a few > questions: > > Q's for Coly: It _looks_ like bcache frees buckets while the `ca->free_inc` list is full, but it could go further. Consider this hypothetical: https://elixir.bootlin.com/linux/v6.4-rc1/source/drivers/md/bcache/alloc.c#L179 static void invalidate_buckets_lru(struct cache *ca) { ... + int available = 0; + mutex_lock(&ca->set->bucket_lock); + for_each_bucket(b, ca) { + if (!GC_SECTORS_USED(b)) + unused++; + if (GC_MARK(b) == GC_MARK_RECLAIMABLE) + available++; + if (GC_MARK(b) == GC_MARK_DIRTY) + dirty++; + if (GC_MARK(b) == GC_MARK_METADATA) + meta++; + } + mutex_unlock(&ca->set->bucket_lock); + - while (!fifo_full(&ca->free_inc)) { + while (!fifo_full(&ca->free_inc) || available < TARGET_AVAIL_BUCKETS) { if (!heap_pop(&ca->heap, b, bucket_min_cmp)) { /* * We don't want to be calling invalidate_buckets() * multiple times when it can't do anything */ ca->invalidate_needs_gc = 1; wake_up_gc(ca->set); return; } bch_invalidate_one_bucket(ca, b); <<<< this does the work } (TARGET_AVAIL_BUCKETS is a placeholder, ultimately it would be a sysfs setting, probably a percentage.) Coly, would this work? Can you think of any serious issues with this (besides the fact that for_each_bucket is slow)? -Eric > > - It looks like it could be a simple change to bch_allocator_thread(). > Is this the right place? > https://elixir.bootlin.com/linux/v6.3-rc5/source/drivers/md/bcache/alloc.c#L317 > - On alloc.c:332 > if (!fifo_pop(&ca->free_inc, bucket)) > does it just need to be modified to something like this: > if (!fifo_pop(&ca->free_inc, bucket) || > total_unused_cache_percent() < 20) > if so, where does bcache store the concept of "Total Unused Cache" ? > > - If I'm going about it wrong above, then where is the code path in bcache > that frees a bucket such that it is completely unused (ie, as it was > after `make-bcache -C`?) > > > Q's Adriano: > > Where did you get these cache details from your earlier post? In /sys > somewhere, probably, but I didn't find them: > > Total Cache Size 553.31GiB > Total Cache Used 547.78GiB (99%) > Total Unused Cache 5.53GiB (1%) > Dirty Data 0B (0%) > Evictable Cache 503.52GiB (91%) > > > > > -- > Eric Wheeler > > > > > > > Coly Li > > > > [snipped] >
