Re: Writeback cache all used.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


> 2023年4月4日 03:27,Eric Wheeler <bcache@xxxxxxxxxxxxxxxxxx> 写道:
> On Mon, 3 Apr 2023, Coly Li wrote:
>>> 2023年4月2日 08:01,Eric Wheeler <bcache@xxxxxxxxxxxxxxxxxx> 写道:
>>> On Fri, 31 Mar 2023, Adriano Silva wrote:
>>>> Thank you very much!
>>>>> I don't know for sure, but I'd think that since 91% of the cache is
>>>>> evictable, writing would just evict some data from the cache (without
>>>>> writing to the HDD, since it's not dirty data) and write to that area of
>>>>> the cache, *not* to the HDD. It wouldn't make sense in many cases to
>>>>> actually remove data from the cache, because then any reads of that data
>>>>> would have to read from the HDD; leaving it in the cache has very little
>>>>> cost and would speed up any reads of that data.
>>>> Maybe you're right, it seems to be writing to the cache, despite it 
>>>> indicating that the cache is at 100% full.
>>>> I noticed that it has excellent reading performance, but the writing 
>>>> performance dropped a lot when the cache was full. It's still a higher 
>>>> performance than the HDD, but much lower than it is when it's half full 
>>>> or empty.
>>>> Sequential writing tests with "_fio" now show me 240MB/s of writing, 
>>>> which was already 900MB/s when the cache was still half full. Write 
>>>> latency has also increased. IOPS on random 4K writes are now in the 5K 
>>>> range. It was 16K with half used cache. At random 4K with Ioping, 
>>>> latency went up. With half cache it was 500us. It is now 945us.
>>>> For reading, nothing has changed.
>>>> However, for systems where writing time is critical, it makes a 
>>>> significant difference. If possible I would like to always keep it with 
>>>> a reasonable amount of empty space, to improve writing responses. Reduce 
>>>> 4K latency, mostly. Even if it were for me to program a script in 
>>>> crontab or something like that, so that during the night or something 
>>>> like that the system executes a command for it to clear a percentage of 
>>>> the cache (about 30% for example) that has been unused for the longest 
>>>> time . This would possibly make the cache more efficient on writes as 
>>>> well.
>>> That is an intersting idea since it saves latency. Keeping a few unused 
>>> ready to go would prevent GC during a cached write. 
>> Currently there are around 10% reserved already, if dirty data exceeds 
>> the threshold further writing will go into backing device directly.
> It doesn't sound like he is referring to dirty data.  If I understand 
> correctly, he means that when the cache is 100% allocated (whether or not 
> anything is dirty) that latency is 2x what it could be compared to when 
> there are unallocated buckets ready for writing (ie, after formatting the 
> cache, but before it fully allocates).  His sequential throughput was 4x 
> slower with a 100% allocated cache: 900MB/s at 50% full after a format, 
> but only 240MB/s when the cache buckets are 100% allocated.

It sounds like a large cache size with limit memory cache for B+tree nodes?

If the memory is limited and all B+tree nodes in the hot I/O paths cannot stay in memory, it is possible for such behavior happens. In this case, shrink the cached data may deduce the meta data and consequential in-memory B+tree nodes as well. Yes it may be helpful for such scenario.

But what is the I/O pattern here? If all the cache space occupied by clean data for read request, and write performance is cared about then. Is this a write tended, or read tended workload, or mixed?

It is suspicious that the meta data I/O to read-in missing B+tree nodes contributes the I/O performance degrade. But this is only a guess from me.

>> Reserve more space doesn’t change too much, if there are always busy 
>> write request arriving. For occupied clean cache space, I tested years 
>> ago, the space can be shrunk very fast and it won’t be a performance 
>> bottleneck. If the situation changes now, please inform me.
> His performance specs above indicate that 100% occupided but clean cache 
> increases latency (due to release/re-allocate overhead).  The increased 
> latency reduces effective throughput.

Maybe, increase a bit memory may help a lot. But if this is not a server machine, e.g. laptop, increasing DIMM size might be unfeasible for now days models.

Let’s check whether the memory is insufficient for this case firstly.

>>> Coly, would this be an easy feature to add?
>> To make it, the change won’t be complexed. But I don’t feel it may solve 
>> the original writing performance issue when space is almost full. In the 
>> code we already have similar lists to hold available buckets for future 
>> data/metadata allocation.
> Questions: 
> - Right after a cache is formated and attached to a bcache volume, which 
>  list contains the buckets that have never been used?

All buckets on cache are allocated in sequential-like order, no such dedicate list to track never-used buckets. There are arrays to trace the buckets dirty state and get number, but not for this purpose. Maintain such list is expensive when cache size is large.

> - Where does bcache insert back onto that list?  Does it?

Even for synchronized bucket invalidate by fifo/lru/random, the selection is decided in run time by heap sort, which is still quite slow. But this is on the very slow code path, doesn’t hurt too much. more.

>> But if the lists are empty, there is still time required to do the dirty 
>> writeback and garbage collection if necessary.
> True, that code would always remain, no change there.
>>> Bcache would need a `cache_min_free` tunable that would (asynchronously) 
>>> free the least recently used buckets that are not dirty.
>> For clean cache space, it has been already.
> I'm not sure what you mean by "it has been already" - do you mean this is 
> already implemented?  If so, what/where is the sysfs tunable (or 
> hard-coded min-free-buckets value) ?

For read request, gc thread is still waken up time to time. See the code path,

cached_dev_read_done_bh() ==> cached_dev_read_done() ==> bch_data_insert() ==> bch_data_insert_start() ==> wake_up_gc()

When read missed in cache, writing the clean data from backing device into cache device still occupies cache space, and default every 1/16 space allocated/occupied, the gc thread is waken up asynchronously.

>> This is very fast to shrink clean cache space, I did a test 2 years ago, 
>> it was just not more than 10 seconds to reclaim around 1TB+ clean cache 
>> space. I guess the time might be much less, because reading the 
>> information from priorities file also takes time.
> Reclaiming large chunks of cache is probably fast in one shot, but 
> reclaiming one "clean but allocated" bucket (or even a few buckets) with 
> each WRITE has latency overhead associated with it.  Early reclaim to some 
> reasonable (configrable) minimum free-space value could hide that latency 
> in many workloads.

As I explained, the re-reclaim has been here already. But it cannot help too much if busy I/O requests always coming and writeback and gc threads have no spare time to perform.

If coming I/Os exceeds the service capacity of the cache service window, disappointed requesters can be expected. 

> Long-running bcache volumes are usually 100% allocated, and if freeing 
> batches of clean buckets is fast, then doing it early would save metadata 
> handling during clean bucket re-allocation for new writes (and maybe 
> read-promotion, too).

Let’s check whether it is just becasue of insuffecient memory to hold the hot B+tree node in memory.


Coly Li

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM Kernel]     [Linux Filesystem Development]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux