On 2/23/25 00:08, Suren Baghdasaryan wrote:
> On Fri, Feb 14, 2025 at 8:27 AM Vlastimil Babka <vbabka@xxxxxxx> wrote:
>>
>> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
>> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
>> addition to main and spare sheaves.
>>
>> kfree_rcu() operations will try to put objects on this sheaf. Once full,
>> the sheaf is detached and submitted to call_rcu() with a handler that
>> will try to put in in the barn, or flush to slab pages using bulk free,
>
> s/in in/it in
>
>> when the barn is full. Then a new empty sheaf must be obtained to put
>> more objects there.
>>
>> It's possible that no free sheaves are available to use for a new
>> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
>> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
>> kfree_rcu() machinery.
>>
>> Expected advantages:
>> - batching the kfree_rcu() operations, that could eventually replace the
>> existing batching
>> - sheaves can be reused for allocations via barn instead of being
>> flushed to slabs, which is more efficient
>> - this includes cases where only some cpus are allowed to process rcu
>> callbacks (Android)
>>
>> Possible disadvantage:
>> - objects might be waiting for more than their grace period (it is
>> determined by the last object freed into the sheaf), increasing memory
>> usage - but the existing batching does that too?
>>
>> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
>> implementation favors smaller memory footprint over performance.
>>
>> Signed-off-by: Vlastimil Babka <vbabka@xxxxxxx>
>
> Reviewed-by: Suren Baghdasaryan <surenb@xxxxxxxxxx>
Thanks.
>> @@ -2569,6 +2571,24 @@ static void sheaf_flush(struct kmem_cache *s, struct slab_sheaf *sheaf)
>> sheaf->size = 0;
>> }
>>
>> +static void __rcu_free_sheaf_prepare(struct kmem_cache *s,
>> + struct slab_sheaf *sheaf);
>> +
>> +static void rcu_free_sheaf_nobarn(struct rcu_head *head)
>> +{
>> + struct slab_sheaf *sheaf;
>> + struct kmem_cache *s;
>> +
>> + sheaf = container_of(head, struct slab_sheaf, rcu_head);
>> + s = sheaf->cache;
>
> Ah, that's where you are using sheaf->cache. Maybe you should
> introduce it in this patch?
Yeah. Will also move the addition of rcu_free to struct slub_percpu_sheaves
instead of those TODOs.
