The following commit has been merged into the core/debugobjects branch of tip:
Commit-ID: 13f9ca723900ae3ae8e0a1e76ba86e7786e60645
Gitweb: https://git.kernel.org/tip/13f9ca723900ae3ae8e0a1e76ba86e7786e60645
Author: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
AuthorDate: Mon, 07 Oct 2024 18:50:20 +02:00
Committer: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
CommitterDate: Tue, 15 Oct 2024 17:30:33 +02:00
debugobjects: Refill per CPU pool more agressively
Right now the per CPU pools are only refilled when they become
empty. That's suboptimal especially when there are still non-freed objects
in the to free list.
Check whether an allocation from the per CPU pool emptied a batch and try
to allocate from the free pool if that still has objects available.
kmem_cache_alloc() kmem_cache_free()
Baseline: 295k 245k
Refill: 225k 173k
Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Reviewed-by: Zhen Lei <thunder.leizhen@xxxxxxxxxx>
Link: https://lore.kernel.org/all/20241007164914.439053085@xxxxxxxxxxxxx
---
lib/debugobjects.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index fc9397d..cc32844 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -255,6 +255,24 @@ static struct debug_obj *pcpu_alloc(void)
if (likely(obj)) {
pcp->cnt--;
+ /*
+ * If this emptied a batch try to refill from the
+ * free pool. Don't do that if this was the top-most
+ * batch as pcpu_free() expects the per CPU pool
+ * to be less than ODEBUG_POOL_PERCPU_SIZE.
+ */
+ if (unlikely(pcp->cnt < (ODEBUG_POOL_PERCPU_SIZE - ODEBUG_BATCH_SIZE) &&
+ !(pcp->cnt % ODEBUG_BATCH_SIZE))) {
+ /*
+ * Don't try to allocate from the regular pool here
+ * to not exhaust it prematurely.
+ */
+ if (pool_count(&pool_to_free)) {
+ guard(raw_spinlock)(&pool_lock);
+ pool_move_batch(pcp, &pool_to_free);
+ pcpu_refill_stats();
+ }
+ }
return obj;
}
|