On Fri, Oct 22, 2021 at 02:18:16PM -0500, Bob Pearson wrote:
> In rxe_pool.c there are two separate pool APIs for creating a new object
> rxe_alloc() and rxe_alloc_locked(). Currently they are identical except
> for GFP_KERNEL vs GFP_ATOMIC. Make rxe_alloc() take the pool lock which
> is in line with the other APIs in the library and was the original intent.
> Keep kzalloc() outside of the lock to avoid using GFP_ATOMIC.
> Make similar change to rxe_add_to_pool. The code checking the pool limit
> is potentially racy without a lock. The lock around finishing the pool
> element provides a memory barrier before 'publishing' the object.
>
> Signed-off-by: Bob Pearson <rpearsonhpe@xxxxxxxxx>
> v3
> Kept rxe_alloc and rxe_alloc_locked separate to allow use of GFP_KERNEL
> in rxe_alloc
>
> drivers/infiniband/sw/rxe/rxe_pool.c | 18 +++++++++++++++---
> 1 file changed, 15 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c
> index 7b4cb46edfd9..8138e459b6c1 100644
> +++ b/drivers/infiniband/sw/rxe/rxe_pool.c
> @@ -356,39 +356,51 @@ void *rxe_alloc(struct rxe_pool *pool)
> {
> struct rxe_type_info *info = &rxe_type_info[pool->type];
> struct rxe_pool_entry *elem;
> + unsigned long flags;
> u8 *obj;
>
> + write_lock_irqsave(&pool->pool_lock, flags);
> if (atomic_inc_return(&pool->num_elem) > pool->max_elem)
> goto out_cnt;
> + write_unlock_irqrestore(&pool->pool_lock, flags);
Atomic's don't need locks this isn't racy as is today
> obj = kzalloc(info->size, GFP_KERNEL);
> - if (!obj)
> - goto out_cnt;
> + if (!obj) {
> + atomic_dec(&pool->num_elem);
> + return NULL;
> + }
>
> + write_lock_irqsave(&pool->pool_lock, flags);
> elem = (struct rxe_pool_entry *)(obj + info->elem_offset);
> -
> elem->pool = pool;
> kref_init(&elem->ref_cnt);
> + write_unlock_irqrestore(&pool->pool_lock, flags);
And none of this needs a lock either
The 'release' operation should be provided by the code that writes a
non-thread-local pointer, not by the code that initializes it a
pointer that never leaves the stack.
Jason
