On Fri, Feb 21, 2025 at 6:04 PM Zhu Yanjun <yanjun.zhu@xxxxxxxxx> wrote: > > On 20.02.25 18:56, Jacob Moroni wrote: > > In workloads where there are many processes establishing > > connections using RDMA CM in parallel (large scale MPI), > > there can be heavy contention for mad_agent_lock in > > cm_alloc_msg. > > > > This contention can occur while inside of a spin_lock_irq > > region, leading to interrupts being disabled for extended > > durations on many cores. Furthermore, it leads to the > > serialization of rdma_create_ah calls, which has negative > > performance impacts for NICs which are capable of processing > > multiple address handle creations in parallel. > > In the link: > https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html > " > ... > spin_lock() / spin_unlock() > > must not lose CPU while holding a spin lock, other threads will wait for > the lock for a long time > > spin_lock() prevents kernel preemption by ++preempt_count in > uniprocessor, that’s all spin_lock() does > > must NOT call any function that can potentially sleep > ex) kmalloc, copy_from_user > > hardware interrupt is ok unless the interrupt handler may try to lock > this spin lock > spin lock not recursive: same thread locking twice will deadlock > > keep the critical section as small as possible > ... > " > And from the source code, it seems that spin_lock/spin_unlock are not > related with interrupts. > > I wonder why "leading to interrupts being disabled for extended > durations on many cores" with spin_lock/spin_unlock? > > I am not against this commit. I am just obvious why > spin_lock/spin_unlock are related with "interrupts being disabled". Look at drivers/infiniband/core/cm.c spin_lock_irqsave(&cm_id_priv->lock, flags); -> Then call cm_alloc_msg() while hard IRQ are masked.
