On 7/19/23 02:49, Leon Romanovsky wrote: > On Tue, Jul 18, 2023 at 12:59:44PM -0500, Bob Pearson wrote: >> Make rcu_read locking of critical sections with the indexed >> verbs objects be protected from early freeing of those objects. >> The AH, QP, MR and MW objects are looked up from their indices >> contained in received packets or WQEs during I/O processing. >> Make these objects be freed using kfree_rcu to avoid races. > > Sorry, how use of RCU avoid races? > > Thanks The races are between destroy/deallocate/dereg verbs API calls and packets arriving or completing send or deferred processing of wqes. Packets and wqes contain indices/keys/numbers that refer to objects. The rxe driver maintains xarrays for each type of object that allow to lookup the address of the object from its index and then take a reference to protect the pointer. The destroy verbs defer completion until the reference count falls to zero and then removes the entry in the xarray. These operations need to be atomic. One alternative is to use spinlocks to protect them but that places a load on performance under heavy load which is typically dominated by the lookup function since objects tend to have a long lifetime. rcu readlocks are a better alternative but depend on the deferred destruction of the objects used in the rcu critical section. In certain benchmarks the use of rcu locking has a very large performance advantage over spinlocks. For example 100s of QPs running over a 200 gbit/s network. Bob
