On 10/19/21 1:43 PM, Jason Gunthorpe wrote: > On Tue, Oct 19, 2021 at 11:35:30AM -0500, Bob Pearson wrote: > >> Take a look at the note I copied you on more recently. There is some >> progress but not complete elimination of rxe_pool. There is another >> project suggested by Jason which is replacing red black trees by >> xarrays as an alternative approach to indexing rdma objects. This >> would still duplicate the indexing done by rdma-core. A while back I >> looked at trying to reuse the rdma-core indexing but no effort was >> made to make that easy. > > I have no expecation that a driver can re-use the various rdma-core > indexes.. that is not what they are for, and they have a different > lifetime semantic from wha the driver needs. > >> of the APIs are private to rdma-core. These indices are managed by >> the rxe driver for use as lkeys/rkeys, qpns, srqns, and more >> recently address handles. xarrays seem to be more efficient when the >> indices are fairly compact. There is a suggestion that IB and RoCE >> should attempt to make indices that are visible on the network more >> sparse. Nothing will make them secure but they could be a lot more >> secure than they are currently. I believe mlx5 is now using random >> keys for this reason. > > Only qpn really benifits from something like this, and it is more > about maximum lifetime before qpn re-use which is a cyclic allocating > xarray. > > Jason > Thanks. I had given up long ago on using the rdma-core indices. Leon would like to get rid of rxe pools. Actually (I just checked) there is only one remaining object type that is not already allocated in rdma-core and that is MR. (The multicast groups are a special case and not really in the same league as PD, QP, CQ, etc. They probably should just be replaced with open coded kzalloc/kfree. They are not shared or visible to rdma-core.) So with this exception the pools are really just a way to add indices to objects which can be done cleanly with xarrays (I have a version that already does this and it works fine. No performance difference with red-black trees though. Still looking to get rid of the spinlocks and use the rcu locking in xarrays.) Bob
