On 10/7/21 2:05 PM, Jason Gunthorpe wrote: > On Thu, Oct 07, 2021 at 01:53:27PM -0500, Bob Pearson wrote: > >> On looking, Rao's patch is not in for-next. Last one was >> January. Which branch are you looking at? > > Oh, it is still in the wip branch, try now > > Jason > I see the issue. Rao is asking for 2^20 objects max by default which will require 128KiB of memory in the index reservation bit mask for each of them. There are 4 indexed objects QP by qpn, SRQ by srqn, MR by rkey and MW by rkey. That's 512KiB of memory which seems excessive to me for many use cases where the number of objects is fairly small. The bit mask is used to allocate and free the indices and there is also a red black tree that is used to look up objects by their index (or key if they use keys instead.) If there is a usual way to address these kinds of issues in Linux maybe we should consider that. If not there are a couple of approaches we could take. First would be to get rid of the index bit mask and just hand out randomly selected indices in (a bigger range) and detect collisions when we insert the object into the red black tree and retry. This is basically what happens with 'keys' for example mgids for multicast group elements. Alternatively we could leave the max big but limit the allocated indices to a smaller amount until the total number of allocated indices reached some threshold and then extend the bit mask table. Then only the use cases that really needed the big index range would pay the price for the memory. Random indices would slightly reduce some of the security issues that have been pointed out about the InfiniBand transport. I am looking for suggestions on how to go forward here. Bob
