> On 13 Apr 2021, at 20:42, Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Tue, Apr 13, 2021 at 11:36:05AM -0700, Manjunath Patil wrote: >> Currently ipoib connected mode queries the device[HCA] to get pkey table >> entry during connection formation. This will increase the time taken to >> form the connection, especially when limited pkeys are in use. This >> gets worse when multiple connection attempts are done in parallel. >> >> Since ipoib interfaces are locked to a single pkey, use the pkey index >> that was determined at link up time instead of searching anything. >> >> This improved the latency from 500ms to 1ms on an internal setup. >> >> Suggested-by: Jason Gunthorpe <jgg@xxxxxxxxxx> >> Signed-off-by: Manjunath Patil <manjunath.b.patil@xxxxxxxxxx> >> --- >> v2: v1 used the cached version[ib_find_cached_pkey()] to get the pkey table >> entry. Following the Jason's comments for v1, I switched to pkey index that was >> determined at link up time in v2. > > Can you confirm that the pkey index does get updated if the SM changes > the pkey table? (and if so how is the locking done for reading the pkey_index?) > > That is about the only reason to have had a dedicated query here Unless I misunderstood you completely, one instance of a netdev has only a single pkey for its lifetime: > On 27 Jan 2021, at 01:16, Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > > Yes, and each new netdev that spawns has a fixed pkey that doesn't > change for the life of the netdev ... and, if you anticipate that the UD QP is using pkey1 at indexX, the pkey table table gets updates by the SM so the new entry in indexX becomes pkey2, the old pkey1 is now at a new position in the table (or not in the table is another case), let's say pkey1 is now found at indexY. Now, the connected mode QP will use pkey1 at indexY if a dedicated query is performed. Then we end up in a split brain, the UD QP uses pkey2 and the RC QPs use pkey1. With Manju's patch, they will at least use the same pkey. Not related to this commit; I find it strange that the return value of update_child_pkey() is not used in __ipoib_ib_dev_flush(). Thxs, Håkon > > Jason
