> On 5 Jul 2021, at 18:26, Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > > On Tue, Jun 29, 2021 at 01:45:35PM +0000, Haakon Bugge wrote: > >>>>> IMHO it is a bug on the sender side to send GMPs to use a pkey that >>>>> doesn't exactly match the data path pkey. >>>> >>>> The active connector calls ib_addr_get_pkey(). This function >>>> extracts the pkey from byte 8/9 in the device's bcast >>>> address. However, RFC 4391 explicitly states: >>> >>> pkeys in CM come only from path records that the SM returns, the above >>> should only be used to feed into a path record query which could then >>> return back a limited pkey. >>> >>> Everything thereafter should use the SM's version of the pkey. >> >> Revisiting this. I think I mis-interpreted the scenario that led to >> the P_Key mismatch messages. >> >> The CM retrieves the pkey_index that matched the P_Key in the BTH >> (cm_get_bth_pkey()) and thereafter calls ib_get_cached_pkey() to get >> the P_Key value of the particular pkey_index. >> >> Assume a full-member sends a REQ. In that case, both P_Keys (BTH and >> primary path_rec) are full. Further, assume the recipient is only a >> limited member. Since full and limited members of the same partition >> are eligible to communicate, the P_Key retrieved by >> cm_get_bth_pkey() will be the limited one. > > It is incorrect for the issuer of the REQ to put a full pkey in the > REQ message when the target is a limited member. Sorry, I mis-interpreted the spec. I though the PKey in the Path record should be that of the initiator, not the target's. OK. Will come up with a fix. Thxs, Håkon > > The CM model in IB has the target fully under the control of the > initiator, and it is up to the initiator to ask the SM how the target > should generate its return traffic. The SM is reponsible to say that > limited->full communication is done using the limited pkey. > > The initiator is reponsible to place that limited pkey in the REQ > message. > > Somewhere in your system this isn't happening properly, and it is a > bug that the CM is correctly identifying. > > Jason
