> On 10 May 2021, at 19:04, Jason Gunthorpe <jgg@xxxxxxxxxx> wrote:
>
> On Wed, May 05, 2021 at 02:54:01PM +0200, Håkon Bugge wrote:
>> There are three conditions that must be fulfilled in order to consider
>> a partition match. Those are:
>>
>> 1. Both P_Keys must valid
>> 2. At least one must be a full member
>> 3. The partitions (lower 15 bits) must match
>>
>> In system employing both limited and full membership ports, we see
>> these false warning messages:
>>
>> RDMA CMA: got different BTH P_Key (0x2a00) and primary path P_Key (0xaa00)
>> RDMA CMA: in the future this may cause the request to be dropped
>>
>> even though the partition is the same.
>>
>> See IBTA 10.9.1.2 Special P_Keys and 10.9.3 Partition Key Matching for
>> a reference.
>>
>> Fixes: 84424a7fc793 ("IB/cma: Print warning on different inner and header P_Keys")
>> Signed-off-by: Håkon Bugge <haakon.bugge@xxxxxxxxxx>
>> ---
>> drivers/infiniband/core/cma.c | 22 ++++++++++++++++++++--
>> 1 file changed, 20 insertions(+), 2 deletions(-)
>
> What is this trying to fix?
The false warning messages. The wrong way though:-)
> IMHO it is a bug on the sender side to send GMPs to use a pkey that
> doesn't exactly match the data path pkey.
The active connector calls ib_addr_get_pkey(). This function extracts the pkey from byte 8/9 in the device's bcast address. However, RFC 4391 explicitly states:
4.1. Broadcast-GID Parameters
The broadcast-GID is set up with the following attributes:
1. P_Key
A "Full Membership" P_Key (high-order bit is set to 1) MUST be
used so that all members may communicate with one another.
In other words, ib_addr_get_pkey() will sometimes wrongly return the full-member version of the partition, when the port is given only the limited member.
Let me do some post-coffee, pre-lunch work tomorrow to come up with a solution, aka ib_find_cached_pkey() followed by an ib_get_cached_pkey()?
Thxs, Håkon
