On 7/28/2020 16:45, Zhu Yanjun wrote: > On 7/29/2020 1:42 AM, Kamal Heib wrote: >> On Tue, Jul 28, 2020 at 11:46:36PM +0800, Zhu Yanjun wrote: >>> On 7/28/2020 9:44 PM, Kamal Heib wrote: >>>> On Tue, Jul 28, 2020 at 09:21:06PM +0800, Zhu Yanjun wrote: >>>>> On 7/28/2020 4:35 PM, Kamal Heib wrote: >>>>>> On Thu, Jul 23, 2020 at 11:15:00PM +0800, Zhu Yanjun wrote: >>>>>>> On 7/23/2020 9:15 PM, Jason Gunthorpe wrote: >>>>>>>> On Thu, Jul 23, 2020 at 09:08:39PM +0800, Zhu Yanjun wrote: >>>>>>>>> On 7/23/2020 3:25 PM, Kamal Heib wrote: >>>>>>>>>> On Thu, Jul 23, 2020 at 02:58:41PM +0800, Zhu Yanjun wrote: >>>>>>>>>>> On 7/23/2020 1:57 PM, Kamal Heib wrote: >>>>>>>>>>>> On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote: >>>>>>>>>>>>> On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu <yanjunz@xxxxxxxxxxxx> wrote: >>>>>>>>>>>>>> From: Kamal Heib <kamalheib1@xxxxxxxxx> >>>>>>>>>>>>>> Sent: Tuesday, July 21, 2020 6:16 PM >>>>>>>>>>>>>> To: linux-rdma@xxxxxxxxxxxxxxx >>>>>>>>>>>>>> Cc: Yanjun Zhu <yanjunz@xxxxxxxxxxxx>; Doug Ledford <dledford@xxxxxxxxxx>; Jason Gunthorpe <jgg@xxxxxxxx>; Kamal Heib <kamalheib1@xxxxxxxxx> >>>>>>>>>>>>>> Subject: [PATCH for-next] RDMA/rxe: Remove pkey table >>>>>>>>>>>>>> >>>>>>>>>>>>>> The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed. >>>>>>>>>>>>>> >>>>>>>>>>>>> Hi Kamal >>>>>>>>>>>>> >>>>>>>>>>>>> After this patch is applied, do you make tests with SoftRoCE and mlx hardware? >>>>>>>>>>>>> >>>>>>>>>>>>> The SoftRoCE should work well with the mlx hardware. >>>>>>>>>>>>> >>>>>>>>>>>>> Zhu Yanjun >>>>>>>>>>>>> >>>>>>>>>>>> Hi Zhu, >>>>>>>>>>>> >>>>>>>>>>>> Yes, please see below: >>>>>>>>>>>> >>>>>>>>>>>> $ ibv_rc_pingpong -d mlx5_0 -g 11 >>>>>>>>>>>> local address: LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121 >>>>>>>>>>> Can you make tests with GSI QP? >>>>>>>>>>> >>>>>>>>>>> Zhu Yanjun >>>>>>>>>>> >>>>>>>>> Is this the GSI ? >>>>>>>>> >>>>>>>>> Please check GSI in "InfiniBandTM Architecture Specification Volume 1 >>>>>>>>> Release 1.3" >>>>>>>>> >>>>>>>>> Then make tests with GSI again. >>>>>>> The followings are also removed by this commit. Not sure if it is good. >>>>>>> >>>>>>> " >>>>>>> >>>>>>> C9-42: If the destination QP is QP1, the BTH:P_Key shall be compared to the >>>>>>> set of P_Keys associated with the port on which the packet arrived. If the >>>>>>> P_Key matches any of the keys associated with the port, it shall be >>>>>>> considered valid. >>>>>>> >>>>>>> " >>>>>>> >>>>>> The above is correct for ports that configured to work in InfiniBand >>>>>> mode, while in RoCEv2 mode only the default P_Key should be associated >>>>>> with the port (Please see below from "ANNEX A17: ROCEV2 (IP ROUTABLE >>>>>> ROCE)): >>>>>> >>>>>> """ >>>>>> 17.7.1 LOADING THE P_KEY TABLE >>>>>> >>>>>> Compliance statement C17-7: on page 1193 describes requirements for >>>>>> setting the P_Key table based on an assumption that the P_Key table is >>>>>> set directly by a Subnet Manager. However, RoCEv2 ports do not support >>>>>> InfiniBand Subnet Management. Therefore, compliance statement C17-7: >>>>>> on page 1193 does not apply to RoCEv2 ports. >>>>> " >>>>> >>>>> C17-7: An HCA shall require no OS involvement to set the P_Key table; >>>>> >>>>> the P_Key table shall be set directly by Subnet Manager MADs. >>>>> >>>>> " >>>>> >>>>> In SoftRoCE, what set the P_Key table? >>>>> >>>> No one is setting the P_Key table in SoftRoCE, and no subnet manager in >>>> the RoCE fabric. >>>> >>>> Could you please tell me what is wrong with this patch? >>> Please read the mail thread again. >>> >>> GSI QP number is 1. In your commits, the handle of qpn == 1 is removed. >>> >>> It seems that it conflicts with IB specification. >>> >>> Not sure if it is good. >>> >> Could you please read my patch again and point to what do you think is >> wrong? > > What I said is very clear. Good luck qpn == 1, qpn == x it, qpn == i, it doesn't matter. Please read the RoCEv2 annex: A17.4.7 INFINIBAND PARTITIONING Methods to populate the P_Key table associated with a RoCEv2 port are outside the scope of this annex. Note that this annex relies on the partition table being initialized at power on time with at least the default P_Key as described in Chapter 10 (Software Transport Interface) of the base specification. The P_Key contained in the BTH is validated for inbound packets as required by the packet header validation protocols defined in Chapter 9 of the base specification. A17.7.1 LOADING THE P_KEY TABLE Compliance statement C17-7 describes requirements for setting the P_Key table based on an assumption that the P_Key table is set directly by a Subnet Manager. However, RoCEv2 ports do not support InfiniBand Subnet Management. Therefore, compliance statement C17-7 does not apply to RoCEv2 ports. Methods for setting the P_Key table associated with a RoCEv2 port are not defined in this specification, except for the requirements for a default P_Key described elsewhere in this annex. rxe = Software RDMA over Ethernet, so we are dealing only with RoCE traffic (no IB). We don't have an SM. The spec requires that at least the default pkey is defined (and rxe defines only the default p_pkey). Kamel is doing just that. Mark > > Zhu Yanjun > >> >> What I did in this patch is to verify that the pkey value in the >> received packet is the default P_Key regardless of the qpn, because RoCE >> devices should maintain only the default P_Key. >> >> Thanks, >> Kamal >> >>>> Thanks, >>>> Kamal >>>> >>>>>> Methods for setting the P_Key table associated with a RoCEv2 port are >>>>>> not defined in this specification, except for the requirements for a >>>>>> default P_Key described elsewhere in this annex. >>>>>> """ >>>>>> >>>>>> Thanks, >>>>>> Kamal >>>>>> >>>>>> >>>>>>>> rping uses RDMA CM which goes over the GSI >>>>>>>> >>>>>>>> Jason >>> >
