> -----Original Message----- > From: Roland Dreier [mailto:roland@xxxxxxxxxxxxxxx] > Sent: Sunday, November 11, 2018 4:23 AM > To: Syed Syed <SYEDS@xxxxxxxxxx> > Cc: linux-rdma@xxxxxxxxxxxxxxx > Subject: Re: Posting mad frame on ib_recv_wr > > > I checked archives and sources as well, but couldn't get a convincing > answer. We have a RDMA NIC, and our nic presents whole rocev2(ipv4) mad > frame on > > the hw receive queue. I am trying to copy(instead of dma, due to some > limitation on hw side) this to ib_recv_wr sglist addr, but I am not sure how > this should be done exactly. > > I tried copying from (frame_start + ETH_HLEN) to sglist[0].addr, but this > resulted in "MAD received with unsupported base version" error in > core/mad.c. > > The only scheme I figured by looking into source is to copy iphr from the > frame at (sglist[0].addr + 20) and then skip udph, bth, deth, and > > copy rest of the frame(mad) at sglist[0].addr + 20 + sizeof(iph). But I am not > sure if this is correct. > > > > Also in the poll what should be the flags on wc? Should I set both > IB_WC_GRH and IB_WC_WITH_NETWORK_HDR_TYPE? And set > network_hdr_type to IPv4? > > For a MAD or any incoming UD message on RoCEv2, the first 40 bytes of > the receive buffer should contain the IP header. See the IB spec > annex defining RoCEv2: > > The first 40 bytes of user posted UD Receive Buffers are reserved for > the L3 header of the incoming packet (as per the InfiniBand Spec > Section 11.4.1.2). In RoCEv2, this area is filled up with the IP > header. IPv6 header uses the entire 40 bytes. IPv4 headers use the 20 > bytes in the second half of the reserved 40 bytes area (i.e. offset 20 > from the beginning of the receive buffer). In this case, the content > of the first 20 bytes is undefined. I had read the spec, but it only mentions message payload, but wasn’t sure if it meant, UD payload or ip payload. There is also mention of other "ipv6 headers may follow routing header before message payload". I am not sure in this case should the message be copied not always at offset 40? > > So yes, for a RoCEv2 packet sent in UDPv4, you should copy the iphdr > from the frame to an offset of 20 bytes from the beginning of the user > supplied buffer, and then copy the payload of the UD message to the > user supplied buffer at offset 40. Note that you need to be careful > about assuming you start with sglist[0] - although the Linux stack > currently doesn't do it, it would be completely valid to post a bunch > of very small scatterlist entries that you would have to skip to reach > offset 20 and offset 40. And the iphdr and/or the payload might span > multiple sglist entries. You should set IB_WC_GRH even for an IPv4 > header. > > I'm assuming you set the network_hdr_type field in the ib_wc structure > to IPv4 or IPv6 as appropriate. You should therefore set > IB_WC_WITH_NETWORK_HDR_TYPE as well. Yes. > > - R.
