> I checked archives and sources as well, but couldn't get a convincing answer. We have a RDMA NIC, and our nic presents whole rocev2(ipv4) mad frame on > the hw receive queue. I am trying to copy(instead of dma, due to some limitation on hw side) this to ib_recv_wr sglist addr, but I am not sure how this should be done exactly. > I tried copying from (frame_start + ETH_HLEN) to sglist[0].addr, but this resulted in "MAD received with unsupported base version" error in core/mad.c. > The only scheme I figured by looking into source is to copy iphr from the frame at (sglist[0].addr + 20) and then skip udph, bth, deth, and > copy rest of the frame(mad) at sglist[0].addr + 20 + sizeof(iph). But I am not sure if this is correct. > > Also in the poll what should be the flags on wc? Should I set both IB_WC_GRH and IB_WC_WITH_NETWORK_HDR_TYPE? And set network_hdr_type to IPv4? For a MAD or any incoming UD message on RoCEv2, the first 40 bytes of the receive buffer should contain the IP header. See the IB spec annex defining RoCEv2: The first 40 bytes of user posted UD Receive Buffers are reserved for the L3 header of the incoming packet (as per the InfiniBand Spec Section 11.4.1.2). In RoCEv2, this area is filled up with the IP header. IPv6 header uses the entire 40 bytes. IPv4 headers use the 20 bytes in the second half of the reserved 40 bytes area (i.e. offset 20 from the beginning of the receive buffer). In this case, the content of the first 20 bytes is undefined. So yes, for a RoCEv2 packet sent in UDPv4, you should copy the iphdr from the frame to an offset of 20 bytes from the beginning of the user supplied buffer, and then copy the payload of the UD message to the user supplied buffer at offset 40. Note that you need to be careful about assuming you start with sglist[0] - although the Linux stack currently doesn't do it, it would be completely valid to post a bunch of very small scatterlist entries that you would have to skip to reach offset 20 and offset 40. And the iphdr and/or the payload might span multiple sglist entries. You should set IB_WC_GRH even for an IPv4 header. I'm assuming you set the network_hdr_type field in the ib_wc structure to IPv4 or IPv6 as appropriate. You should therefore set IB_WC_WITH_NETWORK_HDR_TYPE as well. - R.
