> -----Original Message----- > From: Cheng Xu <chengyou@xxxxxxxxxxxxxxxxx> > Sent: Friday, 20 May 2022 09:04 > To: Bernard Metzler <BMT@xxxxxxxxxxxxxx>; Jason Gunthorpe > <jgg@xxxxxxxxxx>; Tom Talpey <tom@xxxxxxxxxx> > Cc: dledford@xxxxxxxxxx; leon@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; > KaiShen@xxxxxxxxxxxxxxxxx; tonylu@xxxxxxxxxxxxxxxxx > Subject: [EXTERNAL] Re: [PATCH for-next v7 10/12] RDMA/erdma: Add the > erdma module > > > > On 5/20/22 12:20 AM, Bernard Metzler wrote: > > > > > > <...> > > >>> As far as I know, iWarp device only has one GID entry which > generated > >>> from MAC address. > >>> > >>> For iWarp, The CM part in core code resolves address, finds > >>> route with the help of kernel's net subsystem, and then obtains the > >> correct > >>> ibdev by GID matching. The GID matching in iWarp is indeed MAC > address > >>> matching. > >>> > >>> In another words, for iWarp devices, the core code doesn't handle IP > >>> addressing related stuff directly, it is finished by calling net > APIs. > >>> The netdev set by ib_device_set_netdev does not used in iWarp's CM > >>> process. > >>> > >>> The binded netdev in iWarp devices, mainly have two purposes: > >>> 1). generated GID0, using the netdev's mac address. > >>> 2). get the port state and attributes. > >>> > >>> For 1), erdma device binded to net device also by mac address, which > can > >>> be obtained from our PCIe bar registers. > >>> For 2), erdma can also get the information, and may be more > accurately. > >>> For example, erdma can have different MTU with virtio-net in our > cloud. > >>> > >>> For RoCEv2, I know that it has many GIDs, some of them are generated > >>> from IP addresses, and handing IP addressing in core code. > >> > >> Bernard, Tom what do you think? > >> > >> Jason > > > > I think iWarp (and now RoCEv2 with its UDP dependency) drivers > > produce GIDs mostly to satisfy the current RDMA CM infrastructure, > > which depends on this type of unique identifier, inherited from IB. > > Imo, more natural would be to implement IP based RDMA protocols > > connection management by relying on IP addresses. > > > > Sorry for asking again - why erdma does not need to link with netdev? > > Can erdma exist without using a netdev? > > Actually erdma also need a net device binded to, and so does it. > > These days I’m trying to find out acceptable ways to get the reference > of the binded netdev, e,g, the 'struct net_device' pointer. Unlike other > RDMA drivers can get the reference of their binded netdevs' reference > easily (most RDMA devices are based on the extended aux devices), it is > a little more complex for erdma, because erdma and its binded net device > are two separated PCIe devices. > > Then I find that the netdev reference hold in ibdev is rarely used > in core code for iWarp deivces, GID0 is the key attribute (As you and > Tom mentioned, it appears with the historical need for compatibility, > but I think this is another story). > Yes, I think this is right. If you are saying you can go away with a NULL netdev at CM core, then I think that's fine? Of course the erdma driver must somehow keep track of the state of its associated network device - like catching up with link status - and must provide related information/events to the RDMA core. > So, there are two choices for erdma: enum net devices and find the > matched one, or never calling ib_device_set_netdev. The second one has > less code. > > The second way can't work in ROCE. But it works for iWarp (I've tested), > since the netdev reference is rarely used for iWarp in core code, as I > said in last reply. > > In short, the question discussed here is that: is it acceptable that > doesn't hold the netdev reference in core code for a iWarp driver > (indeed it has a netdev binded to) ? Or is it necessary that calling > ib_device_set_netdev to set the binded netdev for iWarp driver? > > You and Tom both are specialists in iWarp, your opinions are important. > > Thanks very much > Cheng Xu > > > > > > Thanks, > > Bernard.
