On 3/19/2021 11:48 AM, Jason Gunthorpe wrote:
On Fri, Mar 19, 2021 at 02:49:29PM +0000, Wan, Kaike wrote:
From: Jason Gunthorpe <jgg@xxxxxxxxxx>
Sent: Friday, March 19, 2021 9:53 AM
To: Wan, Kaike <kaike.wan@xxxxxxxxx>
Cc: dledford@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Rimmer, Todd
<todd.rimmer@xxxxxxxxx>
Subject: Re: [PATCH RFC 0/9] A rendezvous module
On Fri, Mar 19, 2021 at 08:56:26AM -0400, kaike.wan@xxxxxxxxx wrote:
- Basic mode of operations (PSM3 is used as an example for user
applications):
- A middleware (like MPI) has out-of-band communication channels
between any two nodes, which are used to establish high performance
communications for providers such as PSM3.
Huh? Doesn't PSM3 already use it's own special non-verbs char devices that
already have memory caches and other stuff? Now you want to throw that
all away and do yet another char dev just for HFI? Why?
[Wan, Kaike] I think that you are referring to PSM2, which uses the
OPA hfi1 driver that is specific to the OPA hardware. PSM3 uses
standard verbs drivers and supports standard RoCE.
Uhhh.. "PSM" has always been about the ipath special char device, and
if I recall properly the library was semi-discontinued and merged into
libfabric.
This driver is intended to work with a fork of the PSM2 library. The
PSM2 library which is for Omni-Path is now maintained by Cornelis
Networks on our GitHub. PSM3 is something from Intel for Ethernet. I
know it's a bit confusing.
So here you are talking about a libfabric verbs provider that doesn't
use the ipath style char interface but uses verbs and this rv thing so
we call it a libfabric PSM3 provider because thats not confusing to
anyone at all..
A focus is the Intel RDMA Ethernet NICs. As such it cannot use the
hfi1 driver through the special PSM2 interface.
These are the drivers that aren't merged yet, I see. So why are you
sending this now? I'm not interested to look at even more Intel code
when their driver saga is still ongoing for years.
Rather it works with the hfi1 driver through standard verbs
interface.
But nobody would do that right? You'd get better results using the
hif1 native interfaces instead of their slow fake verbs stuff.
I can't imagine why. I'm not sure what you mean by our slow fake verbs
stuff? We support verbs just fine. It's certainly not fake.
I also don't know why you picked the name rv, this looks like it has little to do
with the usual MPI rendezvous protocol. This is all about bulk transfers. It is
actually a lot like RDS. Maybe you should be using RDS?
.
.
The name "rv" is chosen simply because this module is designed to
enable the rendezvous protocol of the MPI/OFI/PSM3 application stack
for large messages. Short messages are handled by eager transfer
through UDP in PSM3.
A bad name seems like it will further limit potential re-use of this
code.
As to the name, I started this driver when I was still at Intel before
handing the reins to Kaike when we spun out. I plucked the name rv out
of the air without much consideration. While the code seems to have
changed a lot, the name seems to have stuck. It was as Kaike mentions
because it helps enable the rendezvous side of PSM3.
-Denny
