Re: Creating new RDMA driver for habanalabs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jul 8, 2022 at 4:29 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote:
>
> On Thu, Jul 07, 2022 at 12:30:03PM +0300, Oded Gabbay wrote:
> > > >    These limitations are not relevant to a deployment where all the NICs are
> > > >    Gaudi NICs, because we can use a single rkey for all MRs.
> > >
> > > Er, that is weird, did you mean to say you have only one MR per PD and
> > > that it always has a fixed value?
>
> > Not exactly. We have multiple MRs per PD, but the driver assigns the
> > same rkey (fixed value) for all created MRs. Our h/w matches the rkey
> > with the one that is written in the QP. The rkey is not part of the actual
> > MMU translation that is done inside our h/w. The MMU translation is
> > done using the PD (we call it ASID - address space ID) and Address.
>
> I don't understand this at all - how can you have multiple MRs if
> there is only one ASID per PD? The MR is logically the ASID since the
> MR is the verbs model for MMU translation.

We don't follow the MR verbs model. This is the meaning of the
hardware constraint I wrote imo.
Our MMU does a pgt walk that starts with ASID and then just goes
according to the virtual address, same as regular CPU does.
The key is not a part of the pgt.
The ASID represents different processes, but because we decided long
ago we support only a single user process,
we only allocate a single ASID, which will translate to a single PD in
our IBverb driver.

>
> So, if you have one ASID per PD and multiple MRs, what are the MRs
> supposed to be?
>
> Jason

Per my understanding, the MRs are meant to notify the driver that the
user would like the h/w MMU to be familiar with these memory regions.
As we also need to pin them, it is preferable to have multiple small
MRs than a single very large MR.
The fact that the key that is returned is the same for all memory
regions shouldn't affect the user. Our MMU will be able to do the
translation correctly using only the ASID+address.
In addition, because we also have on-device memory (HBM), we would
like to allow the user to register memory regions in that memory. So
we need to support at least two MRs.

Thanks,
Oded



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux