Thank you very much Alex and Ben for all the details. Please find my comments in lined.. On Fri, Jun 15, 2018 at 1:34 AM, Walker, Benjamin <benjamin.walker@xxxxxxxxx> wrote: > On Thu, 2018-06-14 at 08:50 -0600, Alex Williamson wrote: >> On Thu, 14 Jun 2018 16:18:15 +0530 >> Srinath Mannam <srinath.mannam@xxxxxxxxxxxx> wrote: >> >> > Hi Sinan Kaya, >> > >> > Here are the details, >> > >> > The issue is, For CMB cards SQs are allocated inside device BAR memory >> > which is different from normal cards. >> > In Normal cards SQ memory allocated at host side. >> > In both the cases physical address of CQ memory is programmed in NVMe >> > controller register. >> > This method works for normal cards because CQ memory is at host side. >> > But in CMB cards pci bus address equivalent to CQ memory needs to program. >> > >> > More details are in the patch: nvme-pci: Use PCI bus address for >> > data/queues in CMB. >> > >> > With the above patch issue is fixed in the NVMe kernel driver, But >> > similar fix is required in SPDK library also. >> > So, We need a mechanism to get pci_bus_address in user space libraries >> > to address this issue. >> >> I don't understand the CQ vs CMB, but I think I gather that there's some Sorry, It was mistake, I meant SQ not CQ. >> sort of buffer that's allocated from within the devices MMIO BAR and >> some programming of the device needs to reference that buffer. >> Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map >> the BAR into the IOVA address space and you can then use the IOVA + >> offset into the BAR for the device to reference the buffer? It seems >> this is the same way we'd setup a peer-to-peer mapping, but we're using >> it for the device to reference itself effectively. Thanks, > > SPDK already does this today. It's capable of placing submission and completion > queues in the device's controller memory buffer (CMB) and will do all of the > right things for both uio (look up physical address) and vfio (send > IOMMU_MAP_DMA ioctl to set a iova and use that). It additionally already > supports using the CMB as a target for peer-to-peer data transfers. > > This was just confirmed to work on the most recent SPDK release. If you are not > on the most recent release, I recommend trying that first. If there is still > some issue, please do take it to the SPDK mailing list. > We are using SPDK version 18.04. Is fix available in 18.04? Please point me to the patch with which fix was added. > Thanks, > Ben
