-----"Christoph Hellwig" <hch@xxxxxx> wrote: ----- >To: "Krishnamraju Eraparaju" <krishna2@xxxxxxxxxxx> >From: "Christoph Hellwig" <hch@xxxxxx> >Date: 03/17/2020 01:45PM >Cc: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>, sagi@xxxxxxxxxxx, >hch@xxxxxx, linux-nvme@xxxxxxxxxxxxxxxxxxx, >linux-rdma@xxxxxxxxxxxxxxx, "Nirranjan Kirubaharan" ><nirranjan@xxxxxxxxxxx>, "Potnuri Bharat Teja" <bharat@xxxxxxxxxxx> >Subject: [EXTERNAL] Re: broken CRCs at NVMeF target with SIW & >NVMe/TCP transports > >On Mon, Mar 16, 2020 at 09:50:10PM +0530, Krishnamraju Eraparaju >wrote: >> >> I'm seeing broken CRCs at NVMeF target while running the below >program >> at host. Here RDMA transport is SoftiWARP, but I'm also seeing the >> same issue with NVMe/TCP aswell. >> >> It appears to me that the same buffer is being rewritten by the >> application/ULP before getting the completion for the previous >requests. >> getting the completion for the previous requests. HW based >> HW based trasports(like iw_cxgb4) are not showing this issue >because >> they copy/DMA and then compute the CRC on copied buffer. > >For TCP we can set BDI_CAP_STABLE_WRITES. For RDMA I don't think >that Hmm, can you elaborate a little more here? I see that flag being set for data digest enabled (e.g. nvme/host/core.c:nvme_alloc_ns()). But enabling that data digest CRC is exactly when the NVMeF/TCP target detects the issue and drops the frame and disconnects...? The current situation for NVMeF/TCP is that the data digest is not enabled per default and buffer changes are not detected then. Krishna first detected it with using siw against hardware iWarp target, since the CRC gets negotiated then. >is a good idea as pretty much all RDMA block drivers rely on the >DMA behavior above. The answer is to bounce buffer the data in >SoftiWARP / SoftRoCE. > > Another extra copy of user data isn't really charming. Can we somehow let the ULP have its fingers crossed until the buffer got transferred, as signaled back? Best, Bernard.
