++ Catalin, Will (ARM64 Maintainers) From: Jon Hunter <jonathanh@xxxxxxxxxx> Date: Jul/29/2019, 11:55:18 (UTC+00:00) > > On 29/07/2019 09:16, Jose Abreu wrote: > > From: Jose Abreu <joabreu@xxxxxxxxxxxx> > > Date: Jul/27/2019, 16:56:37 (UTC+00:00) > > > >> From: Jon Hunter <jonathanh@xxxxxxxxxx> > >> Date: Jul/26/2019, 15:11:00 (UTC+00:00) > >> > >>> > >>> On 25/07/2019 16:12, Jose Abreu wrote: > >>>> From: Jon Hunter <jonathanh@xxxxxxxxxx> > >>>> Date: Jul/25/2019, 15:25:59 (UTC+00:00) > >>>> > >>>>> > >>>>> On 25/07/2019 14:26, Jose Abreu wrote: > >>>>> > >>>>> ... > >>>>> > >>>>>> Well, I wasn't expecting that :/ > >>>>>> > >>>>>> Per documentation of barriers I think we should set descriptor fields > >>>>>> and then barrier and finally ownership to HW so that remaining fields > >>>>>> are coherent before owner is set. > >>>>>> > >>>>>> Anyway, can you also add a dma_rmb() after the call to > >>>>>> stmmac_rx_status() ? > >>>>> > >>>>> Yes. I removed the debug print added the barrier, but that did not help. > >>>> > >>>> So, I was finally able to setup NFS using your replicated setup and I > >>>> can't see the issue :( > >>>> > >>>> The only difference I have from yours is that I'm using TCP in NFS > >>>> whilst you (I believe from the logs), use UDP. > >>> > >>> So I tried TCP by setting the kernel boot params to 'nfsvers=3' and > >>> 'proto=tcp' and this does appear to be more stable, but not 100% stable. > >>> It still appears to fail in the same place about 50% of the time. > >>> > >>>> You do have flow control active right ? And your HW FIFO size is >= 4k ? > >>> > >>> How can I verify if flow control is active? > >> > >> You can check it by dumping register MTL_RxQ_Operation_Mode (0xd30). > > Where would be the appropriate place to dump this? After probe? Maybe > best if you can share a code snippet of where to dump this. > > >> Can you also add IOMMU debug in file "drivers/iommu/iommu.c" ? > > You can find a boot log here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.ubuntu.com_p_qtRqtYKHGF_&d=DwICaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=WHDsc6kcWAl4i96Vm5hJ_19IJiuxx_p_Rzo2g-uHDKw&m=NrxsR2etpZHGb7HkN4XdgaGmKM1XYyldihNPL6qVSv0&s=CMATEcHVoqZw4sIrNOXc7SFE_kV_5CO5EU21-yJez6c&e= > > > And, please try attached debug patch. > > With this patch it appears to boot fine. So far no issues seen. Thank you for testing. Hi Catalin and Will, Sorry to add you in such a long thread but we are seeing a DMA issue with stmmac driver in an ARM64 platform with IOMMU enabled. The issue seems to be solved when buffers allocation for DMA based transfers are *not* mapped with the DMA_ATTR_SKIP_CPU_SYNC flag *OR* when IOMMU is disabled. Notice that after transfer is done we do use dma_sync_single_for_{cpu,device} and then we reuse *the same* page for another transfer. Can you please comment on whether DMA_ATTR_SKIP_CPU_SYNC can not be used in ARM64 platforms with IOMMU ? --- Thanks, Jose Miguel Abreu
