On Tue, Sep 15, 2020 at 11:18:31AM +0100, Lorenzo Pieralisi wrote: > On Tue, Sep 15, 2020 at 09:25:57AM +1000, Benjamin Herrenschmidt wrote: > > On Mon, 2020-09-14 at 19:57 -0300, Jason Gunthorpe wrote: > > > On Tue, Sep 15, 2020 at 08:00:27AM +1000, Benjamin Herrenschmidt wrote: > > > > On Tue, 2020-09-15 at 07:42 +1000, Benjamin Herrenschmidt wrote: > > > > > > > > > > > which is back to my original question, how do you do DMA using > > > > > > /sys/xx/resources? Why not use VFIO like everything else? > > > > > > > > > > Note: All this doesnt' change the fact that sys/xx/resources_wc > > > > > exists > > > > > for other archs and I see no reasons so far not to have it on ARM... > > > > > > > > Also... it looks like VFIO also doesn't provide a way to do WC yet > > > > unfortunately :-( > > > > > > Yes, but if the driving reason for this patch is because a VFIO user > > > like EFA DPDK is trying to work around VFIO limitations, then I'd say > > > the VFIO mmap should be amended, and not so much worring about sysfs. > > > > I don't think the two are exclusive. > > > > > While there is no reason for ARM to not show the sysfs, it really > > > should never be used. Modern kernels in secure boot don't even show > > > it, for instance. > > > > It's useful for random things, I've used it quite a bit in a previous > > life for things like in-lab hw testing etc... There's tooling out > > there, esp. in the more 'embedded' side of thing that uses this, I > > don't see a good reason not to provide the same level of functionality. > > > > So Lorenzo, imho, we should merge the patch. > > To sum it up: > > (1) RDMA drivers need a new mapping function/attribute to define their > message push model. Actually the message model is not necessarily related > to write combining a la x86, so we should probably come up with a better > and consistent naming. Enabling this patchset may trigger performance > regressions on mellanox drivers on arm64 - this ought to be > addressed. It is pretty clear now that the certain ARM chips that don't do write combining with pgprot_writecombine will performance regress if they are running a certain uncommon Mellanox configuration. I suspect these deployments are all running the out of tree patch for DEVICE_GRE though. So, I wouldn't worry about it - but would prefer to see ARM folks advance some proposal to accommodate those chips that need DEVICE_GRE. Jason
