On Mon, Dec 5, 2022 at 7:30 AM Dave Jiang <dave.jiang@xxxxxxxxx> wrote:
>
>
>
> On 12/3/2022 1:51 AM, Eric Pilmore wrote:
> > On Fri, Dec 2, 2022 at 1:57 PM Eric Pilmore <epilmore@xxxxxxxxxx> wrote:
> >>
> >> Was curious if anybody has any expected performance numbers for the
> >> AMD DMA engine "ptdma"?
> >>
> >> I'm doing some testing utilizing the "ntb_netdev" module for TCP/IP
> >> communication between servers via NTB (Non-Transparent Bridge) using
> >> "iperf". I find that on Intel based boxes, utilizing IOAT DMA, I can
> >> get approximately 19-20 Gb/s for a simple untuned single iperf
> >> instance. However, when running on AMD based boxes (Milan CPUs), and
> >> running the latest ptdma driver from the Linux tree, I can only
> >> achieve about 2-3 Gb/s. I'm thinking there must be some driver knob
> >> that I need to tweek or something.
> >>
> >> Any help is greatly appreciated.
> >>
> >> Thanks,
> >> Eric
> >
> >
> > You can disregard this question. The issue turned out to be a bug in
> > the ntb_netdev module. The module was calling dev_kfree_skb() in an
> > inappropriate place (interrupt context). Once that was fixed (changed
> > to dev_kfree_skb_irq()), some assert WARNINGS (that I had previously
> > missed) went away and the performance is as expected.
> >
>
> Curious why this bug only effected the AMD DMA driver.... It should've
> impacted all DMA drivers through NTB right? Did it make any difference
> with the ioatdma after the change?
That would have been my expectation also, but we have never seen an
issue with ioatdma. It is only recently that we've started trying out
ptdma. This particular area of the ntb_netdev module has utilized
dev_kfree_skb() forever as far as I know. The suspect calls are in
ntb_netdev_rx_handler() and ntb_netdev_tx_handler(), each of which
could be called from interrupt context. These handlers are ultimately
called indirectly via function pointer (struct
dma_async_tx_descriptor.callback_result).
I have also seen an issue during driver shutdown where dma_sync_wait()
is hanging up, when ptdma is utilized. Flow is:
- device_del()
- bus_remove_device()
- device_release_driver()
- device_release_driver_internal()
- ntb_transport_bus_remove()
- ntb_netdev_remove()
- ntb_transport_free_queue()
- dma_sync_wait() <<-- HANG
I am still investigating this one further in case it is related to
something in my local logic as I'm using a custom ntb_transport
module.
Eric
