On 12/5/2022 12:40 PM, Eric Pilmore wrote:
On Mon, Dec 5, 2022 at 7:30 AM Dave Jiang <dave.jiang@xxxxxxxxx> wrote:
On 12/3/2022 1:51 AM, Eric Pilmore wrote:
On Fri, Dec 2, 2022 at 1:57 PM Eric Pilmore <epilmore@xxxxxxxxxx> wrote:
Was curious if anybody has any expected performance numbers for the
AMD DMA engine "ptdma"?
I'm doing some testing utilizing the "ntb_netdev" module for TCP/IP
communication between servers via NTB (Non-Transparent Bridge) using
"iperf". I find that on Intel based boxes, utilizing IOAT DMA, I can
get approximately 19-20 Gb/s for a simple untuned single iperf
instance. However, when running on AMD based boxes (Milan CPUs), and
running the latest ptdma driver from the Linux tree, I can only
achieve about 2-3 Gb/s. I'm thinking there must be some driver knob
that I need to tweek or something.
Any help is greatly appreciated.
Thanks,
Eric
You can disregard this question. The issue turned out to be a bug in
the ntb_netdev module. The module was calling dev_kfree_skb() in an
inappropriate place (interrupt context). Once that was fixed (changed
to dev_kfree_skb_irq()), some assert WARNINGS (that I had previously
missed) went away and the performance is as expected.
Curious why this bug only effected the AMD DMA driver.... It should've
impacted all DMA drivers through NTB right? Did it make any difference
with the ioatdma after the change?
That would have been my expectation also, but we have never seen an
issue with ioatdma. It is only recently that we've started trying out
ptdma. This particular area of the ntb_netdev module has utilized
dev_kfree_skb() forever as far as I know. The suspect calls are in
ntb_netdev_rx_handler() and ntb_netdev_tx_handler(), each of which
could be called from interrupt context. These handlers are ultimately
called indirectly via function pointer (struct
dma_async_tx_descriptor.callback_result).
I have also seen an issue during driver shutdown where dma_sync_wait()
is hanging up, when ptdma is utilized. Flow is:
- device_del()
- bus_remove_device()
- device_release_driver()
- device_release_driver_internal()
- ntb_transport_bus_remove()
- ntb_netdev_remove()
- ntb_transport_free_queue()
- dma_sync_wait() <<-- HANG
I am still investigating this one further in case it is related to
something in my local logic as I'm using a custom ntb_transport
module.
Thanks! Look forward to the fix patch on the linux-ntb mailing list. :)
Eric
