On Mon, Dec 5, 2022 at 11:40 AM Eric Pilmore <epilmore@xxxxxxxxxx> wrote:
>
> On Mon, Dec 5, 2022 at 7:30 AM Dave Jiang <dave.jiang@xxxxxxxxx> wrote:
> >
> >
> >
> > On 12/3/2022 1:51 AM, Eric Pilmore wrote:
> > > On Fri, Dec 2, 2022 at 1:57 PM Eric Pilmore <epilmore@xxxxxxxxxx> wrote:
> > >>
> > >> Was curious if anybody has any expected performance numbers for the
> > >> AMD DMA engine "ptdma"?
> > >>
> > >> I'm doing some testing utilizing the "ntb_netdev" module for TCP/IP
> > >> communication between servers via NTB (Non-Transparent Bridge) using
> > >> "iperf". I find that on Intel based boxes, utilizing IOAT DMA, I can
> > >> get approximately 19-20 Gb/s for a simple untuned single iperf
> > >> instance. However, when running on AMD based boxes (Milan CPUs), and
> > >> running the latest ptdma driver from the Linux tree, I can only
> > >> achieve about 2-3 Gb/s. I'm thinking there must be some driver knob
> > >> that I need to tweek or something.
> > >>
> > >> Any help is greatly appreciated.
> > >>
> > >> Thanks,
> > >> Eric
> > >
> > >
> > > You can disregard this question. The issue turned out to be a bug in
> > > the ntb_netdev module. The module was calling dev_kfree_skb() in an
> > > inappropriate place (interrupt context). Once that was fixed (changed
> > > to dev_kfree_skb_irq()), some assert WARNINGS (that I had previously
> > > missed) went away and the performance is as expected.
> > >
> >
> > Curious why this bug only effected the AMD DMA driver.... It should've
> > impacted all DMA drivers through NTB right? Did it make any difference
> > with the ioatdma after the change?
>
> That would have been my expectation also, but we have never seen an
> issue with ioatdma. It is only recently that we've started trying out
> ptdma. This particular area of the ntb_netdev module has utilized
> dev_kfree_skb() forever as far as I know. The suspect calls are in
> ntb_netdev_rx_handler() and ntb_netdev_tx_handler(), each of which
> could be called from interrupt context. These handlers are ultimately
> called indirectly via function pointer (struct
> dma_async_tx_descriptor.callback_result).
>
> I have also seen an issue during driver shutdown where dma_sync_wait()
> is hanging up, when ptdma is utilized. Flow is:
> - device_del()
> - bus_remove_device()
> - device_release_driver()
> - device_release_driver_internal()
> - ntb_transport_bus_remove()
> - ntb_netdev_remove()
> - ntb_transport_free_queue()
> - dma_sync_wait() <<-- HANG
>
> I am still investigating this one further in case it is related to
> something in my local logic as I'm using a custom ntb_transport
> module.
>
> Eric
BTW, I tried the change (dev_kfree_skb -> dev_kfree_skb_irq) in
ntb_netdev, while utilizing ioatdma, and no impact. No warnings/errors
or performance impact before/after the modification.
Just to note, the warning, when utilizing ptdma, was coming from
skb_release_head_state(). I should note that I'm running on an old
Linux base (5.3) and I see this area of logic has changed slightly in
the latest Linux base (6.1).
Linux 5.3:
if (skb->destructor) {
WARN_ON(in_irq()); <<<<<<<
skb->destructor(skb);
}
Linux 6.1:
if (skb->destructor) {
DEBUG_NET_WARN_ON_ONCE(in_hardirq()); <<<<<<<<
skb->destructor(skb);
}
However, I see that both in_irq() and in_hardirq() ultimately resolve
to the same check, hardirq_count().
Eric
--
Eric Pilmore
epilmore@xxxxxxxxxx
http://gigaio.com
Phone: (858) 775 2514
This e-mail message is intended only for the individual(s) to whom
it is addressed and may contain information that is privileged,
confidential, proprietary, or otherwise exempt from disclosure under
applicable law. If you believe you have received this message in
error, please advise the sender by return e-mail and delete it from
your mailbox.
Thank you.
