On 12.12.2022 15:32, Bjorn Helgaas wrote:
On Mon, Dec 12, 2022 at 1:36 PM Alexander Motin <mav@xxxxxxxxxxxxx> wrote:
Hi,
I am writing to you three as the authors of Linux
drivers/pci/setup-bus.c pci_bus_distribute_available_resources()
function. Trying to debug PCI hot-plug issue on passive side of AMD NTB
I hit this function, behavior of which I looks very suspicious to me,
which I believe cause resource allocation problems we observe.
As I see, this function distributes extra size of parent memory window
of hot-plug PCI bridge between memory windows of child bridges. It
probably makes some sense, but I see a problem in the fact that the
function only looks on children bridge memory windows, but not any other
resources (of bridges or other devices that may be there).
In my AMD NTB case PCI topology looks this way:
+-[0000:80]-+-00.0
| +-01.1-[81-83]----00.0-[82-83]----00.0-[83]--+-00.0 Dummy
| | \-00.1 NTB
80:01.1 is the root bridge where the hot-plug happens. The 81:00.0
bridge in addition to memory windows has small 16KB BAR. But since it
is the only bridge on the bus, the function passes all available
resources down to its children. As result, that BAR fails to allocate.
And while that BAR seems not really needed, in some cases the
allocation error makes whole memory window to be disabled, that ends up
in NTB device driver attach failure.
Mika is working on what sounds like the same problem. His current
patch series is at
https://lore.kernel.org/linux-pci/20221130112221.66612-1-mika.westerberg@xxxxxxxxxxxxxxx/
We would appreciate your comments and testing as that series is developed.
Thank you, Bjorn. This definitely looks related, but as you've already
noted in your review there, present patch does not handle BARs of the
bridge itself, that I have in my case. I'd be happy to test the updated
patch. Please keep me in a loop.
I also agree with your comment that the same should be done in case of
multiple bridges. I am generally not sure the cases of single bridge or
not having hot-plug on this level should be any specific.
It may be rare to see PCI bridges with BARs, but I know that at least
some PLX bridges also have BARs for configuration purposes. Also I
suppose the same problem would happen if there are other device on the
bus aside of the bridge.
I've tried to disable pci_bus_distribute_available_resources(), and it
fixed the problem. But I suppose it may cause problems in cases for
which this function was developed (hot-plug of JBOD supporting
hot-plug?). Am I right?
I would appreciate your feedback on this issue. I am new to this code
area of Linux and not sure how to better fix this, but the way the code
is written now looks very wrong to me.
Thanks.
--
Alexander Motin
--
Alexander Motin
