On Tue, Jan 30, 2024 at 11:59:38AM +0530, Manivannan Sadhasivam wrote: > On Wed, Jan 24, 2024 at 12:10:01PM -0500, Frank Li wrote: > > On Tue, Jan 09, 2024 at 01:52:15PM +0000, Niklas Cassel wrote: > > > Hello Frank, > > > > > > On Tue, Dec 19, 2023 at 03:38:33PM +0100, Niklas Cassel wrote: > > > > On Tue, Dec 19, 2023 at 09:20:21AM -0500, Frank Li wrote: > > > > > On Tue, Dec 19, 2023 at 10:07:14AM +0000, Niklas Cassel wrote: > > > > > > On Mon, Dec 18, 2023 at 11:48:43PM -0500, Frank Li wrote: > > > > > > > dw_pcie_ep_inbound_atu() > > > > > > > { > > > > > > > ... > > > > > > > if (!ep->bar_to_atu[bar]) > > > > > > > free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows); > > > > > > > else > > > > > > > free_win = ep->bar_to_atu[bar]; > > > > > > > ... > > > > > > > } > > > > > > > > > > > > > > The atu index 0 is valid case for atu number. The find_first_zero_bit() > > > > > > > will return 6 when second time call into this function if atu is 0. Suppose > > > > > > > it should use branch 'free_win = ep->bar_to_atu[bar]'. > > > > > > > > > > > > > > Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have > > > > > > > not allocate atu to the bar. > > > > > > > > > > > > > > Reported-by: Niklas Cassel <Niklas.Cassel@xxxxxxx> > > > > > > > Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u > > > > > > > Fixes: 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address") > > > > > > > Signed-off-by: Frank Li <Frank.Li@xxxxxxx> > > > > > > > --- > > > > > > > > > > > > > > Notes: > > > > > > > @Niklas: > > > > > > > I have not test your case. I should be equal to previous's fix in > > > > > > > mail list. > > > > > > > > > > > > Hello Frank, > > > > > > > > > > > > Thank you for sending a proper fix for this! > > > > > > > > > > > > Personally, I slightly prefer your fix that saves the iatu index + 1, and > > > > > > keeps 0 to mean unused. That way, you don't need the memset, and you don't > > > > > > need to change the type to signed, but either way is fine by me, so: > > > > > > > > > > index + 1 don't match hardware iATU index. It will be confused because > > > > > other parts is 0 based. > > > > > > > > > > So I choose "-1" as free iATU. > > > > > > > > A s8 can hold a max value of 127. > > > > CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255. > > > > > > > > Since the DWC code can be synthesized with 256 iATUs, > > > > your code will not work on systems with 128 or more iATUs. > > > > > > > > If we continue to use a u8, and offset the saved value by one, > > > > we will at least be able to support 255-1 == 254 iATUs. > > > > > > Do you plan to send out a v2? > > > > @mani: > > Do you have any comments about this fixes except u8's problem? > > IMO, the core issue lies in the EPF driver. It calls set_bar() during init, but > it is not clearing the BARs with clear_bar() during LINK_DOWN. > > If the BARs were cleared properly, then we would not see this issue. > > I'm planning to do a cleanup of the behavior of EPF with core_init_notifier and > it should get addressed there. Hello Mani, Frank, Please reconsider respinning this. I think that the invalid usage of find_first_zero_bit() is very bad, because other people might copy the find_first_zero_bit() usage here. find_first_zero_bit() can (and will) return 0, so this check is just bad: if (!ep->bar_to_atu[bar]) free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows); ... ep->bar_to_atu[bar] = free_win; set_bit(free_win, ep->ib_window_map); No matter which code Mani adds that cleans up the BARs, will not change that the logic is this function is just broken. Kind regards, Niklas