On Mon, Oct 02, 2017 at 12:47:16PM -0700, Ray Jui wrote: > Hi Bjorn, > > On 10/2/2017 12:35 PM, Bjorn Helgaas wrote: > >On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: > >>Hi Bjorn and all, > >> > >>One of our ARM64 based processor (NS2) has our system address > >>reversed for PCIe outbound transaction in the range of 0x0000_0000 > >>and 0x1fff_ffff. If we use 1:1 mapping between the system address > >>and PCIe address, we end up having the PCIe address in the same > >>range. > >> > >>In this case, we noticed that certain USB-PCI adapters will decline > >>access to BARs mapped in this address range with "Unsupported > >>Request". In this case, if we change the PCIe address range to > >>something else like 0x8000_0000, it would work without an issue. > >> > >>This sounds like an issue with the USB PCI adapter that cannot allow > >>BARs to be used with specific PCIe address range. Is my > >>understanding correct? > > > >I assume you're seeing "Unsupported Request" completions on the PCIe > >link? Can you tell with an analyzer that the TLP actually appears on > >the link and it's the device that is responding with UR? I'm just > >wondering if there's something in the RC that could be > >short-circuiting this. But I guess you mention "certain USB-PCI > >adapters", which implies that these BAR addresses work for *some* > >devices, just not these particular USB-PCI ones. > > Yes, UR is observed with a PCIe analyzer. See attached two JPGs for > captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 > and the other uses 0x8000_0000). > > And yes, this behavior is only seen with certain PCIe-USB adapters. > Most other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe > switches, and etc) work fine with PICe address of 0x0000_0000. > > > > >AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff > >region shouldn't be used for BARs. > > > > Okay, thanks for helping to confirm that. > > >It's surprising to me that a device would special-case addresses like > >that, but maybe it does. > > If this is proven to be a shortcoming of some specific PCIe-USB > adapters (and in fact it seems to be the case so far based on all of > the data points). Does it make any sense to you to have a workaround > done by the root complex, i.e., to avoid using 0x0000_0000 outbound > region; instead, use something else, .e.g, 0x1000_0000 We could have a workaround, as long as it's done in a way that doesn't penalize all the correctly-working hardware. I don't want to give up 1/16 of the 32-bit address space for everybody just to accomodate some broken hardware. I'm not sure what form such a workaround would take. A quirk could mark a zero-valued BAR as invalid and force reallocation, but I'm not sure how we'd force the allocator to avoid this region. There might be some development required there. Bjorn
