On Thursday 05 November 2009 10:14:56 am Ira W. Snyder wrote:
> On Wed, Nov 04, 2009 at 04:25:41PM -0800, Ira W. Snyder wrote:
> > On Wed, Nov 04, 2009 at 06:18:19PM -0600, Bjorn Helgaas wrote:
> > > You mentioned using "memmap=" to change the aperture we allocate
> > > from. Can you tweak that to make the after-hotplug mappings be
> > > closer to the original BIOS ones? I know you tried that already;
> > > I guess I'm just wondering if the magic lower boundary is something
> > > like 0xfb000000 rather than 0xf0000000.
>
> Ok, I just ran these tests. Using memmap= to reserve memory and force
> Linux to place allocations higher didn't work at all.
>
> I started at 0xf8000000 and moved up from there. Even after getting so
> the BARs were re-assigned to higher addresses than the original BIOS
> addresses didn't help. Reading the BAR still produces all 0xff's.
>
> Here are some logs from my last run, which show the BIOS and Linux
> assigned addresses:
>
> [ 0.372360] pci 0000:01:0b.0: reg 10: [mem 0xfeb00000-0xfebfffff]
> [ 0.376051] pci 0000:01:0b.0: reg 14: [mem 0xfbfff000-0xfbffffff pref]
> [ 0.380029] pci 0000:01:0b.0: reg 18: [mem 0xfbe00000-0xfbefffff 64bit pref]
> [ 0.384028] pci 0000:01:0b.0: reg 20: [mem 0xfbd00000-0xfbdfffff 64bit pref]
> [ 146.650835] pci 0000:01:0b.0: reg 10: [mem 0xfeb00000-0xfebfffff]
> [ 146.656982] pci 0000:01:0b.0: reg 14: [mem 0xfbfff000-0xfbffffff pref]
> [ 146.663533] pci 0000:01:0b.0: reg 18: [mem 0xfbe00000-0xfbefffff 64bit pref]
> [ 146.670615] pci 0000:01:0b.0: reg 20: [mem 0xfbd00000-0xfbdfffff 64bit pref]
> [ 146.677856] pci 0000:01:0b.0: BAR 0: assigned [mem 0xfef00000-0xfeffffff]
> [ 146.684655] pci 0000:01:0b.0: BAR 0: set to [mem 0xfef00000-0xfeffffff] (PCI address [0xfef00000-0xfeffffff]
> [ 146.694487] pci 0000:01:0b.0: BAR 2: assigned [mem 0xff000000-0xff0fffff 64bit pref]
> [ 146.702247] pci 0000:01:0b.0: BAR 2: set to [mem 0xff000000-0xff0fffff 64bit pref] (PCI address [0xff000000-0xff0fffff]
> [ 146.713031] pci 0000:01:0b.0: BAR 4: assigned [mem 0xff100000-0xff1fffff 64bit pref]
> [ 146.720832] pci 0000:01:0b.0: BAR 4: set to [mem 0xff100000-0xff1fffff 64bit pref] (PCI address [0xff100000-0xff1fffff]
> [ 146.731639] pci 0000:01:0b.0: BAR 1: assigned [mem 0xfee01000-0xfee01fff pref]
> [ 146.738871] pci 0000:01:0b.0: BAR 1: set to [mem 0xfee01000-0xfee01fff pref] (PCI address [0xfee01000-0xfee01fff]
If you run "dmesg -n8", you'll get the dmesg output on the console
directly, and you won't have to use "dmesg | tail" and figure out
which things go with which user actions.
If you turn on CONFIG_PCI_DEBUG, you should get a little more
information. Ideally, I'd like the PCI core to emit enough messages
to debug most problems without recompilation, since that's too
complicated for most users. But obviously we're not there yet.
Since fakephp doesn't actually do anything with a hotplug controller,
I wouldn't think the card would be able to tell that anything happened
when you did the "echo 0 > power" and "echo 1 > rescan". We do a lot
of internal PCI core bookkeeping, but until we update the BARs, that
should be invisible to the card.
I'm also curious about why we decided to assign new values to the BARs.
When we scan the bus, I would think everything would look just like it
did at initial boot. Why can't we just use the existing BAR contents?
On Force, I think the reassignment works because the device is behind
a PCI-to-PCI bridge, we know the apertures of that bridge, and the new
assignments are in those apertures. The apertures don't leave much
flexibility, so the new assignments are almost the same as the original
BIOS ones (only the last two 1MB BARs are swapped):
Force boot-time setup:
pci 0000:00:04.0: PCI bridge to [bus 01-01]
pci 0000:00:04.0: bridge window [io disabled]
pci 0000:00:04.0: bridge window [mem 0xf9100000-0xf91fffff]
pci 0000:00:04.0: bridge window [mem 0xf9400000-0xf96fffff pref]
pci 0000:01:0b.0: reg 10: [mem 0xf9100000-0xf91fffff]
pci 0000:01:0b.0: reg 14: [mem 0xf9600000-0xf9600fff pref]
pci 0000:01:0b.0: reg 18: [mem 0xf9500000-0xf95fffff 64bit pref]
pci 0000:01:0b.0: reg 20: [mem 0xf9400000-0xf94fffff 64bit pref]
Force after rescan:
pci 0000:01:0b.0: BAR 0: assigned [mem 0xf9100000-0xf91fffff]
pci 0000:01:0b.0: BAR 1: assigned [mem 0xf9600000-0xf9600fff pref]
pci 0000:01:0b.0: BAR 2: assigned [mem 0xf9400000-0xf94fffff 64bit pref]
pci 0000:01:0b.0: BAR 4: assigned [mem 0xf9500000-0xf95fffff 64bit pref]
On Trenton, the device is directly below a host bridge, and we don't
know the host bridge apertures, so I think we're picking assignments
that don't work.
Can you turn on CONFIG_PCI_DEBUG and try the patch below? It has a
hack to try to make the assignments use the same BAR values as BIOS
gave us, and a couple printks to find out more about why we try to
assign resources at all. Sorry for the brute-force debugging; I
don't know enough to do it more elegantly.
Bjorn
diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
index 1331fcf..9b78ac7 100644
--- a/arch/x86/pci/common.c
+++ b/arch/x86/pci/common.c
@@ -593,6 +593,29 @@ struct pci_bus * __devinit pci_scan_bus_on_node(int busno, struct pci_ops *ops,
if (!bus)
kfree(sd);
+ if (bus->number == 1) {
+ struct resource *res;
+
+ dev_info(&bus->dev, "test kludge for Trenton\n");
+ dev_info(&bus->dev, "resource 1 %pR\n", bus->resource[1]);
+ dev_info(&bus->dev, "resource 2 %pR\n", bus->resource[2]);
+ res = kzalloc(sizeof(*res), GFP_KERNEL);
+ if (res) {
+ res->start = 0xfeb00000;
+ res->end = 0xfebfffff;
+ res->flags = IORESOURCE_MEM;
+ bus->resource[1] = res;
+ }
+ res = kzalloc(sizeof(*res), GFP_KERNEL);
+ if (res) {
+ res->start = 0xfbd00000;
+ res->end = 0xfbffffff;
+ res->flags = IORESOURCE_MEM | IORESOURCE_PREFETCH;
+ bus->resource[2] = res;
+ }
+ dev_info(&bus->dev, "resource 1 %pR\n", bus->resource[1]);
+ dev_info(&bus->dev, "resource 2 %pR\n", bus->resource[2]);
+ }
return bus;
}
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 5c4ce1b..2442c48 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1246,6 +1246,7 @@ unsigned int __ref pci_rescan_bus(struct pci_bus *bus)
unsigned int max;
struct pci_dev *dev;
+ dev_info(&bus->dev, "rescanning bus\n");
max = pci_scan_child_bus(bus);
down_read(&pci_bus_sem);
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index 176615e..a12930c 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -64,6 +64,7 @@ int pci_remove_device_safe(struct pci_dev *dev)
void pci_remove_bus(struct pci_bus *pci_bus)
{
+ dev_info(&pci_bus->dev, "removing bus\n");
pci_proc_detach_bus(pci_bus);
down_write(&pci_bus_sem);
@@ -93,6 +94,7 @@ EXPORT_SYMBOL(pci_remove_bus);
*/
void pci_remove_bus_device(struct pci_dev *dev)
{
+ dev_info(&dev->dev, "removing device\n");
pci_stop_bus_device(dev);
if (dev->subordinate) {
struct pci_bus *b = dev->subordinate;
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 7d678bb..27b0eb4 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -223,6 +223,7 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)
if (r->flags & IORESOURCE_PCI_FIXED)
continue;
+ dev_info(&dev->dev, "pdev_sort_resources: res %d %pR parent %pR\n", i, r, r->parent);
if (!(r->flags) || r->parent)
continue;
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
