On Thu, Nov 05, 2009 at 04:00:41PM -0600, Bjorn Helgaas wrote: > On Thursday 05 November 2009 10:14:56 am Ira W. Snyder wrote: > > On Wed, Nov 04, 2009 at 04:25:41PM -0800, Ira W. Snyder wrote: > > > On Wed, Nov 04, 2009 at 06:18:19PM -0600, Bjorn Helgaas wrote: > > > > > You mentioned using "memmap=" to change the aperture we allocate > > > > from. Can you tweak that to make the after-hotplug mappings be > > > > closer to the original BIOS ones? I know you tried that already; > > > > I guess I'm just wondering if the magic lower boundary is something > > > > like 0xfb000000 rather than 0xf0000000. > > > > Ok, I just ran these tests. Using memmap= to reserve memory and force > > Linux to place allocations higher didn't work at all. > > > > I started at 0xf8000000 and moved up from there. Even after getting so > > the BARs were re-assigned to higher addresses than the original BIOS > > addresses didn't help. Reading the BAR still produces all 0xff's. > > > > Here are some logs from my last run, which show the BIOS and Linux > > assigned addresses: > > > > [ 0.372360] pci 0000:01:0b.0: reg 10: [mem 0xfeb00000-0xfebfffff] > > [ 0.376051] pci 0000:01:0b.0: reg 14: [mem 0xfbfff000-0xfbffffff pref] > > [ 0.380029] pci 0000:01:0b.0: reg 18: [mem 0xfbe00000-0xfbefffff 64bit pref] > > [ 0.384028] pci 0000:01:0b.0: reg 20: [mem 0xfbd00000-0xfbdfffff 64bit pref] > > [ 146.650835] pci 0000:01:0b.0: reg 10: [mem 0xfeb00000-0xfebfffff] > > [ 146.656982] pci 0000:01:0b.0: reg 14: [mem 0xfbfff000-0xfbffffff pref] > > [ 146.663533] pci 0000:01:0b.0: reg 18: [mem 0xfbe00000-0xfbefffff 64bit pref] > > [ 146.670615] pci 0000:01:0b.0: reg 20: [mem 0xfbd00000-0xfbdfffff 64bit pref] > > [ 146.677856] pci 0000:01:0b.0: BAR 0: assigned [mem 0xfef00000-0xfeffffff] > > [ 146.684655] pci 0000:01:0b.0: BAR 0: set to [mem 0xfef00000-0xfeffffff] (PCI address [0xfef00000-0xfeffffff] > > [ 146.694487] pci 0000:01:0b.0: BAR 2: assigned [mem 0xff000000-0xff0fffff 64bit pref] > > [ 146.702247] pci 0000:01:0b.0: BAR 2: set to [mem 0xff000000-0xff0fffff 64bit pref] (PCI address [0xff000000-0xff0fffff] > > [ 146.713031] pci 0000:01:0b.0: BAR 4: assigned [mem 0xff100000-0xff1fffff 64bit pref] > > [ 146.720832] pci 0000:01:0b.0: BAR 4: set to [mem 0xff100000-0xff1fffff 64bit pref] (PCI address [0xff100000-0xff1fffff] > > [ 146.731639] pci 0000:01:0b.0: BAR 1: assigned [mem 0xfee01000-0xfee01fff pref] > > [ 146.738871] pci 0000:01:0b.0: BAR 1: set to [mem 0xfee01000-0xfee01fff pref] (PCI address [0xfee01000-0xfee01fff] > > If you run "dmesg -n8", you'll get the dmesg output on the console > directly, and you won't have to use "dmesg | tail" and figure out > which things go with which user actions. > Yes, I know. :) I have a serial console hooked up, and I don't know how to tell the kernel to change the destination for log messages to my ssh session. > If you turn on CONFIG_PCI_DEBUG, you should get a little more > information. Ideally, I'd like the PCI core to emit enough messages > to debug most problems without recompilation, since that's too > complicated for most users. But obviously we're not there yet. > > Since fakephp doesn't actually do anything with a hotplug controller, > I wouldn't think the card would be able to tell that anything happened > when you did the "echo 0 > power" and "echo 1 > rescan". We do a lot > of internal PCI core bookkeeping, but until we update the BARs, that > should be invisible to the card. > Right. I can emulate a power-off by pressing the reset switch on my card. It resets the PCI BAR's, along with most of the rest of the board. > I'm also curious about why we decided to assign new values to the BARs. > When we scan the bus, I would think everything would look just like it > did at initial boot. Why can't we just use the existing BAR contents? > I have verified that this is the case. I can scan the PCI bus from the PCI card itself. It is basically a full computer with a powerpc core. The kernel messages do show this as well. The kernel prints the BAR's before they are assigned. > On Force, I think the reassignment works because the device is behind > a PCI-to-PCI bridge, we know the apertures of that bridge, and the new > assignments are in those apertures. The apertures don't leave much > flexibility, so the new assignments are almost the same as the original > BIOS ones (only the last two 1MB BARs are swapped): > > Force boot-time setup: > pci 0000:00:04.0: PCI bridge to [bus 01-01] > pci 0000:00:04.0: bridge window [io disabled] > pci 0000:00:04.0: bridge window [mem 0xf9100000-0xf91fffff] > pci 0000:00:04.0: bridge window [mem 0xf9400000-0xf96fffff pref] > pci 0000:01:0b.0: reg 10: [mem 0xf9100000-0xf91fffff] > pci 0000:01:0b.0: reg 14: [mem 0xf9600000-0xf9600fff pref] > pci 0000:01:0b.0: reg 18: [mem 0xf9500000-0xf95fffff 64bit pref] > pci 0000:01:0b.0: reg 20: [mem 0xf9400000-0xf94fffff 64bit pref] > > Force after rescan: > pci 0000:01:0b.0: BAR 0: assigned [mem 0xf9100000-0xf91fffff] > pci 0000:01:0b.0: BAR 1: assigned [mem 0xf9600000-0xf9600fff pref] > pci 0000:01:0b.0: BAR 2: assigned [mem 0xf9400000-0xf94fffff 64bit pref] > pci 0000:01:0b.0: BAR 4: assigned [mem 0xf9500000-0xf95fffff 64bit pref] > > On Trenton, the device is directly below a host bridge, and we don't > know the host bridge apertures, so I think we're picking assignments > that don't work. > > Can you turn on CONFIG_PCI_DEBUG and try the patch below? It has a > hack to try to make the assignments use the same BAR values as BIOS > gave us, and a couple printks to find out more about why we try to > assign resources at all. Sorry for the brute-force debugging; I > don't know enough to do it more elegantly. > Will do. Brute force is fine by me. :) Ira -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
