Re: PCIe IO space support on Tilera GX: Is there any one who can confirm my modification to fix it is OK?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We're not using 3.6.x, we're using is from MDE-4.1.0 from Tilera and
it patch 3.0.38.
>From its release notes that PCIe I/O space is already supported.
I provide diff of pci_gx.c between 3.6.3 and MDE-4.1.0 for a hint,
since I don't know
if it's allowed to use attached file in mail list, and their patch for
3.0.38 is bigger than
7MB.

For mvsas, it seems do think 0 I/O address invalied. Some code from
drivers/scsi/mvsas/mv_init.c:
int mvs_ioremap(struct mvs_info *mvi, int bar, int bar_ex)
{
	unsigned long res_start, res_len, res_flag, res_flag_ex = 0;
	struct pci_dev *pdev = mvi->pdev;
	if (bar_ex != -1) {
		/*
		 * ioremap main and peripheral registers
		 */
		res_start = pci_resource_start(pdev, bar_ex);
		res_len = pci_resource_len(pdev, bar_ex);
		if (!res_start || !res_len)
			goto err_out;

		res_flag_ex = pci_resource_flags(pdev, bar_ex);
		if (res_flag_ex & IORESOURCE_MEM) {
			if (res_flag_ex & IORESOURCE_CACHEABLE)
				mvi->regs_ex = ioremap(res_start, res_len);
			else
				mvi->regs_ex = ioremap_nocache(res_start,
						res_len);
		} else
			mvi->regs_ex = (void *)res_start;
		if (!mvi->regs_ex)
			goto err_out;
	}

	res_start = pci_resource_start(pdev, bar);
	res_len = pci_resource_len(pdev, bar);
	if (!res_start || !res_len)
		goto err_out;

	res_flag = pci_resource_flags(pdev, bar);
	if (res_flag & IORESOURCE_CACHEABLE)
		mvi->regs = ioremap(res_start, res_len);
	else
		mvi->regs = ioremap_nocache(res_start, res_len);

	if (!mvi->regs) {
		if (mvi->regs_ex && (res_flag_ex & IORESOURCE_MEM))
			iounmap(mvi->regs_ex);
		mvi->regs_ex = NULL;
		goto err_out;
	}

	return 0;
err_out:
	return -1;
}

For 64xx, bar_ex is I/O space, and
		res_start = pci_resource_start(pdev, bar_ex);
		res_len = pci_resource_len(pdev, bar_ex);
		if (!res_start || !res_len)
			goto err_out;
will cause driver loading failed.

When we using MDE-4.0.0 it don't support I/O space, I just bypass
these check since after
investigate all code of mvsas it seems that I/O space map to BAR 2 is
not really used.

When the same card inserted into x86 platform, the allocated I/O space
is not start from 0,
so it works fine.


On Fri, Oct 26, 2012 at 4:03 PM, Bjorn Helgaas <bhelgaas@xxxxxxxxxx> wrote:
> [+cc Chris, also a few comments below]
>
> On Fri, Oct 26, 2012 at 12:59 AM, Cyberman Wu <cypher.w@xxxxxxxxx> wrote:
>> After we upgrade to MDE 4.1.0 from Tilera, we encounter a problem that
>> only on HighPoint 2680 card works, I've
>> tried to fix it, but since most time I'm working in user space, I'm
>> not sure my fix is enough. Their FAE said that
>> the guy who add PCIe I/O space support is on vacation and I can't get
>> help from him now, I hope maybe there
>> will have somebody can help.
>>
>>
>> Problem we encountered:
>>
>> pci 0000:00:00.0: BAR 8: assigned [mem 0x100c0000000-0x100c00fffff]
>> pci 0000:00:00.0: BAR 9: assigned [mem 0x100c0100000-0x100c01fffff pref]
>> pci 0000:00:00.0: BAR 7: assigned [io  0x0000-0x0fff]
>> pci 0000:01:00.0: BAR 6: assigned [mem 0x100c0100000-0x100c013ffff pref]
>> pci 0000:01:00.0: BAR 6: set to [mem 0x100c0100000-0x100c013ffff pref]
>> (PCI address [0xc0100000-0xc013ffff])
>> pci 0000:01:00.0: BAR 4: assigned [mem 0x100c0000000-0x100c000ffff 64bit]
>> pci 0000:01:00.0: BAR 4: set to [mem 0x100c0000000-0x100c000ffff
>> 64bit] (PCI address [0xc0000000-0xc000ffff])
>> pci 0000:01:00.0: BAR 2: assigned [io  0x0000-0x007f]
>> pci 0000:01:00.0: BAR 2: set to [io  0x0000-0x007f] (PCI address [0x0-0x7f])
>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>> pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
>> pci 0000:00:00.0:   bridge window [mem 0x100c0000000-0x100c00fffff]
>> pci 0000:00:00.0:   bridge window [mem 0x100c0100000-0x100c01fffff pref]
>> pci 0001:00:00.0: BAR 8: assigned [mem 0x101c0000000-0x101c00fffff]
>> pci 0001:00:00.0: BAR 9: assigned [mem 0x101c0100000-0x101c01fffff pref]
>> pci 0001:00:00.0: BAR 7: assigned [io  0x0000-0x0fff]
>> pci 0001:01:00.0: BAR 6: assigned [mem 0x101c0100000-0x101c013ffff pref]
>> pci 0001:01:00.0: BAR 6: set to [mem 0x101c0100000-0x101c013ffff pref]
>> (PCI address [0xc0100000-0xc013ffff])
>> pci 0001:01:00.0: BAR 4: assigned [mem 0x101c0000000-0x101c000ffff 64bit]
>> pci 0001:01:00.0: BAR 4: set to [mem 0x101c0000000-0x101c000ffff
>> 64bit] (PCI address [0xc0000000-0xc000ffff])
>> pci 0001:01:00.0: BAR 2: assigned [io  0x0000-0x007f]
>> pci 0001:01:00.0: BAR 2: set to [io  0x0000-0x007f] (PCI address [0x0-0x7f])
>> pci 0001:00:00.0: PCI bridge to [bus 01-01]
>> pci 0001:00:00.0:   bridge window [io  0x0000-0x0fff]
>> pci 0001:00:00.0:   bridge window [mem 0x101c0000000-0x101c00fffff]
>> pci 0001:00:00.0:   bridge window [mem 0x101c0100000-0x101c01fffff pref]
>> pci 0000:00:00.0: enabling device (0006 -> 0007)
>> pci 0001:00:00.0: enabling device (0006 -> 0007)
>> pci_bus 0000:00: resource 0 [io  0x0000-0xffffffff]
>> pci_bus 0000:00: resource 1 [mem 0x100c0000000-0x100ffffffff]
>> pci_bus 0000:01: resource 0 [io  0x0000-0x0fff]
>> pci_bus 0000:01: resource 1 [mem 0x100c0000000-0x100c00fffff]
>> pci_bus 0000:01: resource 2 [mem 0x100c0100000-0x100c01fffff pref]
>> pci_bus 0001:00: resource 0 [io  0x0000-0xffffffff]
>> pci_bus 0001:00: resource 1 [mem 0x101c0000000-0x101ffffffff]
>> pci_bus 0001:01: resource 0 [io  0x0000-0x0fff]
>> pci_bus 0001:01: resource 1 [mem 0x101c0000000-0x101c00fffff]
>> pci_bus 0001:01: resource 2 [mem 0x101c0100000-0x101c01fffff pref]
>> ......
>> mvsas 0000:01:00.0: mvsas: driver version 0.8.2
>> mvsas 0000:01:00.0: enabling device (0000 -> 0003)
>> mvsas 0000:01:00.0: enabling bus mastering
>> mvsas 0000:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
>> mvsas 0000:01:00.0: Phy3 : No sig fis
>> scsi0 : mvsas
>> ......
>> mvsas 0001:01:00.0: mvsas: driver version 0.8.2
>> mvsas 0001:01:00.0: enabling device (0000 -> 0003)
>> mvsas 0001:01:00.0: enabling bus mastering
>> mvsas 0001:01:00.0: BAR 2: can't reserve [io  0x0000-0x007f]
>> mvsas: probe of 0001:01:00.0 failed with error -16
>>
>>
>> My modification:
>>
>> --- /opt/tilera/TileraMDE-4.1.0.148119/tilegx/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c     2012-10-22
>> 14:56:59.783096378 +0800
>> +++ Tilera_src/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c    2012-10-26
>> 13:55:02.731947886 +0800
>> @@ -368,6 +368,10 @@
>>         int num_trio_shims = 0;
>>         int ctl_index = 0;
>>         int i, j;
>> +    // Modified by Cyberman Wu on Oct 25th, 2012.
>> +       resource_size_t io_mem_start;
>> +       resource_size_t io_mem_end;
>> +       resource_size_t io_mem_size;
>>
>>         if (!pci_probe) {
>>                 pr_info("PCI: disabled by boot argument\n");
>> @@ -457,6 +461,18 @@
>>         }
>>
>>  out:
>> +       // Use IO memory space 0~0xffffffff for every controller will
>> +       // cause device on controller other than the first failed to
>> +       // load driver if it using IO regions.
>> +       // Is reserve the first 4K IO address space OK? Tilera use
>> +       // IO space address begin from 0, but some drivers in Linux
>> +       // recognize 0 address a error, say, mvsas, so for compatiblity
>> +       // reserve some address from 0 should be better?
>
> It's not that mvsas thinks I/O address 0 is invalid, it's just that we
> already assigned [io 0x0000-0x007f] to the device at 0000:01:00.0:
>
>   pci 0000:01:00.0: BAR 2: set to [io  0x0000-0x007f]
>
> so that range can't also be assigned to 0001:01:00.0.
>
>> +       // Modified by Cyberman Wu on Oct 25th, 2012.
>> +       io_mem_start = 4096;
>> +       io_mem_end = (resource_size_t)IO_SPACE_LIMIT + 1;
>> +       io_mem_size = (io_mem_end - io_mem_start) / num_rc_controllers;
>> +       io_mem_size &= ~3;
>>         /*
>>          * Configure each PCIe RC port.
>>          */
>> @@ -470,8 +486,9 @@
>>                 controller->index = i;
>>                 controller->ops = &tile_cfg_ops;
>>
>> -               controller->io_space.start = 0;
>> -               controller->io_space.end = IO_SPACE_LIMIT;
>> +               // Modified by Cyberman Wu on Oct 25th, 2012.
>> +               controller->io_space.start = io_mem_start + (i * io_mem_size);
>> +               controller->io_space.end = controller->io_space.start + io_mem_size - 1;
>>                 controller->io_space.flags = IORESOURCE_IO;
>>                 snprintf(controller->io_space_name,
>>                          sizeof(controller->io_space_name),
>>
>>
>> Please note that we're using MDE-4.1.0, which use kernel 3.0.38, patch
>> it and reversion it
>> to 2.6.40.38.
>> I've checked source code under arch/tile of kernel 3.6.3 and PCIe I/O
>> space support is still
>> not here. Below is diff of arch/tile/pci_gx.c between kernel 3.6.3 and
>> MDE-4.1.0:
>
> Per http://lkml.indiana.edu/hypermail/linux/kernel/1205.1/01176.html,
> Chris considered adding I/O space support and decided against it at
> that time, partly because it would use up a TRIO PIO region.
>
> I don't know his current thoughts.  Possibly it could be done under a
> config option or something.
>
> But of course, you'd have to do it by adding I/O space support to the
> current 3.6 kernel *without* reverting all the other changes that have
> been made since 2.6.40.
>
>> --- .cache/.fr-9Oo37J/linux-3.6.3/arch/tile/kernel/pci_gx.c     2012-10-22
>> 00:32:56.000000000 +0800
>> +++ /opt/tilera/TileraMDE-4.1.0.148119/tilegx/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c     2012-10-22
>> 14:56:59.783096378 +0800
>> @@ -69,19 +69,18 @@
>>   * a HW PCIe link-training bug. The exact delay is specified with
>>   * a kernel boot argument in the form of "pcie_rc_delay=T,P,S",
>>   * where T is the TRIO instance number, P is the port number and S is
>> - * the delay in seconds. If the delay is not provided, the value
>> - * will be DEFAULT_RC_DELAY.
>> + * the delay in seconds. If the argument is specified, but the delay is
>> + * not provided, the value will be DEFAULT_RC_DELAY.
>>   */
>>  static int __devinitdata rc_delay[TILEGX_NUM_TRIO][TILEGX_TRIO_PCIES];
>>
>>  /* Default number of seconds that the PCIe RC port probe can be delayed. */
>>  #define DEFAULT_RC_DELAY       10
>>
>> -/* Max number of seconds that the PCIe RC port probe can be delayed. */
>> -#define MAX_RC_DELAY           20
>> -
>> +#if !defined(GX_FPGA)
>>  /* Array of the PCIe ports configuration info obtained from the BIB. */
>>  struct pcie_port_property pcie_ports[TILEGX_NUM_TRIO][TILEGX_TRIO_PCIES];
>> +#endif
>>
>>  /* All drivers share the TRIO contexts defined here. */
>>  gxio_trio_context_t trio_contexts[TILEGX_NUM_TRIO];
>> @@ -97,6 +96,41 @@
>>  static struct cpumask intr_cpus_map;
>>
>>  /*
>> + * Convert a resource to a PCI device bus address or bus window.
>> + */
>> +void __devinit
>> +pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
>> +                       struct resource *res)
>> +{
>> +       struct pci_controller *controller =
>> +               (struct pci_controller *)dev->sysdata;
>> +       unsigned long offset = 0;
>> +
>> +       if (res->flags & IORESOURCE_MEM)
>> +               offset = controller->mem_offset;
>> +
>> +       region->start = res->start - offset;
>> +       region->end = res->end - offset;
>> +}
>> +EXPORT_SYMBOL(pcibios_resource_to_bus);
>> +
>> +void __devinit
>> +pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
>> +                       struct pci_bus_region *region)
>> +{
>> +       struct pci_controller *controller =
>> +               (struct pci_controller *)dev->sysdata;
>> +       unsigned long offset = 0;
>> +
>> +       if (res->flags & IORESOURCE_MEM)
>> +               offset = controller->mem_offset;
>> +
>> +       res->start = region->start + offset;
>> +       res->end = region->end + offset;
>> +}
>> +EXPORT_SYMBOL(pcibios_bus_to_resource);
>> +
>> +/*
>>   * We don't need to worry about the alignment of resources.
>>   */
>>  resource_size_t pcibios_align_resource(void *data, const struct resource *res,
>> @@ -274,6 +308,10 @@
>>
>>         cpumask_copy(&intr_cpus_map, cpu_online_mask);
>>
>> +#ifdef CONFIG_DATAPLANE
>> +       /* Remove dataplane cpus. */
>> +       cpumask_andnot(&intr_cpus_map, &intr_cpus_map, &dataplane_map);
>> +#endif
>>
>>         for (i = 0; i < 4; i++) {
>>                 gxio_trio_context_t *context = controller->trio;
>> @@ -325,7 +363,7 @@
>>   *
>>   * Returns the number of controllers discovered.
>>   */
>> -int __init tile_pci_init(void)
>> +int __devinit tile_pci_init(void)
>>  {
>>         int num_trio_shims = 0;
>>         int ctl_index = 0;
>> @@ -359,6 +397,7 @@
>>          * We look at the Board Information Block first and then see if there
>>          * are any overriding configuration by the HW strapping pin.
>>          */
>> +#if !defined(GX_FPGA)
>>         for (i = 0; i < TILEGX_NUM_TRIO; i++) {
>>                 gxio_trio_context_t *context = &trio_contexts[i];
>>                 int ret;
>> @@ -386,6 +425,13 @@
>>                         }
>>                 }
>>         }
>> +#else
>> +       /*
>> +        * For now, just assume that there is a single RC port on trio/0.
>> +        */
>> +       num_rc_controllers = 1;
>> +       pcie_rc[0][2] = 1;
>> +#endif
>>
>>         /*
>>          * Return if no PCIe ports are configured to operate in RC mode.
>> @@ -424,13 +470,20 @@
>>                 controller->index = i;
>>                 controller->ops = &tile_cfg_ops;
>>
>> +               controller->io_space.start = 0;
>> +               controller->io_space.end = IO_SPACE_LIMIT;
>> +               controller->io_space.flags = IORESOURCE_IO;
>> +               snprintf(controller->io_space_name,
>> +                        sizeof(controller->io_space_name),
>> +                        "PCI I/O domain %d", i);
>> +               controller->io_space.name = controller->io_space_name;
>> +
>>                 /*
>>                  * The PCI memory resource is located above the PA space.
>>                  * For every host bridge, the BAR window or the MMIO aperture
>>                  * is in range [3GB, 4GB - 1] of a 4GB space beyond the
>>                  * PA space.
>>                  */
>> -
>>                 controller->mem_offset = TILE_PCI_MEM_START +
>>                         (i * TILE_PCI_BAR_WINDOW_TOP);
>>                 controller->mem_space.start = controller->mem_offset +
>> @@ -451,7 +504,7 @@
>>   * (pin - 1) converts from the PCI standard's [1:4] convention to
>>   * a normal [0:3] range.
>>   */
>> -static int tile_map_irq(const struct pci_dev *dev, u8 device, u8 pin)
>> +static int tile_map_irq(struct pci_dev *dev, u8 device, u8 pin)
>>  {
>>         struct pci_controller *controller =
>>                 (struct pci_controller *)dev->sysdata;
>> @@ -463,11 +516,12 @@
>>                                                 controller)
>>  {
>>         gxio_trio_context_t *trio_context = controller->trio;
>> -       struct pci_bus *root_bus = controller->root_bus;
>>         TRIO_PCIE_RC_DEVICE_CONTROL_t dev_control;
>>         TRIO_PCIE_RC_DEVICE_CAP_t rc_dev_cap;
>> +       unsigned int smallest_max_payload;
>> +       struct pci_dev *dev = NULL;
>>         unsigned int reg_offset;
>> -       struct pci_bus *child;
>> +       u16 new_values;
>>         int mac;
>>         int err;
>>
>> @@ -508,33 +562,59 @@
>>         __gxio_mmio_write32(trio_context->mmio_base_mac + reg_offset,
>>                                                 rc_dev_cap.word);
>>
>> -       /* Configure PCI Express MPS setting. */
>> -       list_for_each_entry(child, &root_bus->children, node) {
>> -               struct pci_dev *self = child->self;
>> -               if (!self)
>> +       smallest_max_payload = rc_dev_cap.mps_sup;
>> +
>> +       /* Scan for the smallest maximum payload size. */
>> +       while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
>> +               int pcie_caps_offset;
>> +               u32 devcap;
>> +               int max_payload;
>> +
>> +               /* Skip device that is not in this PCIe domain. */
>> +               if ((struct pci_controller *)dev->sysdata != controller)
>>                         continue;
>>
>> -               pcie_bus_configure_settings(child, self->pcie_mpss);
>> +               pcie_caps_offset = pci_find_capability(dev, PCI_CAP_ID_EXP);
>> +               if (pcie_caps_offset == 0)
>> +                       continue;
>> +
>> +               pci_read_config_dword(dev, pcie_caps_offset + PCI_EXP_DEVCAP,
>> +                                     &devcap);
>> +               max_payload = devcap & PCI_EXP_DEVCAP_PAYLOAD;
>> +               if (max_payload < smallest_max_payload)
>> +                       smallest_max_payload = max_payload;
>> +       }
>> +
>> +       /* Now, set the max_payload_size for all devices to that value. */
>> +       new_values = smallest_max_payload << 5;
>> +       while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
>> +               int pcie_caps_offset;
>> +               u16 devctl;
>> +
>> +               /* Skip device that is not in this PCIe domain. */
>> +               if ((struct pci_controller *)dev->sysdata != controller)
>> +                       continue;
>> +
>> +               pcie_caps_offset = pci_find_capability(dev, PCI_CAP_ID_EXP);
>> +               if (pcie_caps_offset == 0)
>> +                       continue;
>> +
>> +               pci_read_config_word(dev, pcie_caps_offset + PCI_EXP_DEVCTL,
>> +                                    &devctl);
>> +               devctl &= ~PCI_EXP_DEVCTL_PAYLOAD;
>> +               devctl |= new_values;
>> +               pci_write_config_word(dev, pcie_caps_offset + PCI_EXP_DEVCTL,
>> +                                     devctl);
>>         }
>>
>>         /*
>>          * Set the mac_config register in trio based on the MPS/MRS of the link.
>>          */
>> -       reg_offset =
>> -               (TRIO_PCIE_RC_DEVICE_CONTROL <<
>> -                       TRIO_CFG_REGION_ADDR__REG_SHIFT) |
>> -               (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_STANDARD <<
>> -                       TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
>> -               (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
>> -
>> -       dev_control.word = __gxio_mmio_read32(trio_context->mmio_base_mac +
>> -                                               reg_offset);
>> -
>>         err = gxio_trio_set_mps_mrs(trio_context,
>> -                                   dev_control.max_payload_size,
>> +                                   smallest_max_payload,
>>                                     dev_control.max_read_req_sz,
>>                                     mac);
>> -        if (err < 0) {
>> +       if (err < 0) {
>>                 pr_err("PCI: PCIE_CONFIGURE_MAC_MPS_MRS failure, "
>>                         "MAC %d on TRIO %d\n",
>>                         mac, controller->trio_index);
>> @@ -571,14 +651,9 @@
>>                 if (!isdigit(*str))
>>                         return -EINVAL;
>>                 delay = simple_strtoul(str, (char **)&str, 10);
>> -               if (delay > MAX_RC_DELAY)
>> -                       return -EINVAL;
>>         }
>>
>>         rc_delay[trio_index][mac] = delay ? : DEFAULT_RC_DELAY;
>> -       pr_info("Delaying PCIe RC link training for %u sec"
>> -               " on MAC %lu on TRIO %lu\n", rc_delay[trio_index][mac],
>> -               mac, trio_index);
>>         return 0;
>>  }
>>  early_param("pcie_rc_delay", setup_pcie_rc_delay);
>> @@ -586,18 +661,14 @@
>>  /*
>>   * PCI initialization entry point, called by subsys_initcall.
>>   */
>> -int __init pcibios_init(void)
>> +int __devinit pcibios_init(void)
>>  {
>>         resource_size_t offset;
>> -       LIST_HEAD(resources);
>>         int next_busno;
>>         int i;
>>
>>         tile_pci_init();
>>
>> -       if (num_rc_controllers == 0 && num_ep_controllers == 0)
>> -               return 0;
>> -
>>         /*
>>          * We loop over all the TRIO shims and set up the MMIO mappings.
>>          */
>> @@ -623,6 +694,9 @@
>>                 }
>>         }
>>
>> +       if (num_rc_controllers == 0 && num_ep_controllers == 0)
>> +               return 0;
>> +
>>         /*
>>          * Delay a bit in case devices aren't ready.  Some devices are
>>          * known to require at least 20ms here, but we use a more
>> @@ -684,15 +758,36 @@
>>                 }
>>
>>                 /*
>> -                * Delay the RC link training if needed.
>> +                * Delay the bus probe if needed.
>>                  */
>> -               if (rc_delay[trio_index][mac])
>> +               if (rc_delay[trio_index][mac]) {
>> +                       pr_info("Delaying PCIe RC link training for %d sec"
>> +                               " on MAC %d on TRIO %d\n",
>> +                               rc_delay[trio_index][mac], mac,
>> +                               trio_index);
>>                         msleep(rc_delay[trio_index][mac] * 1000);
>> +               }
>>
>> -               ret = gxio_trio_force_rc_link_up(trio_context, mac);
>> -               if (ret < 0)
>> -                       pr_err("PCI: PCIE_FORCE_LINK_UP failure, "
>> -                               "MAC %d on TRIO %d\n", mac, trio_index);
>> +               /*
>> +                * Check for PCIe link-up status to decide if we need
>> +                * to force the link to come up.
>> +                */
>> +               reg_offset =
>> +                       (TRIO_PCIE_INTFC_PORT_STATUS <<
>> +                               TRIO_CFG_REGION_ADDR__REG_SHIFT) |
>> +                       (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_INTERFACE <<
>> +                               TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
>> +                       (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
>> +
>> +               port_status.word =
>> +                       __gxio_mmio_read(trio_context->mmio_base_mac +
>> +                                        reg_offset);
>> +               if (!port_status.dl_up) {
>> +                       ret = gxio_trio_force_rc_link_up(trio_context, mac);
>> +                       if (ret < 0)
>> +                               pr_err("PCI: PCIE_FORCE_LINK_UP failure, "
>> +                                       "MAC %d on TRIO %d\n", mac, trio_index);
>> +               }
>>
>>                 pr_info("PCI: Found PCI controller #%d on TRIO %d MAC %d\n", i,
>>                         trio_index, controller->mac);
>> @@ -704,22 +799,20 @@
>>                 msleep(1000);
>>
>>                 /*
>> -                * Check for PCIe link-up status.
>> +                * Check for PCIe link-up status again.
>>                  */
>> -
>> -               reg_offset =
>> -                       (TRIO_PCIE_INTFC_PORT_STATUS <<
>> -                               TRIO_CFG_REGION_ADDR__REG_SHIFT) |
>> -                       (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_INTERFACE <<
>> -                               TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
>> -                       (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
>> -
>>                 port_status.word =
>>                         __gxio_mmio_read(trio_context->mmio_base_mac +
>>                                          reg_offset);
>>                 if (!port_status.dl_up) {
>> -                       pr_err("PCI: link is down, MAC %d on TRIO %d\n",
>> -                               mac, trio_index);
>> +                       if (pcie_ports[trio_index][mac].removable) {
>> +                               pr_info("PCI: link is down, MAC %d on TRIO %d",
>> +                                       mac, trio_index);
>> +                               pr_info("This is expected if no PCIe card"
>> +                                       " is connected to this link");
>> +                       } else
>> +                               pr_err("PCI: link is down, MAC %d on TRIO %d",
>> +                                       mac, trio_index);
>>                         continue;
>>                 }
>>
>> @@ -842,19 +935,22 @@
>>                 }
>>
>>                 /*
>> -                * The PCI memory resource is located above the PA space.
>> -                * The memory range for the PCI root bus should not overlap
>> -                * with the physical RAM
>> +                * This comes from the generic Linux PCI driver.
>> +                *
>> +                * It reads the PCI tree for this bus into the Linux
>> +                * data structures.
>> +                *
>> +                * This is inlined in linux/pci.h and calls into
>> +                * pci_scan_bus_parented() in probe.c.
>>                  */
>> -               pci_add_resource_offset(&resources, &controller->mem_space,
>> -                                       controller->mem_offset);
>> -
>> -               controller->first_busno = next_busno;
>> -               bus = pci_scan_root_bus(NULL, next_busno, controller->ops,
>> -                                       controller, &resources);
>> +               controller->first_busno= next_busno;
>> +               bus = pci_scan_bus(next_busno, controller->ops, controller);
>>                 controller->root_bus = bus;
>> -               next_busno = bus->busn_res.end + 1;
>> -
>> +#if 0
>> +               next_busno = bus->subordinate + 1;
>> +#else
>> +               next_busno = 0;
>> +#endif
>>         }
>>
>>         /* Do machine dependent PCI interrupt routing */
>> @@ -951,6 +1047,37 @@
>>                 }
>>
>>                 /*
>> +                * Alloc a PIO region for PCI I/O space access for each RC port.
>> +                */
>> +               ret = gxio_trio_alloc_pio_regions(trio_context, 1, 0, 0);
>> +               if (ret < 0) {
>> +                       pr_err("PCI: I/O PIO alloc failure on TRIO %d mac %d, "
>> +                               "give up\n", controller->trio_index,
>> +                               controller->mac);
>> +
>> +                       continue;
>> +               }
>> +
>> +               controller->pio_io_index = ret;
>> +
>> +               /*
>> +                * For PIO IO, the bus_address_hi parameter is hard-coded 0
>> +                * because PCI I/O address space is 32-bit.
>> +                */
>> +               ret = gxio_trio_init_pio_region_aux(trio_context,
>> +                                                   controller->pio_io_index,
>> +                                                   controller->mac,
>> +                                                   0,
>> +                                                   HV_TRIO_PIO_FLAG_IO_SPACE);
>> +               if (ret < 0) {
>> +                       pr_err("PCI: I/O PIO init failure on TRIO %d mac %d, "
>> +                               "give up\n", controller->trio_index,
>> +                               controller->mac);
>> +
>> +                       continue;
>> +               }
>> +
>> +               /*
>>                  * Configure a Mem-Map region for each memory controller so
>>                  * that Linux can map all of its PA space to the PCI bus.
>>                  * Use the IOMMU to handle hash-for-home memory.
>> @@ -1015,9 +1142,22 @@
>>  }
>>  subsys_initcall(pcibios_init);
>>
>> -/* Note: to be deleted after Linux 3.6 merge. */
>> +/*
>> + * PCI scan code calls the arch specific pcibios_fixup_bus() each time it scans
>> + * a new bridge. Called after each bus is probed, but before its children are
>> + * examined.
>> + */
>>  void __devinit pcibios_fixup_bus(struct pci_bus *bus)
>>  {
>> +       struct pci_dev *dev = bus->self;
>> +
>> +       if (!dev) {
>> +               struct pci_controller *controller = bus->sysdata;
>> +
>> +               /* This is the root bus. */
>> +               bus->resource[0] = &controller->io_space;
>> +               bus->resource[1] = &controller->mem_space;
>> +       }
>>  }
>>
>>  /*
>> @@ -1043,8 +1183,7 @@
>>
>>  /*
>>   * Enable memory address decoding, as appropriate, for the
>> - * device described by the 'dev' struct. The I/O decoding
>> - * is disabled, though the TILE-Gx supports I/O addressing.
>> + * device described by the 'dev' struct.
>>   *
>>   * This is called from the generic PCI layer, and can be called
>>   * for bridges or endpoints.
>> @@ -1126,10 +1265,95 @@
>>          * We need to keep the PCI bus address's in-page offset in the VA.
>>          */
>>         return iorpc_ioremap(trio_fd, offset, size) +
>> -               (phys_addr & (PAGE_SIZE - 1));
>> +               (start & (PAGE_SIZE - 1));
>>  }
>>  EXPORT_SYMBOL(ioremap);
>>
>> +/* Map a PCI I/O address into VA space. */
>> +void __iomem *ioport_map(unsigned long port, unsigned int size)
>> +{
>> +       struct pci_controller *controller = NULL;
>> +       resource_size_t bar_start;
>> +       resource_size_t bar_end;
>> +       resource_size_t offset;
>> +       resource_size_t start;
>> +       resource_size_t end;
>> +       int trio_fd;
>> +       int i;
>> +
>> +       start = port;
>> +       end = port + size - 1;
>> +
>> +       /*
>> +        * In the following, each PCI controller's mem_resources[0]
>> +        * represents its PCI I/O resource. By searching port in each
>> +        * controller's mem_resources[0], we can determine the controller
>> +        * that should accept the PCI I/O access.
>> +        */
>> +
>> +       for (i = 0; i < num_rc_controllers; i++) {
>> +               /*
>> +                * Skip controllers that are not properly initialized or
>> +                * have down links.
>> +                */
>> +               if (pci_controllers[i].root_bus == NULL)
>> +                       continue;
>> +
>> +               bar_start = pci_controllers[i].mem_resources[0].start;
>> +               bar_end = pci_controllers[i].mem_resources[0].end;
>> +
>> +               if ((start >= bar_start) && (end <= bar_end)) {
>> +
>> +                       controller = &pci_controllers[i];
>> +
>> +                       goto got_it;
>> +               }
>> +       }
>> +
>> +       if (controller == NULL)
>> +               return NULL;
>> +
>> +got_it:
>> +       trio_fd = controller->trio->fd;
>> +
>> +       offset = HV_TRIO_PIO_OFFSET(controller->pio_io_index) + port;
>> +
>> +       /*
>> +        * We need to keep the PCI bus address's in-page offset in the VA.
>> +        */
>> +       return iorpc_ioremap(trio_fd, offset, size) + (port & (PAGE_SIZE - 1));
>> +}
>> +EXPORT_SYMBOL(ioport_map);
>> +
>> +void ioport_unmap(void __iomem *addr)
>> +{
>> +       iounmap(addr);
>> +}
>> +EXPORT_SYMBOL(ioport_unmap);
>> +
>> +/*
>> + * Create a virtual mapping cookie for a PCI BAR (memory or IO).
>> + */
>> +void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max)
>> +{
>> +       resource_size_t start = pci_resource_start(dev, bar);
>> +       resource_size_t len = pci_resource_len(dev, bar);
>> +       unsigned long flags = pci_resource_flags(dev, bar);
>> +
>> +       if (!len)
>> +               return NULL;
>> +       if (max && len > max)
>> +               len = max;
>> +       if (flags & IORESOURCE_IO)
>> +               return ioport_map(start, len);
>> +       if (flags & IORESOURCE_MEM)
>> +               return ioremap(start, len);
>> +
>> +       pr_err("PCI: Trying to map invalid resource %#lx\n", flags);
>> +       return NULL;
>> +}
>> +EXPORT_SYMBOL(pci_iomap);
>> +
>>  void pci_iounmap(struct pci_dev *dev, void __iomem *addr)
>>  {
>>         iounmap(addr);
>> @@ -1478,32 +1702,55 @@
>>         trio_context = controller->trio;
>>
>>         /*
>> -        * Allocate the Mem-Map that will accept the MSI write and
>> -        * trigger the TILE-side interrupts.
>> -        */
>> -       mem_map = gxio_trio_alloc_memory_maps(trio_context, 1, 0, 0);
>> -       if (mem_map < 0) {
>> -               dev_printk(KERN_INFO, &pdev->dev,
>> -                       "%s Mem-Map alloc failure. "
>> -                       "Failed to initialize MSI interrupts. "
>> -                       "Falling back to legacy interrupts.\n",
>> -                       desc->msi_attrib.is_msix ? "MSI-X" : "MSI");
>> +        * Allocate a scatter-queue that will accept the MSI write and
>> +        * trigger the TILE-side interrupts. We use the scatter-queue regions
>> +        * before the mem map regions, because the latter are needed by more
>> +        * applications.
>> +        */
>> +       mem_map = gxio_trio_alloc_scatter_queues(trio_context, 1, 0, 0);
>> +       if (mem_map >= 0) {
>> +               TRIO_MAP_SQ_DOORBELL_FMT_t doorbell_template = {{
>> +                       .pop = 0,
>> +                       .doorbell = 1,
>> +               }};
>> +
>> +               mem_map += TRIO_NUM_MAP_MEM_REGIONS;
>> +               mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
>> +                       mem_map * MEM_MAP_INTR_REGION_SIZE;
>> +               mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
>> +
>> +               msi_addr = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 8;
>> +               msg.data = (unsigned int)doorbell_template.word;
>> +       } else {
>> +               /* SQ regions are out, allocate from map mem regions. */
>> +               mem_map = gxio_trio_alloc_memory_maps(trio_context, 1, 0, 0);
>> +               if (mem_map < 0) {
>> +                       dev_printk(KERN_INFO, &pdev->dev,
>> +                               "%s Mem-Map alloc failure. "
>> +                               "Failed to initialize MSI interrupts. "
>> +                               "Falling back to legacy interrupts.\n",
>> +                               desc->msi_attrib.is_msix ? "MSI-X" : "MSI");
>> +                       ret = -ENOMEM;
>> +                       goto msi_mem_map_alloc_failure;
>> +               }
>>
>> -               ret = -ENOMEM;
>> -               goto msi_mem_map_alloc_failure;
>> +               mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
>> +                       mem_map * MEM_MAP_INTR_REGION_SIZE;
>> +               mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
>> +
>> +               msi_addr = mem_map_base + TRIO_MAP_MEM_REG_INT3 -
>> +                       TRIO_MAP_MEM_REG_INT0;
>> +
>> +               msg.data = mem_map;
>>         }
>>
>>         /* We try to distribute different IRQs to different tiles. */
>>         cpu = tile_irq_cpu(irq);
>>
>>         /*
>> -        * Now call up to the HV to configure the Mem-Map interrupt and
>> +        * Now call up to the HV to configure the MSI interrupt and
>>          * set up the IPI binding.
>>          */
>> -       mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
>> -               mem_map * MEM_MAP_INTR_REGION_SIZE;
>> -       mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
>> -
>>         ret = gxio_trio_config_msi_intr(trio_context, cpu_x(cpu), cpu_y(cpu),
>>                                         KERNEL_PL, irq, controller->mac,
>>                                         mem_map, mem_map_base, mem_map_limit,
>> @@ -1516,13 +1763,9 @@
>>
>>         irq_set_msi_desc(irq, desc);
>>
>> -       msi_addr = mem_map_base + TRIO_MAP_MEM_REG_INT3 - TRIO_MAP_MEM_REG_INT0;
>> -
>>         msg.address_hi = msi_addr >> 32;
>>         msg.address_lo = msi_addr & 0xffffffff;
>>
>> -       msg.data = mem_map;
>> -
>>         write_msi_msg(irq, &msg);
>>         irq_set_chip_and_handler(irq, &tilegx_msi_chip, handle_level_irq);
>>         irq_set_handler_data(irq, controller);
>>
>>
>> What we got after my fix:
>>
>> pci 0000:00:00.0: BAR 8: assigned [mem 0x100c0000000-0x100c00fffff]
>> pci 0000:00:00.0: BAR 9: assigned [mem 0x100c0100000-0x100c01fffff pref]
>> pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> pci 0000:01:00.0: BAR 6: assigned [mem 0x100c0100000-0x100c013ffff pref]
>> pci 0000:01:00.0: BAR 6: set to [mem 0x100c0100000-0x100c013ffff pref]
>> (PCI address [0xc0100000-0xc013ffff])
>> pci 0000:01:00.0: BAR 4: assigned [mem 0x100c0000000-0x100c000ffff
>> 64bit]
>> pci 0000:01:00.0: BAR 4: set to [mem 0x100c0000000-0x100c000ffff
>> 64bit] (PCI address [0xc0000000-0xc000ffff])
>> pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x107f]
>> pci 0000:01:00.0: BAR 2: set to [io  0x1000-0x107f] (PCI address
>> [0x1000-0x107f])
>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>> pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> pci 0000:00:00.0:   bridge window [mem 0x100c0000000-0x100c00fffff]
>> pci 0000:00:00.0:   bridge window [mem 0x100c0100000-0x100c01fffff
>> pref]
>> pci 0001:00:00.0: BAR 8: assigned [mem 0x101c0000000-0x101c00fffff]
>> pci 0001:00:00.0: BAR 9: assigned [mem 0x101c0100000-0x101c01fffff
>> pref]
>> pci 0001:00:00.0: BAR 7: assigned [io  0x80001000-0x80001fff]
>> pci 0001:01:00.0: BAR 6: assigned [mem 0x101c0100000-0x101c013ffff
>> pref]
>> pci 0001:01:00.0: BAR 6: set to [mem 0x101c0100000-0x101c013ffff pref]
>> (PCI address [0xc0100000-0xc013ffff])
>> pci 0001:01:00.0: BAR 4: assigned [mem 0x101c0000000-0x101c000ffff
>> 64bit]
>> pci 0001:01:00.0: BAR 4: set to [mem 0x101c0000000-0x101c000ffff
>> 64bit] (PCI address [0xc0000000-0xc000ffff])
>> pci 0001:01:00.0: BAR 2: assigned [io  0x80001000-0x8000107f]
>> pci 0001:01:00.0: BAR 2: set to [io  0x80001000-0x8000107f] (PCI
>> address [0x80001000-0x8000107f])
>> pci 0001:00:00.0: PCI bridge to [bus 01-01]
>> pci 0001:00:00.0:   bridge window [io  0x80001000-0x80001fff]
>> pci 0001:00:00.0:   bridge window [mem 0x101c0000000-0x101c00fffff]
>> pci 0001:00:00.0:   bridge window [mem 0x101c0100000-0x101c01fffff
>> pref]
>> pci 0000:00:00.0: enabling device (0006 -> 0007)
>> pci 0001:00:00.0: enabling device (0006 -> 0007)
>> pci_bus 0000:00: resource 0 [io  0x1000-0x800007ff]
>> pci_bus 0000:00: resource 1 [mem 0x100c0000000-0x100ffffffff]
>> pci_bus 0000:01: resource 0 [io  0x1000-0x1fff]
>> pci_bus 0000:01: resource 1 [mem 0x100c0000000-0x100c00fffff]
>> pci_bus 0000:01: resource 2 [mem 0x100c0100000-0x100c01fffff pref]
>> pci_bus 0001:00: resource 0 [io  0x80000800-0xffffffff]
>> pci_bus 0001:00: resource 1 [mem 0x101c0000000-0x101ffffffff]
>> pci_bus 0001:01: resource 0 [io  0x80001000-0x80001fff]
>> pci_bus 0001:01: resource 1 [mem 0x101c0000000-0x101c00fffff]
>> pci_bus 0001:01: resource 2 [mem 0x101c0100000-0x101c01fffff pref]
>> ......
>> mvsas 0000:01:00.0: mvsas: driver version 0.8.2
>> mvsas 0000:01:00.0: enabling device (0000 -> 0003)
>> mvsas 0000:01:00.0: enabling bus mastering
>> mvsas 0000:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
>> scsi0 : mvsas
>> ......
>> mvsas 0001:01:00.0: mvsas: driver version 0.8.2
>> mvsas 0001:01:00.0: enabling device (0000 -> 0003)
>> mvsas 0001:01:00.0: enabling bus mastering
>> mvsas 0001:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
>> scsi1 : mvsas
>>
>>
>> It works now. But I really need some one to confirm whether my
>> modification is enough or not,
>> if there have other potential problems.
>>
>>
>>
>> Best regards.
>>
>> --
>> Cyberman Wu
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Cyberman Wu
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux