Re: PCIe IO space support on Tilera GX: Is there any one who can confirm my modification to fix it is OK?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



[+cc Chris, also a few comments below]

On Fri, Oct 26, 2012 at 12:59 AM, Cyberman Wu <cypher.w@xxxxxxxxx> wrote:
> After we upgrade to MDE 4.1.0 from Tilera, we encounter a problem that
> only on HighPoint 2680 card works, I've
> tried to fix it, but since most time I'm working in user space, I'm
> not sure my fix is enough. Their FAE said that
> the guy who add PCIe I/O space support is on vacation and I can't get
> help from him now, I hope maybe there
> will have somebody can help.
>
>
> Problem we encountered:
>
> pci 0000:00:00.0: BAR 8: assigned [mem 0x100c0000000-0x100c00fffff]
> pci 0000:00:00.0: BAR 9: assigned [mem 0x100c0100000-0x100c01fffff pref]
> pci 0000:00:00.0: BAR 7: assigned [io  0x0000-0x0fff]
> pci 0000:01:00.0: BAR 6: assigned [mem 0x100c0100000-0x100c013ffff pref]
> pci 0000:01:00.0: BAR 6: set to [mem 0x100c0100000-0x100c013ffff pref]
> (PCI address [0xc0100000-0xc013ffff])
> pci 0000:01:00.0: BAR 4: assigned [mem 0x100c0000000-0x100c000ffff 64bit]
> pci 0000:01:00.0: BAR 4: set to [mem 0x100c0000000-0x100c000ffff
> 64bit] (PCI address [0xc0000000-0xc000ffff])
> pci 0000:01:00.0: BAR 2: assigned [io  0x0000-0x007f]
> pci 0000:01:00.0: BAR 2: set to [io  0x0000-0x007f] (PCI address [0x0-0x7f])
> pci 0000:00:00.0: PCI bridge to [bus 01-01]
> pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
> pci 0000:00:00.0:   bridge window [mem 0x100c0000000-0x100c00fffff]
> pci 0000:00:00.0:   bridge window [mem 0x100c0100000-0x100c01fffff pref]
> pci 0001:00:00.0: BAR 8: assigned [mem 0x101c0000000-0x101c00fffff]
> pci 0001:00:00.0: BAR 9: assigned [mem 0x101c0100000-0x101c01fffff pref]
> pci 0001:00:00.0: BAR 7: assigned [io  0x0000-0x0fff]
> pci 0001:01:00.0: BAR 6: assigned [mem 0x101c0100000-0x101c013ffff pref]
> pci 0001:01:00.0: BAR 6: set to [mem 0x101c0100000-0x101c013ffff pref]
> (PCI address [0xc0100000-0xc013ffff])
> pci 0001:01:00.0: BAR 4: assigned [mem 0x101c0000000-0x101c000ffff 64bit]
> pci 0001:01:00.0: BAR 4: set to [mem 0x101c0000000-0x101c000ffff
> 64bit] (PCI address [0xc0000000-0xc000ffff])
> pci 0001:01:00.0: BAR 2: assigned [io  0x0000-0x007f]
> pci 0001:01:00.0: BAR 2: set to [io  0x0000-0x007f] (PCI address [0x0-0x7f])
> pci 0001:00:00.0: PCI bridge to [bus 01-01]
> pci 0001:00:00.0:   bridge window [io  0x0000-0x0fff]
> pci 0001:00:00.0:   bridge window [mem 0x101c0000000-0x101c00fffff]
> pci 0001:00:00.0:   bridge window [mem 0x101c0100000-0x101c01fffff pref]
> pci 0000:00:00.0: enabling device (0006 -> 0007)
> pci 0001:00:00.0: enabling device (0006 -> 0007)
> pci_bus 0000:00: resource 0 [io  0x0000-0xffffffff]
> pci_bus 0000:00: resource 1 [mem 0x100c0000000-0x100ffffffff]
> pci_bus 0000:01: resource 0 [io  0x0000-0x0fff]
> pci_bus 0000:01: resource 1 [mem 0x100c0000000-0x100c00fffff]
> pci_bus 0000:01: resource 2 [mem 0x100c0100000-0x100c01fffff pref]
> pci_bus 0001:00: resource 0 [io  0x0000-0xffffffff]
> pci_bus 0001:00: resource 1 [mem 0x101c0000000-0x101ffffffff]
> pci_bus 0001:01: resource 0 [io  0x0000-0x0fff]
> pci_bus 0001:01: resource 1 [mem 0x101c0000000-0x101c00fffff]
> pci_bus 0001:01: resource 2 [mem 0x101c0100000-0x101c01fffff pref]
> ......
> mvsas 0000:01:00.0: mvsas: driver version 0.8.2
> mvsas 0000:01:00.0: enabling device (0000 -> 0003)
> mvsas 0000:01:00.0: enabling bus mastering
> mvsas 0000:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
> mvsas 0000:01:00.0: Phy3 : No sig fis
> scsi0 : mvsas
> ......
> mvsas 0001:01:00.0: mvsas: driver version 0.8.2
> mvsas 0001:01:00.0: enabling device (0000 -> 0003)
> mvsas 0001:01:00.0: enabling bus mastering
> mvsas 0001:01:00.0: BAR 2: can't reserve [io  0x0000-0x007f]
> mvsas: probe of 0001:01:00.0 failed with error -16
>
>
> My modification:
>
> --- /opt/tilera/TileraMDE-4.1.0.148119/tilegx/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c     2012-10-22
> 14:56:59.783096378 +0800
> +++ Tilera_src/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c    2012-10-26
> 13:55:02.731947886 +0800
> @@ -368,6 +368,10 @@
>         int num_trio_shims = 0;
>         int ctl_index = 0;
>         int i, j;
> +    // Modified by Cyberman Wu on Oct 25th, 2012.
> +       resource_size_t io_mem_start;
> +       resource_size_t io_mem_end;
> +       resource_size_t io_mem_size;
>
>         if (!pci_probe) {
>                 pr_info("PCI: disabled by boot argument\n");
> @@ -457,6 +461,18 @@
>         }
>
>  out:
> +       // Use IO memory space 0~0xffffffff for every controller will
> +       // cause device on controller other than the first failed to
> +       // load driver if it using IO regions.
> +       // Is reserve the first 4K IO address space OK? Tilera use
> +       // IO space address begin from 0, but some drivers in Linux
> +       // recognize 0 address a error, say, mvsas, so for compatiblity
> +       // reserve some address from 0 should be better?

It's not that mvsas thinks I/O address 0 is invalid, it's just that we
already assigned [io 0x0000-0x007f] to the device at 0000:01:00.0:

  pci 0000:01:00.0: BAR 2: set to [io  0x0000-0x007f]

so that range can't also be assigned to 0001:01:00.0.

> +       // Modified by Cyberman Wu on Oct 25th, 2012.
> +       io_mem_start = 4096;
> +       io_mem_end = (resource_size_t)IO_SPACE_LIMIT + 1;
> +       io_mem_size = (io_mem_end - io_mem_start) / num_rc_controllers;
> +       io_mem_size &= ~3;
>         /*
>          * Configure each PCIe RC port.
>          */
> @@ -470,8 +486,9 @@
>                 controller->index = i;
>                 controller->ops = &tile_cfg_ops;
>
> -               controller->io_space.start = 0;
> -               controller->io_space.end = IO_SPACE_LIMIT;
> +               // Modified by Cyberman Wu on Oct 25th, 2012.
> +               controller->io_space.start = io_mem_start + (i * io_mem_size);
> +               controller->io_space.end = controller->io_space.start + io_mem_size - 1;
>                 controller->io_space.flags = IORESOURCE_IO;
>                 snprintf(controller->io_space_name,
>                          sizeof(controller->io_space_name),
>
>
> Please note that we're using MDE-4.1.0, which use kernel 3.0.38, patch
> it and reversion it
> to 2.6.40.38.
> I've checked source code under arch/tile of kernel 3.6.3 and PCIe I/O
> space support is still
> not here. Below is diff of arch/tile/pci_gx.c between kernel 3.6.3 and
> MDE-4.1.0:

Per http://lkml.indiana.edu/hypermail/linux/kernel/1205.1/01176.html,
Chris considered adding I/O space support and decided against it at
that time, partly because it would use up a TRIO PIO region.

I don't know his current thoughts.  Possibly it could be done under a
config option or something.

But of course, you'd have to do it by adding I/O space support to the
current 3.6 kernel *without* reverting all the other changes that have
been made since 2.6.40.

> --- .cache/.fr-9Oo37J/linux-3.6.3/arch/tile/kernel/pci_gx.c     2012-10-22
> 00:32:56.000000000 +0800
> +++ /opt/tilera/TileraMDE-4.1.0.148119/tilegx/src/linux-2.6.40.38/arch/tile/kernel/pci_gx.c     2012-10-22
> 14:56:59.783096378 +0800
> @@ -69,19 +69,18 @@
>   * a HW PCIe link-training bug. The exact delay is specified with
>   * a kernel boot argument in the form of "pcie_rc_delay=T,P,S",
>   * where T is the TRIO instance number, P is the port number and S is
> - * the delay in seconds. If the delay is not provided, the value
> - * will be DEFAULT_RC_DELAY.
> + * the delay in seconds. If the argument is specified, but the delay is
> + * not provided, the value will be DEFAULT_RC_DELAY.
>   */
>  static int __devinitdata rc_delay[TILEGX_NUM_TRIO][TILEGX_TRIO_PCIES];
>
>  /* Default number of seconds that the PCIe RC port probe can be delayed. */
>  #define DEFAULT_RC_DELAY       10
>
> -/* Max number of seconds that the PCIe RC port probe can be delayed. */
> -#define MAX_RC_DELAY           20
> -
> +#if !defined(GX_FPGA)
>  /* Array of the PCIe ports configuration info obtained from the BIB. */
>  struct pcie_port_property pcie_ports[TILEGX_NUM_TRIO][TILEGX_TRIO_PCIES];
> +#endif
>
>  /* All drivers share the TRIO contexts defined here. */
>  gxio_trio_context_t trio_contexts[TILEGX_NUM_TRIO];
> @@ -97,6 +96,41 @@
>  static struct cpumask intr_cpus_map;
>
>  /*
> + * Convert a resource to a PCI device bus address or bus window.
> + */
> +void __devinit
> +pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
> +                       struct resource *res)
> +{
> +       struct pci_controller *controller =
> +               (struct pci_controller *)dev->sysdata;
> +       unsigned long offset = 0;
> +
> +       if (res->flags & IORESOURCE_MEM)
> +               offset = controller->mem_offset;
> +
> +       region->start = res->start - offset;
> +       region->end = res->end - offset;
> +}
> +EXPORT_SYMBOL(pcibios_resource_to_bus);
> +
> +void __devinit
> +pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
> +                       struct pci_bus_region *region)
> +{
> +       struct pci_controller *controller =
> +               (struct pci_controller *)dev->sysdata;
> +       unsigned long offset = 0;
> +
> +       if (res->flags & IORESOURCE_MEM)
> +               offset = controller->mem_offset;
> +
> +       res->start = region->start + offset;
> +       res->end = region->end + offset;
> +}
> +EXPORT_SYMBOL(pcibios_bus_to_resource);
> +
> +/*
>   * We don't need to worry about the alignment of resources.
>   */
>  resource_size_t pcibios_align_resource(void *data, const struct resource *res,
> @@ -274,6 +308,10 @@
>
>         cpumask_copy(&intr_cpus_map, cpu_online_mask);
>
> +#ifdef CONFIG_DATAPLANE
> +       /* Remove dataplane cpus. */
> +       cpumask_andnot(&intr_cpus_map, &intr_cpus_map, &dataplane_map);
> +#endif
>
>         for (i = 0; i < 4; i++) {
>                 gxio_trio_context_t *context = controller->trio;
> @@ -325,7 +363,7 @@
>   *
>   * Returns the number of controllers discovered.
>   */
> -int __init tile_pci_init(void)
> +int __devinit tile_pci_init(void)
>  {
>         int num_trio_shims = 0;
>         int ctl_index = 0;
> @@ -359,6 +397,7 @@
>          * We look at the Board Information Block first and then see if there
>          * are any overriding configuration by the HW strapping pin.
>          */
> +#if !defined(GX_FPGA)
>         for (i = 0; i < TILEGX_NUM_TRIO; i++) {
>                 gxio_trio_context_t *context = &trio_contexts[i];
>                 int ret;
> @@ -386,6 +425,13 @@
>                         }
>                 }
>         }
> +#else
> +       /*
> +        * For now, just assume that there is a single RC port on trio/0.
> +        */
> +       num_rc_controllers = 1;
> +       pcie_rc[0][2] = 1;
> +#endif
>
>         /*
>          * Return if no PCIe ports are configured to operate in RC mode.
> @@ -424,13 +470,20 @@
>                 controller->index = i;
>                 controller->ops = &tile_cfg_ops;
>
> +               controller->io_space.start = 0;
> +               controller->io_space.end = IO_SPACE_LIMIT;
> +               controller->io_space.flags = IORESOURCE_IO;
> +               snprintf(controller->io_space_name,
> +                        sizeof(controller->io_space_name),
> +                        "PCI I/O domain %d", i);
> +               controller->io_space.name = controller->io_space_name;
> +
>                 /*
>                  * The PCI memory resource is located above the PA space.
>                  * For every host bridge, the BAR window or the MMIO aperture
>                  * is in range [3GB, 4GB - 1] of a 4GB space beyond the
>                  * PA space.
>                  */
> -
>                 controller->mem_offset = TILE_PCI_MEM_START +
>                         (i * TILE_PCI_BAR_WINDOW_TOP);
>                 controller->mem_space.start = controller->mem_offset +
> @@ -451,7 +504,7 @@
>   * (pin - 1) converts from the PCI standard's [1:4] convention to
>   * a normal [0:3] range.
>   */
> -static int tile_map_irq(const struct pci_dev *dev, u8 device, u8 pin)
> +static int tile_map_irq(struct pci_dev *dev, u8 device, u8 pin)
>  {
>         struct pci_controller *controller =
>                 (struct pci_controller *)dev->sysdata;
> @@ -463,11 +516,12 @@
>                                                 controller)
>  {
>         gxio_trio_context_t *trio_context = controller->trio;
> -       struct pci_bus *root_bus = controller->root_bus;
>         TRIO_PCIE_RC_DEVICE_CONTROL_t dev_control;
>         TRIO_PCIE_RC_DEVICE_CAP_t rc_dev_cap;
> +       unsigned int smallest_max_payload;
> +       struct pci_dev *dev = NULL;
>         unsigned int reg_offset;
> -       struct pci_bus *child;
> +       u16 new_values;
>         int mac;
>         int err;
>
> @@ -508,33 +562,59 @@
>         __gxio_mmio_write32(trio_context->mmio_base_mac + reg_offset,
>                                                 rc_dev_cap.word);
>
> -       /* Configure PCI Express MPS setting. */
> -       list_for_each_entry(child, &root_bus->children, node) {
> -               struct pci_dev *self = child->self;
> -               if (!self)
> +       smallest_max_payload = rc_dev_cap.mps_sup;
> +
> +       /* Scan for the smallest maximum payload size. */
> +       while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
> +               int pcie_caps_offset;
> +               u32 devcap;
> +               int max_payload;
> +
> +               /* Skip device that is not in this PCIe domain. */
> +               if ((struct pci_controller *)dev->sysdata != controller)
>                         continue;
>
> -               pcie_bus_configure_settings(child, self->pcie_mpss);
> +               pcie_caps_offset = pci_find_capability(dev, PCI_CAP_ID_EXP);
> +               if (pcie_caps_offset == 0)
> +                       continue;
> +
> +               pci_read_config_dword(dev, pcie_caps_offset + PCI_EXP_DEVCAP,
> +                                     &devcap);
> +               max_payload = devcap & PCI_EXP_DEVCAP_PAYLOAD;
> +               if (max_payload < smallest_max_payload)
> +                       smallest_max_payload = max_payload;
> +       }
> +
> +       /* Now, set the max_payload_size for all devices to that value. */
> +       new_values = smallest_max_payload << 5;
> +       while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
> +               int pcie_caps_offset;
> +               u16 devctl;
> +
> +               /* Skip device that is not in this PCIe domain. */
> +               if ((struct pci_controller *)dev->sysdata != controller)
> +                       continue;
> +
> +               pcie_caps_offset = pci_find_capability(dev, PCI_CAP_ID_EXP);
> +               if (pcie_caps_offset == 0)
> +                       continue;
> +
> +               pci_read_config_word(dev, pcie_caps_offset + PCI_EXP_DEVCTL,
> +                                    &devctl);
> +               devctl &= ~PCI_EXP_DEVCTL_PAYLOAD;
> +               devctl |= new_values;
> +               pci_write_config_word(dev, pcie_caps_offset + PCI_EXP_DEVCTL,
> +                                     devctl);
>         }
>
>         /*
>          * Set the mac_config register in trio based on the MPS/MRS of the link.
>          */
> -       reg_offset =
> -               (TRIO_PCIE_RC_DEVICE_CONTROL <<
> -                       TRIO_CFG_REGION_ADDR__REG_SHIFT) |
> -               (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_STANDARD <<
> -                       TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
> -               (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
> -
> -       dev_control.word = __gxio_mmio_read32(trio_context->mmio_base_mac +
> -                                               reg_offset);
> -
>         err = gxio_trio_set_mps_mrs(trio_context,
> -                                   dev_control.max_payload_size,
> +                                   smallest_max_payload,
>                                     dev_control.max_read_req_sz,
>                                     mac);
> -        if (err < 0) {
> +       if (err < 0) {
>                 pr_err("PCI: PCIE_CONFIGURE_MAC_MPS_MRS failure, "
>                         "MAC %d on TRIO %d\n",
>                         mac, controller->trio_index);
> @@ -571,14 +651,9 @@
>                 if (!isdigit(*str))
>                         return -EINVAL;
>                 delay = simple_strtoul(str, (char **)&str, 10);
> -               if (delay > MAX_RC_DELAY)
> -                       return -EINVAL;
>         }
>
>         rc_delay[trio_index][mac] = delay ? : DEFAULT_RC_DELAY;
> -       pr_info("Delaying PCIe RC link training for %u sec"
> -               " on MAC %lu on TRIO %lu\n", rc_delay[trio_index][mac],
> -               mac, trio_index);
>         return 0;
>  }
>  early_param("pcie_rc_delay", setup_pcie_rc_delay);
> @@ -586,18 +661,14 @@
>  /*
>   * PCI initialization entry point, called by subsys_initcall.
>   */
> -int __init pcibios_init(void)
> +int __devinit pcibios_init(void)
>  {
>         resource_size_t offset;
> -       LIST_HEAD(resources);
>         int next_busno;
>         int i;
>
>         tile_pci_init();
>
> -       if (num_rc_controllers == 0 && num_ep_controllers == 0)
> -               return 0;
> -
>         /*
>          * We loop over all the TRIO shims and set up the MMIO mappings.
>          */
> @@ -623,6 +694,9 @@
>                 }
>         }
>
> +       if (num_rc_controllers == 0 && num_ep_controllers == 0)
> +               return 0;
> +
>         /*
>          * Delay a bit in case devices aren't ready.  Some devices are
>          * known to require at least 20ms here, but we use a more
> @@ -684,15 +758,36 @@
>                 }
>
>                 /*
> -                * Delay the RC link training if needed.
> +                * Delay the bus probe if needed.
>                  */
> -               if (rc_delay[trio_index][mac])
> +               if (rc_delay[trio_index][mac]) {
> +                       pr_info("Delaying PCIe RC link training for %d sec"
> +                               " on MAC %d on TRIO %d\n",
> +                               rc_delay[trio_index][mac], mac,
> +                               trio_index);
>                         msleep(rc_delay[trio_index][mac] * 1000);
> +               }
>
> -               ret = gxio_trio_force_rc_link_up(trio_context, mac);
> -               if (ret < 0)
> -                       pr_err("PCI: PCIE_FORCE_LINK_UP failure, "
> -                               "MAC %d on TRIO %d\n", mac, trio_index);
> +               /*
> +                * Check for PCIe link-up status to decide if we need
> +                * to force the link to come up.
> +                */
> +               reg_offset =
> +                       (TRIO_PCIE_INTFC_PORT_STATUS <<
> +                               TRIO_CFG_REGION_ADDR__REG_SHIFT) |
> +                       (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_INTERFACE <<
> +                               TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
> +                       (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
> +
> +               port_status.word =
> +                       __gxio_mmio_read(trio_context->mmio_base_mac +
> +                                        reg_offset);
> +               if (!port_status.dl_up) {
> +                       ret = gxio_trio_force_rc_link_up(trio_context, mac);
> +                       if (ret < 0)
> +                               pr_err("PCI: PCIE_FORCE_LINK_UP failure, "
> +                                       "MAC %d on TRIO %d\n", mac, trio_index);
> +               }
>
>                 pr_info("PCI: Found PCI controller #%d on TRIO %d MAC %d\n", i,
>                         trio_index, controller->mac);
> @@ -704,22 +799,20 @@
>                 msleep(1000);
>
>                 /*
> -                * Check for PCIe link-up status.
> +                * Check for PCIe link-up status again.
>                  */
> -
> -               reg_offset =
> -                       (TRIO_PCIE_INTFC_PORT_STATUS <<
> -                               TRIO_CFG_REGION_ADDR__REG_SHIFT) |
> -                       (TRIO_CFG_REGION_ADDR__INTFC_VAL_MAC_INTERFACE <<
> -                               TRIO_CFG_REGION_ADDR__INTFC_SHIFT ) |
> -                       (mac << TRIO_CFG_REGION_ADDR__MAC_SEL_SHIFT);
> -
>                 port_status.word =
>                         __gxio_mmio_read(trio_context->mmio_base_mac +
>                                          reg_offset);
>                 if (!port_status.dl_up) {
> -                       pr_err("PCI: link is down, MAC %d on TRIO %d\n",
> -                               mac, trio_index);
> +                       if (pcie_ports[trio_index][mac].removable) {
> +                               pr_info("PCI: link is down, MAC %d on TRIO %d",
> +                                       mac, trio_index);
> +                               pr_info("This is expected if no PCIe card"
> +                                       " is connected to this link");
> +                       } else
> +                               pr_err("PCI: link is down, MAC %d on TRIO %d",
> +                                       mac, trio_index);
>                         continue;
>                 }
>
> @@ -842,19 +935,22 @@
>                 }
>
>                 /*
> -                * The PCI memory resource is located above the PA space.
> -                * The memory range for the PCI root bus should not overlap
> -                * with the physical RAM
> +                * This comes from the generic Linux PCI driver.
> +                *
> +                * It reads the PCI tree for this bus into the Linux
> +                * data structures.
> +                *
> +                * This is inlined in linux/pci.h and calls into
> +                * pci_scan_bus_parented() in probe.c.
>                  */
> -               pci_add_resource_offset(&resources, &controller->mem_space,
> -                                       controller->mem_offset);
> -
> -               controller->first_busno = next_busno;
> -               bus = pci_scan_root_bus(NULL, next_busno, controller->ops,
> -                                       controller, &resources);
> +               controller->first_busno= next_busno;
> +               bus = pci_scan_bus(next_busno, controller->ops, controller);
>                 controller->root_bus = bus;
> -               next_busno = bus->busn_res.end + 1;
> -
> +#if 0
> +               next_busno = bus->subordinate + 1;
> +#else
> +               next_busno = 0;
> +#endif
>         }
>
>         /* Do machine dependent PCI interrupt routing */
> @@ -951,6 +1047,37 @@
>                 }
>
>                 /*
> +                * Alloc a PIO region for PCI I/O space access for each RC port.
> +                */
> +               ret = gxio_trio_alloc_pio_regions(trio_context, 1, 0, 0);
> +               if (ret < 0) {
> +                       pr_err("PCI: I/O PIO alloc failure on TRIO %d mac %d, "
> +                               "give up\n", controller->trio_index,
> +                               controller->mac);
> +
> +                       continue;
> +               }
> +
> +               controller->pio_io_index = ret;
> +
> +               /*
> +                * For PIO IO, the bus_address_hi parameter is hard-coded 0
> +                * because PCI I/O address space is 32-bit.
> +                */
> +               ret = gxio_trio_init_pio_region_aux(trio_context,
> +                                                   controller->pio_io_index,
> +                                                   controller->mac,
> +                                                   0,
> +                                                   HV_TRIO_PIO_FLAG_IO_SPACE);
> +               if (ret < 0) {
> +                       pr_err("PCI: I/O PIO init failure on TRIO %d mac %d, "
> +                               "give up\n", controller->trio_index,
> +                               controller->mac);
> +
> +                       continue;
> +               }
> +
> +               /*
>                  * Configure a Mem-Map region for each memory controller so
>                  * that Linux can map all of its PA space to the PCI bus.
>                  * Use the IOMMU to handle hash-for-home memory.
> @@ -1015,9 +1142,22 @@
>  }
>  subsys_initcall(pcibios_init);
>
> -/* Note: to be deleted after Linux 3.6 merge. */
> +/*
> + * PCI scan code calls the arch specific pcibios_fixup_bus() each time it scans
> + * a new bridge. Called after each bus is probed, but before its children are
> + * examined.
> + */
>  void __devinit pcibios_fixup_bus(struct pci_bus *bus)
>  {
> +       struct pci_dev *dev = bus->self;
> +
> +       if (!dev) {
> +               struct pci_controller *controller = bus->sysdata;
> +
> +               /* This is the root bus. */
> +               bus->resource[0] = &controller->io_space;
> +               bus->resource[1] = &controller->mem_space;
> +       }
>  }
>
>  /*
> @@ -1043,8 +1183,7 @@
>
>  /*
>   * Enable memory address decoding, as appropriate, for the
> - * device described by the 'dev' struct. The I/O decoding
> - * is disabled, though the TILE-Gx supports I/O addressing.
> + * device described by the 'dev' struct.
>   *
>   * This is called from the generic PCI layer, and can be called
>   * for bridges or endpoints.
> @@ -1126,10 +1265,95 @@
>          * We need to keep the PCI bus address's in-page offset in the VA.
>          */
>         return iorpc_ioremap(trio_fd, offset, size) +
> -               (phys_addr & (PAGE_SIZE - 1));
> +               (start & (PAGE_SIZE - 1));
>  }
>  EXPORT_SYMBOL(ioremap);
>
> +/* Map a PCI I/O address into VA space. */
> +void __iomem *ioport_map(unsigned long port, unsigned int size)
> +{
> +       struct pci_controller *controller = NULL;
> +       resource_size_t bar_start;
> +       resource_size_t bar_end;
> +       resource_size_t offset;
> +       resource_size_t start;
> +       resource_size_t end;
> +       int trio_fd;
> +       int i;
> +
> +       start = port;
> +       end = port + size - 1;
> +
> +       /*
> +        * In the following, each PCI controller's mem_resources[0]
> +        * represents its PCI I/O resource. By searching port in each
> +        * controller's mem_resources[0], we can determine the controller
> +        * that should accept the PCI I/O access.
> +        */
> +
> +       for (i = 0; i < num_rc_controllers; i++) {
> +               /*
> +                * Skip controllers that are not properly initialized or
> +                * have down links.
> +                */
> +               if (pci_controllers[i].root_bus == NULL)
> +                       continue;
> +
> +               bar_start = pci_controllers[i].mem_resources[0].start;
> +               bar_end = pci_controllers[i].mem_resources[0].end;
> +
> +               if ((start >= bar_start) && (end <= bar_end)) {
> +
> +                       controller = &pci_controllers[i];
> +
> +                       goto got_it;
> +               }
> +       }
> +
> +       if (controller == NULL)
> +               return NULL;
> +
> +got_it:
> +       trio_fd = controller->trio->fd;
> +
> +       offset = HV_TRIO_PIO_OFFSET(controller->pio_io_index) + port;
> +
> +       /*
> +        * We need to keep the PCI bus address's in-page offset in the VA.
> +        */
> +       return iorpc_ioremap(trio_fd, offset, size) + (port & (PAGE_SIZE - 1));
> +}
> +EXPORT_SYMBOL(ioport_map);
> +
> +void ioport_unmap(void __iomem *addr)
> +{
> +       iounmap(addr);
> +}
> +EXPORT_SYMBOL(ioport_unmap);
> +
> +/*
> + * Create a virtual mapping cookie for a PCI BAR (memory or IO).
> + */
> +void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max)
> +{
> +       resource_size_t start = pci_resource_start(dev, bar);
> +       resource_size_t len = pci_resource_len(dev, bar);
> +       unsigned long flags = pci_resource_flags(dev, bar);
> +
> +       if (!len)
> +               return NULL;
> +       if (max && len > max)
> +               len = max;
> +       if (flags & IORESOURCE_IO)
> +               return ioport_map(start, len);
> +       if (flags & IORESOURCE_MEM)
> +               return ioremap(start, len);
> +
> +       pr_err("PCI: Trying to map invalid resource %#lx\n", flags);
> +       return NULL;
> +}
> +EXPORT_SYMBOL(pci_iomap);
> +
>  void pci_iounmap(struct pci_dev *dev, void __iomem *addr)
>  {
>         iounmap(addr);
> @@ -1478,32 +1702,55 @@
>         trio_context = controller->trio;
>
>         /*
> -        * Allocate the Mem-Map that will accept the MSI write and
> -        * trigger the TILE-side interrupts.
> -        */
> -       mem_map = gxio_trio_alloc_memory_maps(trio_context, 1, 0, 0);
> -       if (mem_map < 0) {
> -               dev_printk(KERN_INFO, &pdev->dev,
> -                       "%s Mem-Map alloc failure. "
> -                       "Failed to initialize MSI interrupts. "
> -                       "Falling back to legacy interrupts.\n",
> -                       desc->msi_attrib.is_msix ? "MSI-X" : "MSI");
> +        * Allocate a scatter-queue that will accept the MSI write and
> +        * trigger the TILE-side interrupts. We use the scatter-queue regions
> +        * before the mem map regions, because the latter are needed by more
> +        * applications.
> +        */
> +       mem_map = gxio_trio_alloc_scatter_queues(trio_context, 1, 0, 0);
> +       if (mem_map >= 0) {
> +               TRIO_MAP_SQ_DOORBELL_FMT_t doorbell_template = {{
> +                       .pop = 0,
> +                       .doorbell = 1,
> +               }};
> +
> +               mem_map += TRIO_NUM_MAP_MEM_REGIONS;
> +               mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
> +                       mem_map * MEM_MAP_INTR_REGION_SIZE;
> +               mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
> +
> +               msi_addr = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 8;
> +               msg.data = (unsigned int)doorbell_template.word;
> +       } else {
> +               /* SQ regions are out, allocate from map mem regions. */
> +               mem_map = gxio_trio_alloc_memory_maps(trio_context, 1, 0, 0);
> +               if (mem_map < 0) {
> +                       dev_printk(KERN_INFO, &pdev->dev,
> +                               "%s Mem-Map alloc failure. "
> +                               "Failed to initialize MSI interrupts. "
> +                               "Falling back to legacy interrupts.\n",
> +                               desc->msi_attrib.is_msix ? "MSI-X" : "MSI");
> +                       ret = -ENOMEM;
> +                       goto msi_mem_map_alloc_failure;
> +               }
>
> -               ret = -ENOMEM;
> -               goto msi_mem_map_alloc_failure;
> +               mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
> +                       mem_map * MEM_MAP_INTR_REGION_SIZE;
> +               mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
> +
> +               msi_addr = mem_map_base + TRIO_MAP_MEM_REG_INT3 -
> +                       TRIO_MAP_MEM_REG_INT0;
> +
> +               msg.data = mem_map;
>         }
>
>         /* We try to distribute different IRQs to different tiles. */
>         cpu = tile_irq_cpu(irq);
>
>         /*
> -        * Now call up to the HV to configure the Mem-Map interrupt and
> +        * Now call up to the HV to configure the MSI interrupt and
>          * set up the IPI binding.
>          */
> -       mem_map_base = MEM_MAP_INTR_REGIONS_BASE +
> -               mem_map * MEM_MAP_INTR_REGION_SIZE;
> -       mem_map_limit = mem_map_base + MEM_MAP_INTR_REGION_SIZE - 1;
> -
>         ret = gxio_trio_config_msi_intr(trio_context, cpu_x(cpu), cpu_y(cpu),
>                                         KERNEL_PL, irq, controller->mac,
>                                         mem_map, mem_map_base, mem_map_limit,
> @@ -1516,13 +1763,9 @@
>
>         irq_set_msi_desc(irq, desc);
>
> -       msi_addr = mem_map_base + TRIO_MAP_MEM_REG_INT3 - TRIO_MAP_MEM_REG_INT0;
> -
>         msg.address_hi = msi_addr >> 32;
>         msg.address_lo = msi_addr & 0xffffffff;
>
> -       msg.data = mem_map;
> -
>         write_msi_msg(irq, &msg);
>         irq_set_chip_and_handler(irq, &tilegx_msi_chip, handle_level_irq);
>         irq_set_handler_data(irq, controller);
>
>
> What we got after my fix:
>
> pci 0000:00:00.0: BAR 8: assigned [mem 0x100c0000000-0x100c00fffff]
> pci 0000:00:00.0: BAR 9: assigned [mem 0x100c0100000-0x100c01fffff pref]
> pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> pci 0000:01:00.0: BAR 6: assigned [mem 0x100c0100000-0x100c013ffff pref]
> pci 0000:01:00.0: BAR 6: set to [mem 0x100c0100000-0x100c013ffff pref]
> (PCI address [0xc0100000-0xc013ffff])
> pci 0000:01:00.0: BAR 4: assigned [mem 0x100c0000000-0x100c000ffff
> 64bit]
> pci 0000:01:00.0: BAR 4: set to [mem 0x100c0000000-0x100c000ffff
> 64bit] (PCI address [0xc0000000-0xc000ffff])
> pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x107f]
> pci 0000:01:00.0: BAR 2: set to [io  0x1000-0x107f] (PCI address
> [0x1000-0x107f])
> pci 0000:00:00.0: PCI bridge to [bus 01-01]
> pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> pci 0000:00:00.0:   bridge window [mem 0x100c0000000-0x100c00fffff]
> pci 0000:00:00.0:   bridge window [mem 0x100c0100000-0x100c01fffff
> pref]
> pci 0001:00:00.0: BAR 8: assigned [mem 0x101c0000000-0x101c00fffff]
> pci 0001:00:00.0: BAR 9: assigned [mem 0x101c0100000-0x101c01fffff
> pref]
> pci 0001:00:00.0: BAR 7: assigned [io  0x80001000-0x80001fff]
> pci 0001:01:00.0: BAR 6: assigned [mem 0x101c0100000-0x101c013ffff
> pref]
> pci 0001:01:00.0: BAR 6: set to [mem 0x101c0100000-0x101c013ffff pref]
> (PCI address [0xc0100000-0xc013ffff])
> pci 0001:01:00.0: BAR 4: assigned [mem 0x101c0000000-0x101c000ffff
> 64bit]
> pci 0001:01:00.0: BAR 4: set to [mem 0x101c0000000-0x101c000ffff
> 64bit] (PCI address [0xc0000000-0xc000ffff])
> pci 0001:01:00.0: BAR 2: assigned [io  0x80001000-0x8000107f]
> pci 0001:01:00.0: BAR 2: set to [io  0x80001000-0x8000107f] (PCI
> address [0x80001000-0x8000107f])
> pci 0001:00:00.0: PCI bridge to [bus 01-01]
> pci 0001:00:00.0:   bridge window [io  0x80001000-0x80001fff]
> pci 0001:00:00.0:   bridge window [mem 0x101c0000000-0x101c00fffff]
> pci 0001:00:00.0:   bridge window [mem 0x101c0100000-0x101c01fffff
> pref]
> pci 0000:00:00.0: enabling device (0006 -> 0007)
> pci 0001:00:00.0: enabling device (0006 -> 0007)
> pci_bus 0000:00: resource 0 [io  0x1000-0x800007ff]
> pci_bus 0000:00: resource 1 [mem 0x100c0000000-0x100ffffffff]
> pci_bus 0000:01: resource 0 [io  0x1000-0x1fff]
> pci_bus 0000:01: resource 1 [mem 0x100c0000000-0x100c00fffff]
> pci_bus 0000:01: resource 2 [mem 0x100c0100000-0x100c01fffff pref]
> pci_bus 0001:00: resource 0 [io  0x80000800-0xffffffff]
> pci_bus 0001:00: resource 1 [mem 0x101c0000000-0x101ffffffff]
> pci_bus 0001:01: resource 0 [io  0x80001000-0x80001fff]
> pci_bus 0001:01: resource 1 [mem 0x101c0000000-0x101c00fffff]
> pci_bus 0001:01: resource 2 [mem 0x101c0100000-0x101c01fffff pref]
> ......
> mvsas 0000:01:00.0: mvsas: driver version 0.8.2
> mvsas 0000:01:00.0: enabling device (0000 -> 0003)
> mvsas 0000:01:00.0: enabling bus mastering
> mvsas 0000:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
> scsi0 : mvsas
> ......
> mvsas 0001:01:00.0: mvsas: driver version 0.8.2
> mvsas 0001:01:00.0: enabling device (0000 -> 0003)
> mvsas 0001:01:00.0: enabling bus mastering
> mvsas 0001:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
> scsi1 : mvsas
>
>
> It works now. But I really need some one to confirm whether my
> modification is enough or not,
> if there have other potential problems.
>
>
>
> Best regards.
>
> --
> Cyberman Wu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux