Hi Greg,
Thanks for your comments. I managed to install a client which will do
the auto format. Hopefully this is better than before. Please find my
responses inline. Also added extra questions and information about the
setup.
Questions:
1. Since the modified_my_driver_probe is working, is that the recommended
way of doing it? I couldn't find any sample drivers using it. So wanted
your guidance
2. There is no hotplug controller as far is know off in the setup (
assuming native x86 soc has root port which has inbuilt hp controller
and pcieport driver takes care of it). Am i missing something?
3. When card 2 is inserted there is no hotplug event. Not sure why.
Should anything be done from driver side ? I just started looking into
the pcie switch eeprom config space settings to check if there is any
mask for the event notification , assuming if masks cleared, pcieport
driver can be notified by pciehp similar to card 1 insertion.
HW details:
x86 ( Soc) -> pcie root port 0 -> Device 0 (FPGA_0 )
x86 ( Soc) -> pcie root port 1 -> [ pcie-switch_1_port 1 -> Device 1
(FPGA_1 ) ]
x86 ( Soc) -> pcie root port 1 -> [ pcie-switch_1_port 2 -> Device 2 ]
x86 ( Soc) -> pcie root port 2 -> pcie-switch_2_port 1 -> [ pcie-
switch_3_port 1 -> Device 3 ( FPGA_2) ]
x86 ( Soc) -> pcie root port 2 -> pcie-switch_2_port 1 -> [ pcie-
switch_3_port 2 -> Device 4 ]
card 1- pcie-switch_1, Device 1 and 2 are on the same physical pcb card
which can be hotplugged ( inserted or removed )
card 2- pcie-switch_3, Device 3 and 4 are on the same physical pcb card
which can be hotplugged ( inserted or removed )
Drivers details:
- FPGA_0-2 are devices have "my_pci_driver"
- Device 2 and Device 4 are third party devices which have their drivers
- All pcie ports has handled by "pcieport" driver (output of lspci -vvv)
- pciehp core driver is enabled
Sequence of operations:
- Boot up
- Insmod my_pci_driver, third party device driver
- Insert card 1 ( output sequence explained below happens )
- Insert card 2 ( nothing happens )
- issue /sys/pci/bus/rescan command ( output sequence explained below
happens again )
Output sequence:
- pcieport driver see an event and will start enabling downstream
devices
- my_pci_driver and third party device drivers probe functions get
called
- third party driver probe function completes and the BAR is assigned (
output of lspci -vvv -s B:D:F )
- my_pci_driver probe function throws out an error and the probe fails.
error as below
pcieport 0000:54:05.0: enabling device (0000 -> 0002)
my-driver 0000:56:00.3: can't enable device: BAR 0 [mem
0x00000000-0x000fffff 64bit pref] not claimed
my-driver 0000:56:00.3: Failed to enable My device rc=-22
my-driver: probe of 0000:56:00.3 failed with error -22
Modify driver:
- when i add the pci_reassign_resource function before the
pci_enable_device i nolonger see the error
- BAR is assigned in the lspci -vvv output
my_driver_probe:
static int my_probe(struct pci_dev *pdev, const struct pci_device_id
*id)
{
ret = pci_enable_device(pdev);
if (ret) {
dev_err(&pdev->dev,"Failed to enable my device
rc=%d\n",ret);
goto err_pci_enable;
}
ret = pci_request_regions(pdev, dev_driver_string(&pdev->dev));
if (ret) {
dev_err(&pdev->dev, "Failed to request PCI regions
rc=%d\n", ret);
goto err_pci_req;
}
base_addr = pci_iomap(pdev, 0, 0);
if (IS_ERR(base_addr)) {
dev_err(&pdev->dev, "Failed to iomap PCI BAR %d\n", 0);
goto err_pci_iomap;
}
...
...
...
}
modified_my_driver_probe:
static int my_probe(struct pci_dev *pdev, const struct pci_device_id
*id)
{
/* Enable hotplug support
TODO: Check if this is right way to do it */
if( pci_resource_start (pdev,0) <= 0) {
ret = pci_assign_resource(pdev,0);
if( 0 > ret) {
dev_err(&pdev->dev, " Failed to assign resource
\n");
return ret;
}
}
ret = pci_enable_device(pdev);
if (ret) {
dev_err(&pdev->dev,"Failed to enable my device
rc=%d\n",ret);
goto err_pci_enable;
}
ret = pci_request_regions(pdev, dev_driver_string(&pdev->dev));
if (ret) {
dev_err(&pdev->dev, "Failed to request PCI regions
rc=%d\n", ret);
goto err_pci_req;
}
base_addr = pci_iomap(pdev, 0, 0);
if (IS_ERR(base_addr)) {
dev_err(&pdev->dev, "Failed to iomap PCI BAR %d\n", 0);
goto err_pci_iomap;
}
}
-Divakar
On Wednesday, August 03, 2016 07:12:16 AM Greg KH wrote:
> On Tue, Aug 02, 2016 at 09:11:09PM +0000, Chitturi, Divakar wrote:
> > Hi
> >
> > Below are two scenarios for which i would like to understand the
> > right way to assign the BAR address for my PCI device
> >
> > Device details:
> > pci switch upstrem port connected to root complex
> > pci switch downstream port 1 connected to my_pci_device
> > my_pci_device is FPGA
> > FPGA has bunch of controllers and other logic
> >
> > Driver details:
> > Have a dedicated pci kernel driver for my_device
> > Enabled pcihp kernel driver
>
> Which pci hotplug driver? Just the PCI Hotplug core isn't going to do
> anything, you need a controller driver for your specific PCI hotplug
> hardware.
>
> > At power on, BIOS assigns/reserves specific region of memory for my
> > pci device. during the linux boot process, pci core will assign the
> > bar and other initialization and call my probe function in the
> > driver. probe function will perform some initialization which
> > includes reading the BAR and setting some registers in IO space of
> > the device.
> >
> > Scn1:
> > At some point the device is removed and reinserted
> >
> > Scn2:
> > At some point similar device is inserted in a new pci slot
>
> How are they added/removed without a hotplug controller?
>
> > In both the scenarios explained about, BIOS is not involved as
> > system was never turned off completely. So 1. how do we obtain the
> > BAR address for the device.
>
> What architecture is this? x86? If so, it just uses whatever the
> BIOS said was there for the last time.
Yes this is x86 CPU. BIOS has reserved the memory region for the bus
slot on which new device sits during boot. When a new device (downstream
fpga of pcie switch) is inserted and i issue the "echo 1 >
/sys/bus/pci/rescan/ the device shows up in lscpi -t but no BAR is
assigned.
>
> > 2. Should i assign it explicitly?
>
> Yes.
>
> > 3. if yes, should it be done in my driver ( in both the case i am
> > assuming driver's probe function will be called )
> No, in your pci hotplug driver, you need to handle it there.
>
> > 4. can i use pci_assign_resource() in drivers probe function
>
> Nope.
>
> > 5. is something like below the right way to do it ?
> >
> > static int my_probe(struct pci_dev *pdev, const struct pci_device_id
> > *id) {
> >
> > if( pci_resource_start (pdev,BAR0) <= 0) {
> >
> > pr_debug(" Assign BAR0 \n");
> >
> > ret = pci_assign_resource(pdev,BAR0);
> >
> > if( 0 > ret) {
> >
> > dev_err(&pdev->dev, " Failed to assign resource
> > \n");
> > return ret;
> >
> > }
> >
> > }
> >
> > ret = pci_enable_device(pdev);
> > ret = pci_request_regions(pdev,
> > dev_driver_string(&pdev->dev));
> > base_addr = pci_iomap(pdev, BAR0, 0);
> >
> > }
>
> You need to write a pci hotplug controller driver to manage the
> resources and assign them properly. That's its responsibility, see
> the PCI hotplug specification for all of the details...
Right now by using this function in the probe function of the kernel
driver for the new device (downstream fpga of pcie switch ) inserted (
scn2 ) it is able to obtain the BAR and I can access the IO space and
read /write.
However I am not convinced if this is the right way to do it.
>
> good luck!
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-hotplug" in the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-hotplug" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
