Hi Marc,
Thanks for your answer. Unfortunately, this device is only MSI capable.
Here are the results of lspci -vv when the (MSI) driver is loaded on both archs:
First, on a Intel PC equipped with a i7 core:
$ uname -a
Linux linux-dev 4.16.0-rc1 #4 SMP Thu Feb 22 17:11:44 CET 2018 x86_64 x86_64 x86_64 GNU/Linux
$ dmesg | grep peak_pciefd
[ 5532.242280] peak_pciefd 0000:0b:00.0: 4x CAN-FD PCAN-PCIe FPGA v3.2.1:
[ 5532.242381] peak_pciefd 0000:0b:00.0: MSI[1..4] enabling status: 4
[ 5532.242571] peak_pciefd 0000:0b:00.0: can0 at reg_base=0x00000000aab55a4d irq=70
[ 5532.242712] peak_pciefd 0000:0b:00.0: can1 at reg_base=0x0000000015bb0963 irq=71
[ 5532.242869] peak_pciefd 0000:0b:00.0: can2 at reg_base=0x000000004f07a1fc irq=72
[ 5532.242997] peak_pciefd 0000:0b:00.0: can3 at reg_base=0x00000000da023608 irq=73
$ sudo lspci -vv -s 0b:00.0
0b:00.0 Network controller: PEAK-System Technik GmbH Device 0013 (rev 01)
Subsystem: PEAK-System Technik GmbH Device 0014
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 70
NUMA node: 0
Region 0: Memory at fb200000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [50] Power Management version 0
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] MSI: Enable+ Count=4/4 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [90] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Vendor Specific Information: ID=0000 Rev=0 Len=00c <?>
Kernel driver in use: peak_pciefd
Kernel modules: peak_pciefd
Then, on the the BeagleBoard x15 (the same card but in mini-PCIe format):
# uname -a
Linux PCAN-Ethernet Gateway DR 4.9.13-rt12 #4 SMP Mon Feb 5 11:36:39 CET 2018 armv7l GNU/Linux
# dmesg |grep peak_pciefd
[ 3.457424] peak_pciefd 0000:01:00.0: 4x CAN-FD PCAN-PCIe FPGA v3.2.1:
[ 3.464444] peak_pciefd 0000:01:00.0: MSI[1..4] enabling status 4
[ 3.472992] peak_pciefd 0000:01:00.0: can0 at reg_base=0xf1b61000 irq=420
[ 3.481750] peak_pciefd 0000:01:00.0: can1 at reg_base=0xf1b62000 irq=421
[ 3.490217] peak_pciefd 0000:01:00.0: can2 at reg_base=0xf1b63000 irq=422
[ 3.498664] peak_pciefd 0000:01:00.0: can3 at reg_base=0xf1b64000 irq=423
# lspci -vv -s 01:00.0
01:00.0 Network controller: PEAK-System Technik GmbH Device 0018 (rev 01)
Subsystem: PEAK-System Technik GmbH Device 0012
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 420
Region 0: Memory at 20200000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [50] Power Management version 0
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] MSI: Enable+ Count=4/4 Maskable- 64bit+
Address: 00000000ae625000 Data: 0004
Capabilities: [90] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Vendor Specific Information: ID=0000 Rev=0 Len=00c <?>
Kernel driver in use: peak_pciefd
Once again, thank you for your help.
> -----Original Message-----
> From: Marc Zyngier <marc.zyngier@xxxxxxx>
> Sent: jeudi 22 février 2018 19:03
> To: Bjorn Helgaas <helgaas@xxxxxxxxxx>; Stéphane Grosjean
> <s.grosjean@xxxxxxxxxxxxxxx>
> Cc: linux-pci@xxxxxxxxxxxxxxx; Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> Subject: Re: free_irq(IRQ#0) on PCIe device with MSI enabled issue...
>
> On 22/02/18 15:19, Bjorn Helgaas wrote:
> > [+cc Marc, Thomas]
> >
> > Hi Stéphane,
> >
> > Thanks for your email!
> >
> > On Thu, Feb 22, 2018 at 08:23:52AM +0000, Stéphane Grosjean wrote:
> >> I've been writing some drivers for Linux-can in the past years and
> >> want to add support of MSI in one of them (see
> >>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drive
> rs/net/can/peak_canfd).
> >>
> >> This device driver works with legacy shared INTx over a CAN PCIe
> >> device. I'd like to switch to a full MSI usage of this device, which
> >> proposes up to 4 vectors (this device is proposed with 1, 2 or 4 CAN
> >> FD ports).
> >>
> >> I'm running x86_64 Kernel 4.13.0-32 from Ubuntu 17.10. Everything
> >> looks ok as I successfully get the 4 consecutive IRQ levels. My
> >> problem is this one: as soon as the driver free_irq() the FISRT IRQ
> >> level, then it does not receive any interrupt from the device
> >> anymore, for all other 3 IRQs... On the contrary, when the driver
> >> free_irq() one of the 3 other IRQ (instead of the 1st one), then
> >> there is no problem and all other interrupts (including the first IRQ
> >> vector) are working as they should.
> >
> > Can you share the actual driver code you're running? The peak_canfd.c
> > in v4.16-rc1 doesn't have the MSI code in it, so I can't see exactly
> > what you're doing.
> >
> > The fact that it does work on ARM suggests there's some difference in
> > the arch code. I don't know enough about this area to figure it out
> > from scratch, but maybe working through the actual code would help.
> >
> > It would be ideal if you could reproduce the problem on v4.16-rc1, in
> > case it's something fixed since v4.13.
> >
> >> I contact you because this "issue" seems to be arch dependent:
> >> running the same driver and the same board on another arch (ARM),
> >> then everything works like a charm: the driver can free_irq() any IRQ
> >> in any order without disturbing the other ones. But running on x86,
> >> free_irq() on the FIRST MSI seems to (also) free all the other
> >> ones...
> >>
> >> Note that the driver set up MSI as described in the last version of
> >> Documentation/PCI/MSI-HOWTO.txt.
>
> Could it be that the driver is ending up with Multi-MSI instead of MSI-X? If
> using Multi-MSI, I think that would be the expected behaviour, as the various
> MSIs are not controllable independently (there is a single interrupt mask for
> all MSIs). On the other hand, MSI-X gives you a bunch of completely
> independent interrupts.
>
> An lspci -vv dump on both platforms would go a long way in helping to
> understand the setup.
>
> Thanks,
>
> M.
> --
> Jazz is not dead. It just smells funny...
--
PEAK-System Technik GmbH
Sitz der Gesellschaft Darmstadt - HRB 9183
Geschaeftsfuehrung: Alexander Gach / Uwe Wilhelm
--
