From: Thomas Gleixner > Sent: 10 April 2023 07:50 ... > Whatever order this reads does not matter. The point is: > > "Mask Bit - When this bit is Set, the Function is prohibited from > sending a message using this MSI-X Table entry." > > So you cannot make this read order dependent at all. > > > Anything fpga based is likely to be using a 32bit memory > > block for the MSI-X data (possibly even 16bit). > > It's trivial enough to latch the message on unmask into a shadow > register set and let the state engine work from there. I could implement the memory block inside the MSI_X logic and then detect writes. Anything else uses a lot more fpga resource. > And no, we are not adding random delays to that code just because. I was mostly suggesting one be moved. (Changing the address/data is hardly a hot path.) Actually I'm trying to work out what the readbacks in the MSI-X code are actually for (apart from adding a 'random delay'). If you are masking one of the PCI INTx lines, then a readback delays the cpu until the physical line is de-asserted so that the IRQ line into the interrupt controlled is de-asserted and thus the IRQ to the cpu is also de-asserted. Even if the cpu commits to the interrupt, the interrupt controller reports 'no vector' (or vector 7 if it is the 8259). So you can safely do a readback, enable interrupts and know the interrupt won't happen. (Although if you have a real 8259 you need to worry about the cycle recovery time, even a 286 will break it!) But MSI-X is different, it is very much edge sensitive. The target MSI-X logic can decide to raise an interrupt just before the mask bit it set. It will then request the target's PCIe interface issue the write TLP, this is probably a 'posted' write in the internal logic. The PCIe interface could be busy generating a long write TLP (or two) before actually issuing write for the interrupt. The readback will be pretty much back-to-back with the mask write (as seen on the target). So maybe the rad can get completed while the write is still queued. Now I can't remember whether the data TLP for reads is allowed to overtake a posted write request, but it really doesn't matter. Even if the IRQ write arrives at the root hub before the read completion they then head off in different directions through the 'cpu' logic - so the read could easily complete well before the interrupt gets processed. In fact an interrupt requested well before the mask bit is set could be pending in the ioapic waiting for the cpu to enable interrupts. So I don't see any reason for those readbacks at all. I've never looked at the cpu end of MSI-X, but I suspect that stop using an interrupts you need to mask it there first, then disable the hardware and later make sure you clear any lurking pending request (probably before unmasking for later use). David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
