On Thu, 24 Jun 2021 17:14:39 +0200 Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote: > After studying the MSI-X specification again, I think there is another > option to solve this for MSI-X, i.e. the dynamic sizing part: > > MSI requires to disable MSI in order to update the number of enabled > vectors in the control word. Exactly what part of the spec requires this? This is generally the convention I expect too, and there are complications around contiguous vectors and data field alignment, but I'm not actually able to find a requirement in the spec that MSI Enable must be 0 when modifying other writable fields or that writable fields are latched when MSI Enable is set. > MSI-X does not have that requirement as there is no 'number of used > vectors' control field. MSI-X provides a fixed sized vector table and > enabling MSI-X "activates" the full table. > > System software has to set proper messages in the table and eventually > associate the table entries to device (sub)functions if that's not > hardwired in the device and controlled by queue enablement etc. > > According to the specification there is no requirement for masked table > entries to contain a valid message: > > "Mask Bit: ... When this bit is set, the function is prohibited from > sending a message using this MSI-X Table entry." > > which means that the function must reread the table entry when the mask > bit in the vector control word is cleared. What is a "valid" message as far as the device is concerned? "Valid" is meaningful to system software and hardware, the device doesn't care. Like MSI above, I think the real question is when is the data latched by the hardware. For MSI-X this seems to be addressed in (PCIe 5.0 spec) 6.1.4.2 MSI-X Configuration: Software must not modify the Address, Data, or Steering Tag fields of an entry while it is unmasked. Followed by 6.1.4.5 Per-vector Masking and Function Masking: For MSI-X, a Function is permitted to cache Address and Data values from unmasked MSI-X Table entries. However, anytime software unmasks a currently masked MSI-X Table entry either by Clearing its Mask bit or by Clearing the Function Mask bit, the Function must update any Address or Data values that it cached from that entry. If software changes the Address or Data value of an entry while the entry is unmasked, the result is undefined. So caching/latching occurs on unmask for MSI-X, but I can't find similar statements for MSI. If you have, please note them. It's possible MSI is per interrupt. Anyway, at least MSI-X if not also MSI could have a !NORESIZE implementation, which is why this flag exists in vfio. Thanks, Alex
