Hi Thomas,
On 8/6/2020 10:10 AM, Thomas Gleixner wrote:
Megha,
"Dey, Megha" <megha.dey@xxxxxxxxx> writes:
-----Original Message-----
From: Jason Gunthorpe <jgg@xxxxxxxxxxxx>
<SNIP>
Subject: Re: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI
irq domain
can you please fix your mail client not to copy the whole header of the
mail you are replying to into the mail body?
oops, i hope i have fixed it now..
Well, I had suggested to pass in the parent struct device, but it
Oops, I was thinking of platform_msi_domain_alloc_irqs() not
create_device_domain()
ie call it in the device driver that wishes to consume the extra MSIs.
Is there a harm if each device driver creates a new irq_domain for its use?
Well, the only harm is if we want to reuse the irq domain.
You cannot reuse the irq domain if you create a domain per driver. The
way how hierarchical domains work is:
vector --- DMAR-MSI
|
|-- ....
|
|-- IR-0 --- IO/APIC-0
| |
| |-- IO/APIC-1
| |
| |-- PCI/MSI-0
| |
| |-- HPET/MSI-0
|
|-- IR-1 --- PCI/MSI-1
| |
The outermost domain is what the actual device driver uses. I.e. for
PCI-MSI it's the msi domain which is associated to the bus the device is
connected to. Each domain has its own interrupt chip instance and its
own data set.
Domains of the same type share the code, but neither the data nor the
interrupt chip instance.
Also there is a strict parent child relationship in terms of resources.
Let's look at PCI.
PCI/MSI-0 depends on IR-0 which depends on the vector domain. That's
reflecting both the flow of the interrupt and the steps required for
various tasks, e.g. allocation/deallocation and also interrupt chip
operations. In order to allocate a PCI/MSI interrupt in domain PCI/MSI-0
a slot in the remapping unit and a vector needs to be allocated.
If you disable interrupt remapping all the outermost domains in the
scheme above become childs of the vector domain.
So if we look at DEV/MSI as a infrastructure domain then the scheme
looks like this:
vector --- DMAR-MSI
|
|-- ....
|
|-- IR-0 --- IO/APIC-0
| |
| |-- IO/APIC-1
| |
| |-- PCI/MSI-0
| |
| |-- HPET/MSI-0
| |
| |-- DEV/MSI-0
|
|-- IR-1 --- PCI/MSI-1
| |
| |-- DEV/MSI-1
But if you make it per device then you have multiple DEV/MSI domains per
IR unit.
What's the right thing to do?
If the DEV/MSI domain has it's own per IR unit resource management, then
you need one per IR unit.
If the resource management is solely per device then having a domain per
device is the right choice.
Thanks a lot Thomas for this detailed explanation!!
The dev-msi domain can be used by other devices if they too would want
to follow the
vector->intel IR->dev-msi IRQ hierarchy.
I do create one dev-msi IRQ domain instance per IR unit. So I guess for
this case,
it makes most sense to have a dev-msi IRQ domain per IR unit as opposed
to create one
per individual driver..
Thanks,
tglx
