> On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@xxxxxxx> wrote:
>
> On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
>>
>>> On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland@xxxxxxx> wrote:
>>>
>>> On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
>>>> This is some thing we also like to see in ITS and SMMU drivers.
>>>>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@xxxxxxx> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> For some devices, identification of particular masters is critical to
>>>>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>>>>> is determined from sideband signals on the bus, and it may or may not be
>>>>> possible to probe and/or configure this information. Worse still, these
>>>>> information may be rewritten by intermediate buses, so the
>>>>> information for a given master may differ at different points in the bus
>>>>> hierarchy.
>>>>>
>>>>> We currently describe this master information for devices attached to
>>>>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
>>>>> only covers the ID between the master and its IOMMU(s), and we don't
>>>>> currently have a mechanism to describe the master IDs as they are seen
>>>>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
>>>>>
>>>>> The following are example of master IDs:
>>>>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>>>>> * SMMU Stream IDs (SIDs)
>>>>> * GICv3 ITS Device IDs (DIDs)
>>>>>
>>>>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>>>>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>>>>>
>>>>> +-------------+
>>>>> | PCIe master |
>>>>> +-------------+
>>>>> ||
>>>>> || Requester ID (RID)
>>>>> || Probeable from RC.
>>>>> \/
>>>>> +-------------------+
>>>>> | PCIe Root Complex |
>>>>> +-------------------+
>>>>> ||
>>>>> || SMMU Stream ID (SID)
>>>>> || Derived from RID, static non-probeable mapping.
>>>>> \/
>>>>> +--------------+
>>>>> | SMMU (IOMMU) |
>>>>> +--------------+
>>>>> ||
>>>>> || ITS Device ID (DID)
>>>>> || Derived from SID, static non-probeable mapping.
>>>>> \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>>
>>>>> In simpler cases, you might have a simpler set of master ID
>>>>> translations, e.g. just a DID:
>>>>>
>>>>> +-----------------+
>>>>> | Platform device |
>>>>> +-----------------+
>>>>> ||
>>>>> || ITS Device ID (DID)
>>>>> || Hard-wired on the bus.
>>>>> \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>>
>>>>> Ignoring current bindings for the moment, I can see how we can describe
>>>>> this with a generic master id-binding, with master-id-map along the
>>>>> lines of interrupt-map, with a tuple format like:
>>>>> <child-id-base child-id-length parent parent-id-base>
>>>>>
>>>>> For the PCIe example, this would look something like the following (with
>>>>> properties omitted for brevity):
>>>>>
>>>>> PCI: pci@af000000 {
>>>>> ...
>>>>>
>>>>> /* Requester ID of PCIe endpoints, implicit at runtime */
>>>>> master-id-cells = <1>;
>>>>>
>>>>> /* RIDS idmapped to SIDS @ SMMU */
>>>>> master-id-map = <0 0x10000 &SMMU 0>;
>>>>> }
>>>>>
>>>>> SMMU: iommu@bf000000 {
>>>>> ...
>>>>>
>>>>> /* SID, derived from RID */
>>>>> master-id-cells = <1>;
>>>>>
>>>>> /*
>>>>> * For some reason the high bit of the ID was negated.
>>>>> */
>>>>> master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>>>>> <0x8000 0x8000 &ITS 0x0 0x0000>;
>>>>>
>>>>> };
>>>>>
>>>>> ITS: its@cf000000 {
>>>>> ...
>>>>>
>>>>> /* DID, derived from SID */
>>>>> master-id-cells = <2>;
>>>>>
>>>>> /*
>>>>> * Master traffic not propagated beyond this point, so no
>>>>> * master-id-ranges
>>>>> */
>>>>> };
>>>>
>>>> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.
>>>
>>> The set of IDs that we expect to transform (i.e. those masters use)
>>> would be implicit in the first master-id-map from a master. In the PCI
>>> example above, no master is expected to generate an ID outside of the
>>> range 0-0x10000 inclusive (and that's all the SMMU would see).
>>>
>>> For devices which are not hotpluggable, the nodes for those devices
>>> would describe the specific set of IDs they use.
>>>
>>> Generally, between a master and a slave there might not be one
>>> contiguous set of valid IDs as these may be rewritten along the way (as
>>> happens at the SMMU between the PCI root complex and the ITS in the
>>> example above).
>>>
>>> Which drivers do you think need this information? What exactly are they
>>> trying to check and enforce?
>>>
>>
>> I think, i miss understood by the names, so it is not base it is some thing like
>>
>> x => f(x) => y (and f(x) is bitwise | )
>>
>> ok now i see it.
>
> It's subtraction and addition rather than bitwise ORing.
>
> Consider a SID 0x8040 entering the SMMU. This falls in the range
> described by the SMMU's second master-id-map entry (I've added brackets
> to separate each entry:
>
> <(0x8000) (0x8000) (&ITS) (0x0 0x0000)>
>
> As 0x8040 falls in the range described by the first two elements, we kow
> we can translate the ID to calculate the ID at the ITS. To do so, we
> subtract the child-id-base (0x8000), and add the parent-id-base (0x0
> 0x0000). That give us 0x0 0x0040 as our ITS device ID.
fine.
>
>>>>> For the simpler case, this would look something like:
>>>>>
>>>>> DEV: device@af000000 {
>>>>> master-id-cells = <1>;
>>>>> master-ids = <0xf>, <0xb>;
>>>>> master-id-map = <0 0xf &ITS 0 0>;
>>>>> };
>>>>>
>>>>> ITS: its@cf000000 {
>>>>> ...
>>>>>
>>>>> /* DID */
>>>>> master-id-cells = <2>;
>>>>> };
>>>>>
>>>> Is this is not depending heavily on discover order, how do drivers
>>>> know which device to get which ID. is it implicitly assumed in
>>>> discovery order?
>>>
>>> I'm not sure I follow the question. Could you elaborate?
>>>
>>>> what happens to hot pluggable devices.
>>>
>>> It would be necessary to be able to discover the ID assigned to the
>>> device by the hotpluggable bus. For example, this could depend on which
>>> slot the device is plugged into.
>>>
>>> If you can't discover the ID associated with a hotpluggable device from
>>> the bus it is plugged into I can't see how hotplug could work.
>>>
>>> From that point on I would expect the ID transformations to be static as
>>> in the example.
>>>
>>
>> what if i have an SMMU with two ITS masters and i want to decide which
>> one to use based on runtime like which CPU i am serving.
>>
>> i am thinking in terms go NUMA based systems, where my pci device is
>> on node 0, if the interrupt is targeted to CPU on node 1, i would like
>> to use ITS1, otherwise
>> ITS0.
>
> The binding can express that both these translation paths exist. For
> example, on the SMMU node you could have:
>
> master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
> <0x0000 0x8000 &ITS_2 0x0 0x0000>;
>
> Which would mean that master IDs are idmapped to both ITS instances.
>
>> is it possible to specify some thing like that or do you think its is
>> best to leave to ITS driver.
>
> It's possible to specify that the paths exist. I expect that software
> would select which to use at runtime.
>
My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.
Thanks,
Tirumalesh.
> Thanks,
> Mark.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
