(sending again, by mistake it was set to html mode)
On Tue, Sep 29, 2015 at 2:05 PM, Ganapatrao Kulkarni
<gpkulkarni@xxxxxxxxx> wrote:
> Hi Mark,
>
> I have tried to answer your comments, in the meantime we are waiting for Ben
> to share the details.
>
> On Fri, Aug 28, 2015 at 6:02 PM, Mark Rutland <mark.rutland@xxxxxxx> wrote:
>>
>> Hi,
>>
>> On Fri, Aug 14, 2015 at 05:39:32PM +0100, Ganapatrao Kulkarni wrote:
>> > DT bindings for numa map for memory, cores and IOs using
>> > arm,associativity device node property.
>>
>> Given this is just a copy of ibm,associativity, I'm not sure I see much
>> point in renaming the properties.
>>
>> However, (somewhat counter to that) I'm also concerned that this isn't
>> sufficient for systems we're beginning to see today (more on that
>> below), so I don't think a simple copy of ibm,associativity is good
>> enough.
>
> it is just copy right now, however it can evolve when we come across more
> arm64 numa platforms
>>
>>
>> >
>> > Signed-off-by: Ganapatrao Kulkarni <gkulkarni@xxxxxxxxxxxxxxxxxx>
>> > ---
>> > Documentation/devicetree/bindings/arm/numa.txt | 212
>> > +++++++++++++++++++++++++
>> > 1 file changed, 212 insertions(+)
>> > create mode 100644 Documentation/devicetree/bindings/arm/numa.txt
>> >
>> > diff --git a/Documentation/devicetree/bindings/arm/numa.txt
>> > b/Documentation/devicetree/bindings/arm/numa.txt
>> > new file mode 100644
>> > index 0000000..dc3ef86
>> > --- /dev/null
>> > +++ b/Documentation/devicetree/bindings/arm/numa.txt
>> > @@ -0,0 +1,212 @@
>> >
>> > +==============================================================================
>> > +NUMA binding description.
>> >
>> > +==============================================================================
>> > +
>> >
>> > +==============================================================================
>> > +1 - Introduction
>> >
>> > +==============================================================================
>> > +
>> > +Systems employing a Non Uniform Memory Access (NUMA) architecture
>> > contain
>> > +collections of hardware resources including processors, memory, and I/O
>> > buses,
>> > +that comprise what is commonly known as a NUMA node.
>> > +Processor accesses to memory within the local NUMA node is generally
>> > faster
>> > +than processor accesses to memory outside of the local NUMA node.
>> > +DT defines interfaces that allow the platform to convey NUMA node
>> > +topology information to OS.
>> > +
>> >
>> > +==============================================================================
>> > +2 - arm,associativity
>> >
>> > +==============================================================================
>> > +The mapping is done using arm,associativity device property.
>> > +this property needs to be present in every device node which needs to
>> > to be
>> > +mapped to numa nodes.
>>
>> Can't there be some inheritance? e.g. all devices on a bus with an
>> arm,associativity property being assumed to share that value?
>
> yes there is inheritance and respective bus drivers should take care of it,
> like pci driver does at present.
>>
>>
>> > +
>> > +arm,associativity property is set of 32-bit integers which defines
>> > level of
>>
>> s/set/list/ -- the order is important.
>
> ok
>>
>>
>> > +topology and boundary in the system at which a significant difference
>> > in
>> > +performance can be measured between cross-device accesses within
>> > +a single location and those spanning multiple locations.
>> > +The first cell always contains the broadest subdivision within the
>> > system,
>> > +while the last cell enumerates the individual devices, such as an SMT
>> > thread
>> > +of a CPU, or a bus bridge within an SoC".
>>
>> While this gives us some hierarchy, this doesn't seem to encode relative
>> distances at all. That seems like an oversight.
>
>
> distance is computed, will add the details to document.
> local nodes will have distance as 10(LOCAL_DISTANCE) and every level, the
> distance multiplies by 2.
> for example, for level 1 numa topology, distance from local node to remote
> node will be 20.
>
>>
>>
>> Additionally, I'm somewhat unclear on how what you'd be expected to
>> provide for this property in cases like ring or mesh interconnects,
>> where there isn't a strict hierarchy (see systems with ARM's own CCN, or
>> Tilera's TILE-Mx), but there is some measure of closeness.
>
>
> IIUC, as per ARMs CCN architecture, all core/clusters are at equal distance
> of DDR, i dont see any NUMA topology.
> however, if there are 2 SoC connected thorough the CCN, then it is very much
> similar to cavium topology.
>
>> Must all of these have the same length? If so, why not have a
>> #(whatever)-cells property in the root to describe the expected length?
>> If not, how are they to be interpreted relative to each other?
>
>
> yes, all are of default size.
> IMHO, there is no need to add cells property.
>>
>>
>> > +
>> > +ex:
>>
>> s/ex/Example:/, please. There's no need to contract that.
>>
>> > + /* board 0, socket 0, cluster 0, core 0 thread 0 */
>> > + arm,associativity = <0 0 0 0 0>;
>> > +
>> >
>> > +==============================================================================
>> > +3 - arm,associativity-reference-points
>> >
>> > +==============================================================================
>> > +This property is a set of 32-bit integers, each representing an index
>> > into
>>
>> Likeise, s/set/list/
>
> ok
>>
>>
>> > +the arm,associativity nodes. The first integer is the most significant
>> > +NUMA boundary and the following are progressively less significant
>> > boundaries.
>> > +There can be more than one level of NUMA.
>>
>> I'm not clear on why this is necessary; the arm,associativity property
>> is already ordered from most significant to least significant per its
>> description.
>
>
> first entry in arm,associativity-reference-points is used to find which
> entry in associativity defines node id.
> also entries in arm,associativity-reference-points defines,
> how many entries(depth) in associativity can be used to calculate node
> distance
> in both level 1 and multi level(hierarchical) numa topology.
>
>>
>>
>> What does this property achieve?
>>
>> The description also doesn't describe where this property is expected to
>> live. The example isn't sufficient to disambiguate that, especially as
>> it seems like a trivial case.
>
> sure, will add one more example to describe the
> arm,associativity-reference-points
>>
>>
>> Is this only expected at the root of the tree? Can it be re-defined in
>> sub-nodes?
>
> yes it is defined only at the root.
>>
>>
>> > +
>> > +Ex:
>>
>> s/Ex/Example:/, please
>
> sure.
>>
>>
>> > + arm,associativity-reference-points = <0 1>;
>> > + The board Id(index 0) used first to calculate the associativity
>> > (node
>> > + distance), then follows the socket id(index 1).
>> > +
>> > + arm,associativity-reference-points = <1 0>;
>> > + The socket Id(index 1) used first to calculate the
>> > associativity,
>> > + then follows the board id(index 0).
>> > +
>> > + arm,associativity-reference-points = <0>;
>> > + Only the board Id(index 0) used to calculate the associativity.
>> > +
>> > + arm,associativity-reference-points = <1>;
>> > + Only socket Id(index 1) used to calculate the associativity.
>> > +
>> >
>> > +==============================================================================
>> > +4 - Example dts
>> >
>> > +==============================================================================
>> > +
>> > +Example: 2 Node system consists of 2 boards and each board having one
>> > socket
>> > +and 8 core in each socket.
>> > +
>> > + arm,associativity-reference-points = <0>;
>> > +
>> > + memory@00c00000 {
>> > + device_type = "memory";
>> > + reg = <0x0 0x00c00000 0x0 0x80000000>;
>> > + /* board 0, socket 0, no specific core */
>> > + arm,associativity = <0 0 0xffff>;
>> > + };
>> > +
>> > + memory@10000000000 {
>> > + device_type = "memory";
>> > + reg = <0x100 0x00000000 0x0 0x80000000>;
>> > + /* board 1, socket 0, no specific core */
>> > + arm,associativity = <1 0 0xffff>;
>> > + };
>> > +
>> > + cpus {
>> > + #address-cells = <2>;
>> > + #size-cells = <0>;
>> > +
>> > + cpu@000 {
>> > + device_type = "cpu";
>> > + compatible = "arm,armv8";
>> > + reg = <0x0 0x000>;
>> > + enable-method = "psci";
>> > + /* board 0, socket 0, core 0*/
>> > + arm,associativity = <0 0 0>;
>>
>> We should specify w.r.t. memory and CPUs how the property is expected to
>> be used (e.g. in the CPU nodes rather than the cpu-map, with separate
>> memory nodes, etc). The generic description of arm,associativity isn't
>> sufficient to limit confusion there.
>
> ok, will add the details like which nodes can use this property.
>
>>
>>
>> Thanks,
>> Mark.
>
>
> thanks
> Ganapat
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
