# Title: Enumerate "Inclusive Linear Address Mode" memory-side caches
# Status: Draft v2
# Document: ACPI Specification 6.6
# License
SPDX-License Identifier: CC-BY-4.0
# Submitter:
* Sponsor: Dan Williams, Intel
* Creators/Contributors:
* Andy Rudoff, retired
* Mahesh Natu, Intel
* Ishwar Agarwal, Intel
# Changelog
* v2: Clarify the "Inclusive" term as "including the capacity of the cache
in the SRAT range length"
* v2: Clarify that 0 is an undeclared / transparent Address Mode, and
that Address Mode values other than 1 are Reserved.
# Summary of the Change
Enable an OSPM to enumerate that the capacity for a memory-side cache is
"included" in an SRAT range. Typically the "Memory Side Cache Size"
enumerated in the HMAT is "excluded" from the SRAT range length because
it is a transparent cache of the SRAT capacity. The enumeration of this
addressing mode enables OSPM memory RAS (Reliability, Availability, and
Serviceability) flows.
Recall that the CXL specification allows for platform address ranges to
be interleaved across CXL and non-CXL targets. CXL 3.1 Table 9-22 CFMWS
Structure states "If the Interleave Set spans non-CXL domains, this list
may contain values that do not match \_UID field in any CHBS structures.
These entries represent Interleave Targets that are not CXL Host
Bridges". For an OSPM this means address translation needs to be
prepared for non-CXL targets. Now consider the case when that CXL
address range is flagged as a memory side cache in the ACPI HMAT.
Address translation needs to consider that the decode for an error may
impact multiple components (FRUs fields replaceable units).
Now consider the implications of ["Flat Memory Mode" (Intel presentation
at Hot Chips
2023)](https://cdrdv2-public.intel.com/787386/Hot%20Chips%20-%20Aug%2023%20-%20BHS%20and%20Granite%20Rapid%20-%20Xeon%20-%20Architecture%20-%20Public.pdf).
This cache geometry implies an address space that includes the
memory-side cache size in the reported address range. For example, a
typical address space layout for a memory-side-cache of 32GB of DDR
fronting 64GB of CXL would report 64GB in the "Length" field of the
SRAT's "Memory Affinity Structure" and 32GB in the "Memory Side Cache
Size" field of the HMAT's "Memory Side Cache Information Structure". An
inclusive address-space layout of the same configuration would report
96GB in the "Length" field of the SRAT's "Memory Affinity Structure" and
32GB in the "Memory Side Cache Size" field of the HMAT's "Memory Side
Cache Information Structure". The implication for address translation in
the inclusive case, is that there are N potential aliased address
impacted by a memory error where N is the ratio of:
SRAT.MemoryAffinityStructure.Length /
HMAT.MemorySideCacheInformation.CacheSize
This change request is not exclusive to CXL, the concept is applicable
to any memory-side-cache configuration that the HMAT+SRAT can describe.
However, CXL is a primary motivator given the OSPM role in address
translation for device-physical-address (DPA) events being translated to
impacted host-physical-address (HPA) events.
# Benefits of the Change
An OSPM, when it knows about inclusive cache address space, can take
actions like quarantine / offline all the impacted aliased pages to
prevent further consumption of poison, or run repair operations on all
the affected targets. Without this change an OSPM may not accurately
identify the HPA associated with a given CXL FRU event, or it may
misunderstand that an SRAT memory affinity range is an amalgam of CXL
and cache capacity.
# Impact of the Change
The proposed "Address Mode" field to convey this configuration consumes
the 2 Reserved bytes following the "Cache Attributes" field in the
"Memory Side Cache Information Structure". The default reserved value of
0 indicates the status quo of an undeclared addressing mode where the
expectation is that it is safe to assume a transparent cache where the
cache-capacity is not included in the SRAT range capacity. An OSPM that
knows about new values can perform address decode according to the
proposed details below and a legacy OSPM will ignore it as a Reserved
field.
# References
* Compute Express Link Specification v3.1,
<https://www.computeexpresslink.org/>
# Detailed Description of the Change
* Section Table 5.149: Memory Side Cache Information Structure redefine
the 2 Reserved bytes starting at offset 28 as "Address Mode":
* 0 - Reserved (OSPM may assume transparent cache addressing)
* 1 - Inclusive linear (N aliases linearly mapped)
* 2..65535 - Reserved (Unknown Address Mode)
* Extend the implementation note after Table 5.149 to explain how to
interpret the "Inclusive linear" mode.
* "When Address Mode is 1 'Inclusive Linear' it indicates that there
are N directly addressable aliases of a given cacheline
where N is the ratio of target memory proximity domain size and
the memory side cache size. Where the N aliased addresses for a
given cacheline all share the same result for the operation
'address modulo cache size'."
