Jonathan Cameron wrote: > On Fri, 17 May 2024 13:20:06 -0700 > Dan Williams <dan.j.williams@xxxxxxxxx> wrote: [..] > > I would prefer to just pile on more explicit clarifications to overcome that > > instinct to map the word "Inclusive" to a multi-level cache attribute. Something > > like "note, 'Inclusive Linear' address-mode not to be confused with > > 'Inclusive/Exclusive' multi-level cache organization". > > Hmm. I'll go with 'maybe'. When you have to add a bunch of distinctions of > a term that is applying to a 'cache' to say that you don't mean the standard > meaning for caches, it feels like a new term is a better path. Right, but that's the crux I am talking about the address organization and not the cache policy, and it seems all too easy fall into the trap of considering this to be a caching-policy attribute. > Possibly as I suggest later a hyphen might avoid misreads. > inclusive-linear address mode. Yeah, that is a good idea. > > > Whilst the CXL side of things (and I assume your hardware migration engine) > > > don't provide a way to recover this, it would be possible to build > > > a system that otherwise looked like you describe that did provide access > > > to the tag bits and so wouldn't present the aliasing problem. > > > > Aliasing problem? All direct-mapped caches have aliases, it just happens that > > this address mode allows direct-addressability of at least one alias. > > As I understand this the problem is you get address A in the error record, > but that actually means any of A, A + N, A + 2N etc and the issue is you > have no way of recovering which alias you have. > > Another implementation might have the same aliasing in the cache, but allow > for establishing which one you have (the hardware inherently has to know that > but I presume in this case doesn't provide a way to look it up - or if it > does, then issue here is that the OS querying of the CXL device doesn't know > about that interface?). So I think the critical here is that information is > not available, not that aliasing occurs. The critical information is that the address range is extended by the cache capacity compared to the typical case. Maybe "extended-linear" is the term I was searching for last Friday when I could not think of a better bikeshed color? The reason an "extended-linear" indicator is important is for the driver to recognize that the CXL address range programmed into the decoders is only a subset of the system-physical address ranges that may route traffic to CXL. So when the memory-side-cache is in this "extended" mode there are more addresses that may route to CXL. Again, whether the address mode is extended-linear, or "non-extended"-linear the math to find aliases is the same. Rather, Linux needs this indication to break its assumptions around which system-physical-address ranges may decode to CXL, and avoid misinterpretations of ACPI SRAT/HMAT and CEDT.CFMWS. [..] > > But it is relevant. If the near memory (cache memory) is 64GB and the far memory > > (backing store) is 64GB then the SRAT range is 64GB (cache-excluded). With this > > new mode the SRAT range is 128GB. > > Sure, but this is a cache, not normal memory so in what people normally expect > from a memory-side cache (write through / write back etc) there is no > reason to ever include it in SRAT. Hence it's not 'excluded' it just has > nothing to do with SRAT which is about memory, not caches. I'm not > arguing it is irrelevant in your case (where it clearly is because it is part > of the memory), but it is irrelevant in for example a write-through cache. > Saying it was excluded is implying a lot more that 'not-included' would for > example. Ok, that's starting to get through, and it seems to support the proposal to call this address-mode "extended-linear". [..] > > It was an attempt to show precedent for why Linux needs to care about the memory > > organization and how CFMWS does not achieve this description. That said, as this > > is text that only appears in the justification for the ECN I do not mind > > dropping it. > > I think it risks confusion given it's not directly relevant to this. ...to be deleted for the next rev. > > > > Address translation needs to consider that the decode for an error may > > > > impact multiple components (FRUs fields replaceable units). > > > > > > > > Now consider the implications of ["Flat Memory Mode" (Intel presentation > > > > at Hot Chips > > > > 2023)](https://cdrdv2-public.intel.com/787386/Hot%20Chips%20-%20Aug%2023%20-%20BHS%20and%20Granite%20Rapid%20-%20Xeon%20-%20Architecture%20-%20Public.pdf). > > > > > > Other than telling us someone put it on a slide, that slide provides > > > very little useful info! > > > > Hence this write-up in the ECN, felt it was better than nothing to include a > > picture for reference. > > I think that left me more confused than anything ;) ...to be deleted for the next rev. [..] > > Sure, "Linear" implies direct-mapped since fully-set associative is a > > non-linear arrangement. > > Sure, but you haven't introduced linear yet so this reads as more general > than intended. I'd call out explicitly that if your new mode is set then > direct-mapped must also be set. Will do. [..] > > I still disagree with the implication that "inclusion" is a property of the > > cache and not the address layout for this ECN. > > It's an ECN about caches - the chance of misunderstanding is high. > Maybe there isn't a better option, but it definitely makes me feel uncomfortable. [..] > Maybe hyphen will help? Inclusive-linear Address mode? > to avoid reading this as separate adjectives as in that this is an > 'inclusive' cache that has a 'linear address' mode? Try this on for size: * "When Address Mode is 1 'Extended-Linear' it indicates that the associated address range (SRAT.MemoryAffinityStructure.Length) is comprised of the backing store capacity extended by the cache capacity. It is arranged such that there are N directly addressable aliases of a given cacheline where N is the ratio of target memory proximity domain size and the memory side cache size. Where the N aliased addresses for a given cacheline all share the same result for the operation 'address modulo cache size'. This setting is only allowed when 'Cache Associativity' is 'Direct Map'." [..]
