Hi, Jonathan,
Sorry for late reply.
Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx> writes:
> On Fri, 31 Jan 2025 12:28:03 +0000
> Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx> wrote:
>
>> > Here is the list of potential discussion points:
>> ...
>>
>> > 2. Possibility of maintaining single source of truth for page hotness that would
>> > maintain hot page information from multiple sources and let other sub-systems
>> > use that info.
>> Hi,
>>
>> I was thinking of proposing a separate topic on a single source of hotness,
>> but this question covers it so I'll add some thoughts here instead.
>> I think we are very early, but sharing some experience and thoughts in a
>> session may be useful.
>
> Thinking more on this over lunch, I think it is worth calling this out as a
> potential session topic in it's own right rather than trying to find
> time within other sessions. Hence the title change.
>
> I think a session would start with a brief listing of the temperature sources
> we have and those on the horizon to motivate what we are unifying, then
> discussion to focus on need for such a unification + requirements
> (maybe with a straw man).
>
>>
>> What do the other subsystems that want to use a single source of page hotness
>> want to be able to find out? (subject to filters like memory range, process etc)
>>
>> A) How hot is page X?
>> - Is this useful, or too much data? What would use it?
>> * Application optimization maybe. Very handy for developing algorithms
>> to do the rest of the options here as an Oracle!
>> - Provides both the cold and hot end of the scale, but maybe measurement
>> techniques vary and can not be easily combined. Hard in general to combine
>> multiple sources of truth if aiming for an absolute number.
>>
>> B) Which pages are super hot?
>> - Probably these that make the most difference if they are in a slower memory tier.
>>
>> C) Some pages are hot enough to consider moving?
>> - This may be good enough to get the key data into the fast memory over time.
>> - Can combine sources of info as being able to compare precise numbers doesn't matter.
>>
>> D) Which pages are fairly cold?
>> - Likewise maybe good enough over time.
>>
>> E) Which pages are very cold?
>> - Ideal case for tiering. Swap these with the super hot ones.
>> - Maybe extra signal for swap / zswap etc
>>
>> F) Did these hot pages remain hot (and same for cold)
>> - This is needed to know when to back off doing things as we have unstable
>> hotness (two phase applications are a pain for this), sampling a few
>> pages may be fine.
>>
>> Messy corners:
>>
>> Temporal aspects.
>> - If only providing lists of hottest / coldest in last second, very hard
>> to find those that are of a stable temperature. We end up moving
>> very hot data (which is disruptive) and it doesn't stay hot.
>> - Can reduce that affect by long sampling windows on some measurement approaches
>> (on hardware trackers that can trash accuracy due to resource exhaustion
>> and other subtle effects).
>> - bistable / phase based applications are a pain but perhaps up to higher
>> levels to back off.
>>
>> My main interest is migrating in tiered systems but good to look at what
>> else would use a common layer.
>>
>> Mostly I want to know something that is useful to move, and assume convergence
>> over the long term with the best things to move so to me the ideal layer has
>> following interface (strawman so shoot holes in it!):
>>
>> 1) Give me up to X hotish pages from a slow tier (greater than a specific measure
>> of temperature)
Because the hot pages may be available upon page accessing (such PROT_NONE
page fault), the interface may be "push" style instead of "pull" style,
e.g.,
int register_hot_page_handler(void (*handler)(struct page *hot_page, int temperature));
>> 2) Give me X coldish pages a faster tier.
>> 3) I expect to ask again in X seconds so please have some info ready for me!
>> 4) (a path to get an idea of 'unhelpful moves' from earlier iterations - this
>> is bleeding the tiering application into a shared interface though).
In addition to get a list hot/cold pages, it's also useful to get
hot/cold statistics of a memory device (NUMA node), e.g., something like
below,
Access frequency percent
> 1000 HZ 10%
600-1000 HZ 20%
200- 600 HZ 50%
1- 200 HZ 15%
< 1 HZ 5%
Compared with hot/cold pages list, this may be gotten with lower
overhead and can be useful to tune the promotion/demotion alrogithm. At
the same time, a sampled (incomplete) list of hot/cold page list may be
available too.
>> If we have multiple subsystems using the data we will need to resolve their
>> conflicting demands to generate good enough data with appropriate overhead.
>>
>> I'd also like a virtualized solution for case of hardware PA trackers (what
>> I have with CXL Hotness Monitoring Units) and classic memory pool / stranding
>> avoidance case where the VM is the right entity to make migration decisions.
>> Making that interface convey what the kernel is going to use would be an
>> efficient option. I'd like to hide how the sausage was made from the VM.
---
Best Regards,
Huang, Ying
