On 10.03.23 19:28, Stefan Roesch wrote:
So far KSM can only be enabled by calling madvise for memory regions. To be able to use KSM for more workloads, KSM needs to have the ability to be enabled / disabled at the process / cgroup level. Use case 1: The madvise call is not available in the programming language. An example for this are programs with forked workloads using a garbage collected language without pointers. In such a language madvise cannot be made available. In addition the addresses of objects get moved around as they are garbage collected. KSM sharing needs to be enabled "from the outside" for these type of workloads. Use case 2: The same interpreter can also be used for workloads where KSM brings no benefit or even has overhead. We'd like to be able to enable KSM on a workload by workload basis. Use case 3: With the madvise call sharing opportunities are only enabled for the current process: it is a workload-local decision. A considerable number of sharing opportuniites may exist across multiple workloads or jobs. Only a higler level entity like a job scheduler or container can know for certain if its running one or more instances of a job. That job scheduler however doesn't have the necessary internal worklaod knowledge to make targeted madvise calls. Security concerns: In previous discussions security concerns have been brought up. The problem is that an individual workload does not have the knowledge about what else is running on a machine. Therefore it has to be very conservative in what memory areas can be shared or not. However, if the system is dedicated to running multiple jobs within the same security domain, its the job scheduler that has the knowledge that sharing can be safely enabled and is even desirable. Performance: Experiments with using UKSM have shown a capacity increase of around 20%.
Stefan, can you do me a favor and investigate which pages we end up deduplicating -- especially if it's mostly only the zeropage and if it's still that significant when disabling THP?
I'm currently investigating with some engineers on playing with enabling KSM on some selected processes (enabling it blindly on all VMAs of that process via madvise() ).
One thing we noticed is that such (~50 times) 20MiB processes end up saving ~2MiB of memory per process. That made me suspicious, because it's the THP size.
What I think happens is that we have a 2 MiB area (stack?) and only touch a single page. We get a whole 2 MiB THP populated. Most of that THP is zeroes.
KSM somehow ends up splitting that THP and deduplicates all resulting zeropages. Thus, we "save" 2 MiB. Actually, it's more like we no longer "waste" 2 MiB. I think the processes with KSM have less (none) THP than the processes with THP enabled, but I only took a look at a sample of the process' smaps so far.
I recall that there was a proposal to split underutilized THP and free up the zeropages (IIRC Rik was involved).
I also recall that Mike reported memory waste due to THP. -- Thanks, David / dhildenb
