On 8/26/19 12:43 AM, Michal Hocko wrote:
On Thu 22-08-19 08:33:40, Yang Shi wrote:
On 8/22/19 1:04 AM, Michal Hocko wrote:
On Thu 22-08-19 01:55:25, Yang Shi wrote:
[...]
And, they seems very common with the common workloads when THP is
enabled. A simple run with MariaDB test of mmtest with THP enabled as
always shows it could generate over fifteen thousand deferred split THPs
(accumulated around 30G in one hour run, 75% of 40G memory for my VM).
It looks worth accounting in MemAvailable.
OK, this makes sense. But your above numbers are really worrying.
Accumulating such a large amount of pages that are likely not going to
be used is really bad. They are essentially blocking any higher order
allocations and also push the system towards more memory pressure.
That is accumulated number, during the running of the test, some of them
were freed by shrinker already. IOW, it should not reach that much at any
given time.
Then the above description is highly misleading. What is the actual
number of lingering THPs that wait for the memory pressure in the peak?
By rerunning sysbench mariadb test of mmtest, I didn't see too many THPs
in the peak. I saw around 2K THPs sometimes on my VM with 40G memory.
But they were short-lived (should be freed when the test exit). And, the
number of accumulated THPs are variable.
And, this reminded me to go back double check our internal bug report
which lead to the "make deferred split shrinker memcg aware" patchset.
In that case, a mysql instance with real production load was running in
a memcg with ~86G limit, the number of deferred split THPs may reach to
~68G (~34K deferred split THPs) in a few hours. The deferred split THP
shrinker was not invoked since global memory pressure is still fine
since the host has 256G memory, but memcg limit reclaim was triggered.
And, I can't tell if all those deferred split THPs came from mysql or
not since there were some other processes run in that container too
according to the oom log.
I will update the commit log with the more solid data from production
environment.
IIUC deferred splitting is mostly a workaround for nasty locking issues
during splitting, right? This is not really an optimization to cache
THPs for reuse or something like that. What is the reason this is not
done from a worker context? At least THPs which would be freed
completely sound like a good candidate for kworker tear down, no?
Yes, deferred split THP was introduced to avoid locking issues according to
the document. Memcg awareness would help to trigger the shrinker more often.
I think it could be done in a worker context, but when to trigger to worker
is a subtle problem.
Why? What is the problem to trigger it after unmap of a batch worth of
THPs?
This leads to another question, how many THPs are "a batch of worth"?
And, they may be short-lived as showed by Kirill's example, we can't
tell in advance how long the THPs life time is. We may waste cpu cycles
to do something unneeded.
