On Mon 26-08-19 21:27:38, Yang Shi wrote: > > > On 8/26/19 12:43 AM, Michal Hocko wrote: > > On Thu 22-08-19 08:33:40, Yang Shi wrote: > > > > > > On 8/22/19 1:04 AM, Michal Hocko wrote: > > > > On Thu 22-08-19 01:55:25, Yang Shi wrote: > > [...] > > > > > And, they seems very common with the common workloads when THP is > > > > > enabled. A simple run with MariaDB test of mmtest with THP enabled as > > > > > always shows it could generate over fifteen thousand deferred split THPs > > > > > (accumulated around 30G in one hour run, 75% of 40G memory for my VM). > > > > > It looks worth accounting in MemAvailable. > > > > OK, this makes sense. But your above numbers are really worrying. > > > > Accumulating such a large amount of pages that are likely not going to > > > > be used is really bad. They are essentially blocking any higher order > > > > allocations and also push the system towards more memory pressure. > > > That is accumulated number, during the running of the test, some of them > > > were freed by shrinker already. IOW, it should not reach that much at any > > > given time. > > Then the above description is highly misleading. What is the actual > > number of lingering THPs that wait for the memory pressure in the peak? > > By rerunning sysbench mariadb test of mmtest, I didn't see too many THPs in > the peak. I saw around 2K THPs sometimes on my VM with 40G memory. But they > were short-lived (should be freed when the test exit). And, the number of > accumulated THPs are variable. > > And, this reminded me to go back double check our internal bug report which > lead to the "make deferred split shrinker memcg aware" patchset. > > In that case, a mysql instance with real production load was running in a > memcg with ~86G limit, the number of deferred split THPs may reach to ~68G > (~34K deferred split THPs) in a few hours. The deferred split THP shrinker > was not invoked since global memory pressure is still fine since the host > has 256G memory, but memcg limit reclaim was triggered. > > And, I can't tell if all those deferred split THPs came from mysql or not > since there were some other processes run in that container too according to > the oom log. > > I will update the commit log with the more solid data from production > environment. This is a very useful information. Thanks! > > > > IIUC deferred splitting is mostly a workaround for nasty locking issues > > > > during splitting, right? This is not really an optimization to cache > > > > THPs for reuse or something like that. What is the reason this is not > > > > done from a worker context? At least THPs which would be freed > > > > completely sound like a good candidate for kworker tear down, no? > > > Yes, deferred split THP was introduced to avoid locking issues according to > > > the document. Memcg awareness would help to trigger the shrinker more often. > > > > > > I think it could be done in a worker context, but when to trigger to worker > > > is a subtle problem. > > Why? What is the problem to trigger it after unmap of a batch worth of > > THPs? > > This leads to another question, how many THPs are "a batch of worth"? Some arbitrary reasonable number. Few dozens of THPs waiting for split are no big deal. Going into GB as you pointed out above is definitely a problem. -- Michal Hocko SUSE Labs
