On 09/18/2017 02:20 PM, Waiman Long wrote: > > A rogue application can potentially create a large number of negative > dentries in the system consuming most of the memory available even if > memory controller is enabled to limit memory usage. This can impact > performance of other applications running on the system. > > We have customers seeing soft lockup and unresponsive system when > tearing down a container because of the large number of negative > dentries accumulated during its up time that had to be cleaned up at > exit time when the container's filesystem was unmounted. So we need > to do something about it. > > This patchset introduces changes to the dcache subsystem to limit > the number of negative dentries allowed to be created thus limiting > the amount of memory that can be consumed by negative dentries. > > Patch 1 just relocates the postion of the dentry_kill() function. > > Patch 2 tracks the number of negative dentries present in the LRU > lists and reports it in /proc/sys/fs/dentry-state. > > Patch 3 sets a limit on the number of negative dentries allowable as a > small percentage (2%) of total system memory. So the larger the system, > the more negative dentries can be allowed. Once the limit is reached, > new negative dentries will be killed after use. > > Patch 4 enables automatic pruning of least recently used negative > dentries when it is close to the limit so that we won't end up killing > recently used negative dentries. > > Patch 5 shows the number of forced negative dentry killings in > /proc/sys/fs/dentry-state. > > Patch 6 enables auto-tuneup of free pool negative dentry count to > no more than the maximum number of positive dentries ever used. > > With a 4.13 based kernel, the positive & negative dentries lookup rates > (lookups per second) after initial boot on a 36-core 50GB memory VM > with and without the patch were as follows: > > Metric w/o patch with patch > ------ --------- ---------- > Positive dentry lookup 840269 845762 > Negative dentry lookup 1903405 1962514 > Negative dentry creation 6817957 6928768 > > The last row refers to the creation rate of 1 millions negative > dentries. With 50GB of memory, 1 millions negative dentries can be > created with the patched kernel without any pruning or dentry killing. > > Ignoring some inherent noise in the test results, there wasn't any > noticeable difference in term of lookup and negative dentry creation > performance with or without this patch. > > By creating 10 millions negative dentries, however, the performance > differed. > > Metric w/o patch with patch > ------ --------- ---------- > Negative dentry creation 651663 190105 > > > For the patched kernel, the corresponding dentry-state was: > > 1608833 1590416 45 0 1579878 8286952 > > This was expected as negative dentry creation throttling with forced > dentry deletion happened in this case. > > Running the AIM7 high-systime workload on the same VM, the baseline > performance was 186770 jobs/min. By running a single-thread rogue > negative dentry creation program in the background until the patched > kernel with 2% limit started throttling, the performance was 183746 > jobs/min. On an unpatched kernel with memory almost exhausted and > memory shrinker was kicked in, the performance was 148997 jobs/min. > > So the patch does protect the system from suffering significant > performance degradation in case a negative dentry creation rogue > program is runninig in the background. > > Waiman Long (6): > fs/dcache: Relocate dentry_kill() after lock_parent() > fs/dcache: Track & report number of negative dentries > fs/dcache: Limit numbers of negative dentries > fs/dcache: Enable automatic pruning of negative dentries > fs/dcache: Track count of negative dentries forcibly killed > fs/dcache: Autotuning of negative dentry limit > > fs/dcache.c | 462 +++++++++++++++++++++++++++++++++++++++++++---- > include/linux/dcache.h | 8 +- > include/linux/list_lru.h | 1 + > mm/list_lru.c | 4 +- > 4 files changed, 439 insertions(+), 36 deletions(-) > Ping! Any comments on this v4 patchset. We have customers seeing problems because of the unlimited growth of negative dentries. I am hoping that at least part of this patchset, in some form, can be merged eventually. Thanks, Longman
