On Sun, May 26, 2013 at 7:23 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote: > On Fri, May 24, 2013 at 11:12:50PM -0400, Keyur Govande wrote: >> On Mon, May 20, 2013 at 6:53 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote: >> > On Sun, May 19, 2013 at 11:50:55PM -0400, Keyur Govande wrote: >> >> Hello, >> >> >> >> We have a bunch of servers that create a lot of temp files, or check >> >> for the existence of non-existent files. Every such operation creates >> >> a dentry object and soon most of the free memory is consumed for >> >> 'negative' dentry entries. This behavior was observed on both CentOS >> >> kernel v.2.6.32-358 and Amazon Linux kernel v.3.4.43-4. >> >> >> >> There are also some processes running that occasionally allocate large >> >> chunks of memory, and when this happens the kernel clears out a bunch >> >> of stale dentry caches. This clearing takes some time. kswapd kicks >> >> in, and allocations and bzero() of 4GB that normally takes <1s, takes >> >> 20s or more. >> >> >> >> Because the memory needs are non-continuous but negative dentry >> >> generation is fairly continuous, vfs_cache_pressure doesn't help much. >> >> >> >> The thought I had was to have a sysctl that limits the number of >> >> dentries per super-block (sb-max-dentry). Everytime a new dentry is >> >> allocated in d_alloc(), check if dentry_stat.nr_dentry exceeds (number >> >> of super blocks * sb-max-dentry). If yes, queue up an asynchronous >> >> workqueue call to prune_dcache(). Also have a separate sysctl to >> >> indicate by what percentage to reduce the dentry entries when this >> >> happens. >> > >> > This request does come up every so often. There are valid reasons >> > for being able to control the exact size of the dentry and page >> > caches - I've seen a few implementations in storage appliance >> > vendor kernels where total control of memory usage yields a few >> > percent better performance of industry specific benchmarks. Indeed, >> > years ago I thought that capping the size of the dnetry cache was a >> > good idea, too. >> > >> > However, the problem that I've seen with every single on of these >> > implementations is that the limit is carefully tuned for best all >> > round performance in a given set of canned workloads. When the limit >> > is wrong, performance tanks, and it is just about impossible to set >> > a limit correctly for a machine that has a changing workload. >> > >> > If your problem is negative dentries building up, where do you set >> > the limit? Set it low enough to keep only a small number of total >> > dentries to keep the negative dentries down, and you'll end up >> > with a dentry cache that isn't big enough to hold all th dentries >> > needed for efficient performance with workloads that do directory >> > traversals. It's a two-edged sword, and most people do not have >> > enough knowledge to tune a knob correctly. >> > >> > IOWs, the automatic sizing of the dentry cache based on memory >> > pressure is the correct thing to do. Capping it, or allowing it to >> > be capped will simply generate bug reports for strange performance >> > problems.... >> > >> > That said, keeping lots of negative dentries around until memory >> > pressure kicks them out is probably the wrong thing to do. Negative >> > dentries are an optimisation for some workloads, but they tend to >> > have references to negative dentries with a temporal locality that >> > matches the unlink time. >> > >> > Perhaps we need to separately reclaim negative dentries i.e. not >> > wait for memory pressure to reclaim them but use some other kind of >> > trigger for reclamation. That doesn't cap the size of the dentry >> > cache, but would address the problem of negative dentry buildup.... >> > >> > Cheers, >> > >> > Dave. >> > -- >> > Dave Chinner >> > david@xxxxxxxxxxxxx >> >> Hi Dave, >> >> Thank you for responding. Sorry it took so long for me to get back, >> been a bit busy. >> >> I do agree that having a knob, and then setting a bad value can tank >> performance. But not having a knob IMO is worse. Currently there are >> no options for controlling the cache, bar dropping the caches >> altogether every so often. The knob would have a default value of >> ((unsigned long) -1)), so if one does not care for it, they would >> experience the same behavior as today. > > And therein lies the problem with a knob. What's the point of having > a knob that nobody but a handful of people know what it does or > evenhow to recognise when they need to tweak it. It's long been a > linux kernel policy that the kernel should do the right thing by > default. As such, knobs to tweak things are a last resort. > >> Also, setting a bad value for the knob would negatively impact file-IO >> performance, which on a spinning disk isn't guaranteed anyway. The >> current situation tanks memory performance which is more unexpected to >> a normal user. > > Which is precisely why a knob is the wrong solution. If it's > something a normal, unsuspecting user has problems with, then it > needs to be handled automatically by the kernel. Expecting users who > don't even know what a dentry is to know about a magic knob that > fixes a problem they don't even know they have is not an acceptable > solution. > > The first step to solving such a problem is to provide a > reproducable, measurable test case in a simple script that > demonstrates the problem that needs solving. If we can reproduce it > at will, then half the battle is already won.... > Here's a simple test case: https://gist.github.com/keyurdg/5660719 to create a ton of dentry cache entries, and https://gist.github.com/keyurdg/5660723 to allocate some memory. I kicked off 3 instances of fopen in 3 different prefixed directories. After all the memory was filled up with dentry entries, I tried allocating 4GB of memory. It took ~20s. If I turned off the dentry generation programs and attempted to allocate 4GB again, it only took 2s (because the memory was already free). Here's a quick graph of this behavior: http://i.imgur.com/XhgX84d.png I understand that in general, the kernel should do "the right thing" without user input. But this seems to be a case where the user should be allowed input into how memory is used. After all, there are already lots of knobs in Linux that if set wrongly can cause user pain/bad performance. IMO this new knob needs the right kind of documentation, like suggesting the use of slabtop and perf to identify dentry as an issue before setting the knob. I'm also not tied to the idea of the knob being a limit on the number of dentry cache entries. A limit just seems easiest to administer; but if there are other ways of alleviating this issue, then I'd love to explore those as well. > Cheers, > > Dave. > -- > Dave Chinner > david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
