On Wed, Feb 13, 2013 at 07:14:50AM -0800, Dave Hansen wrote: > On 02/12/2013 08:25 PM, Jonathan Woithe wrote: > > I will see whether I can gain access to a test system and if so, try a more > > recent kernel to see if it makes any difference. > > > > I'll advise which of these options proves practical as soon as possible and > > report any findings which come out of them. > > Are there any non-upstream bits in the kernel? Any third-party drivers > or filesystems? No to all three questions. The kernel is a plain unpatched kernel.org 2.6.35.11 kernel compiled with the configuration I included in the original email. No third party modules have been loaded. I should add that I have managed to get access to a test system and over the next few days I will run tests on a number of kernels to try to narrow down some of the unknowns associated with this problem. > David's analysis looks spot-on. The only other thing I'll add is that > it just looks weird that all three kmalloc() caches are so _even_: > > >> kmalloc-128 1234556 1235168 128 32 1 : tunables 0 0 0 : slabdata 38599 38599 0 > >> kmalloc-64 1238117 1238144 64 64 1 : tunables 0 0 0 : slabdata 19346 19346 0 > >> kmalloc-32 1236600 1236608 32 128 1 : tunables 0 0 0 : slabdata 9661 9661 0 > > It's almost like something goes and does 3 allocations in series and > leaks them all. > > There are also quite a few buffer_heads: > > > buffer_head 496273 640794 56 73 1 : tunables 0 0 0 : slabdata 8778 8778 0 > > which seem out-of-whack for the small amount of memory being used for > I/O-related stuff. That kinda points in the direction of I/O or > filesystems. As previously stated, there is a lot of network I/O going on (input to the tune of 20 MBytes/s) but I don't know if this is the I/O class you're referring to. We are also writing data to disc periodically, but since this is post-process the amount is *much* less than the raw input rate (for the data acquisition system we're talking of the order of 5 MBytes per minute or less). Regards jonathan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>