On 04/27/2012 02:10 PM, Konstantin Svist wrote:
Hi all, I have a strange recurring problem on some of my servers, maybe someone can help me figure out what might be causing it. After running a MySQL machine with fairly high load for a while (~month), RAM usage in top stops making sense. Normally, RES column accounts for everything currently present in RAM (not swap) and corresponds pretty well with mem_used-buffers-cache. This is always the case after a fresh boot and seems to be the case on most other servers. But that's not the case here: 59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G) supposedly used by all processes sum(RES column from top) ~= 35127296k (33.5G) So where's the other 21G?? I'm pretty sure I'm not nitpicking here, this is >30% of total system RAM unaccounted for. I've tried stopping all non-OS specific processes (and restarting some services that seemed to eat more RAM that they should have (irqbalance)) -- and memory is not reclaimed. Over the few years that I've seen this problem, I've already replaced all the hardware and upgraded Fedora from 8 to 14 (currently running 2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of improvement. Interestingly, another machine with same hardware & OS runs MySQL in slave mode to replicate the DB -- that machine has uptime of 134 days and does not exhibit the same symptoms. In fact, here is its mem footprint: 63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G) sum(RES column from top) ~= 45.5G (which makes sense since a few RAM-hungry processes share memory)
We've seen this before in some of our MySQL platforms. It appears that some of the older MySQL servers hemorrhage shared memory segments which will not be reclaimed when the process is terminated. You might try having a look at the output of ipcs after stopping MySQL and see if your missing memory is in one or more of the shm segments. If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be surprised at how many programs don't release IPC resources--especially if they are rudely terminated (e.g. SIGSEGV or SIGKILL). Also have a serious look at updating your MySQL platform if possible. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@xxxxxxxxxxxxxx - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Try to look unimportant. The bad guys may be low on ammo. - ---------------------------------------------------------------------- -- users mailing list users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
