|
Hi all,
Recently I found out one server generating alot of load on our storage. I did a few steps to narrow down what is the cause and what might be a solution. As first I used iostat to get what device is using most IO: Linux 2.6.18-274.7.1.el5 (hostname) 05/14/2012 avg-cpu: %user %nice %system %iowait %steal %idle 11.99 0.00 8.53 2.85 0.00 76.63 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 952.75 5.54 14090.70 42708891 108632689706 sda1 0.00 0.00 0.00 6988 1802 sda2 0.51 0.17 8.29 1322786 63885168 sda3 952.24 5.37 14082.42 41378693 108568802736 sdb 8.29 1.01 739.31 7791930 5699710024 dm-0 2.14 1.48 17.03 11382146 131278120 dm-1 0.76 0.78 5.47 6030578 42202752 dm-2 8.69 3.08 68.72 23737626 529796056 dm-3 1748.90 0.03 13991.19 226754 107865527240 dm-4 0.00 0.00 0.00 1192 216 dm-5 92.49 1.01 739.31 7790146 5699710024 Using ls -lh /dev/mapper I found out that dm-3 corresponds with the mountpoint for /tmp (note that the system is using LVM). One thing that seems strange to me is that there is a huge load of blocks written however almost none get red. Then I used lsof | grep tmp: httpd 25142 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25386 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25429 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25431 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25454 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25622 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25792 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25794 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25846 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS httpd 25849 apache DEL REG 253,3 13 /tmp/apc.Fm8SAS ... The only other is for sshd and another for vmware tools. If I try to do ls -lh /tmp/apc.Fm8SAS I get no such file or directory. Using the access time I would see if this file is accessed over and over again. Am I missing something here? Is this apache doing a fstat() call for example? Or is this file created and deleted over and over again at a very fast pace that I can't access it (this seems less likely to me). Has anyone seen such behaviour in httpd? Or does someone knows a little more of the inner workings of httpd? The name does me think that APC (sort of php module iirc) is having something to do with this: apc.ini:apc.mmap_file_mask=/tmp/apc.XXXXXX The config file seems to confirm this. Anyone has an idea how to fix this or to go further investigate? Thanks alot in advance! |
|