FYI everyone, I found a bug within openssh-server which caused this problem. I've patched and submitted to the openssh list. You can find details of this by googling for:"port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions"
It's extremely strange.. :S Cal On 30/05/2011 18:36, Cal Leeming [Simplicity Media Ltd] wrote:
Here is an strace of the SSH process (which is somehow inheriting the -17 oom_adj on all forked user instances)(broken server - with bnx2 module loaded)[pid 2200] [ 7f13a09c9cb0] open("/proc/self/oom_adj", O_WRONLY|O_CREAT|O_TRUNC, 0666 <unfinished ...> [pid 2120] [ 7f13a09c9f00] write(7, "\0\0\2\240\n\n\n\nPort 22\n\n\n\nProtocol 2\n\nH"..., 680 <unfinished ...>[pid 2200] [ 7f13a09c9cb0] <... open resumed> ) = 9 [pid 2120] [ 7f13a09c9f00] <... write resumed> ) = 680 [pid 2120] [ 7f13a09c9e40] close(7 <unfinished ...> [pid 2200] [ 7f13a09c9844] fstat(9, <unfinished ...> [pid 2120] [ 7f13a09c9e40] <... close resumed> ) = 0[pid 2200] [ 7f13a09c9844] <... fstat resumed> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0[pid 2120] [ 7f13a09c9e40] close(8 <unfinished ...>[pid 2200] [ 7f13a09d2a2a] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>[pid 2120] [ 7f13a09c9e40] <... close resumed> ) = 0 [pid 2200] [ 7f13a09d2a2a] <... mmap resumed> ) = 0x7f13a25a6000 [pid 2120] [ 7f13a09c9e40] close(4 <unfinished ...> [pid 2200] [ 7f13a09c9f00] write(9, "-17\n", 4 <unfinished ...> (working server - with bnx2 module unloaded) [pid 1323] [ 7fae577fbe40] close(7) = 0[pid 1631] [ 7fae577fbcb0] open("/proc/self/oom_adj", O_WRONLY|O_CREAT|O_TRUNC, 0666 <unfinished ...> [pid 1323] [ 7fae577fbf00] write(8, "\0\0\2\217\0", 5 <unfinished ...>[pid 1631] [ 7fae577fbcb0] <... open resumed> ) = 10 [pid 1323] [ 7fae577fbf00] <... write resumed> ) = 5[pid 1323] [ 7fae577fbf00] write(8, "\0\0\2\206\n\n\n\nPort 22\n\n\n\nProtocol 2\n\nH"..., 654 <unfinished ...>[pid 1631] [ 7fae577fb844] fstat(10, <unfinished ...> [pid 1323] [ 7fae577fbf00] <... write resumed> ) = 654[pid 1631] [ 7fae577fb844] <... fstat resumed> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0[pid 1323] [ 7fae577fbe40] close(8) = 0[pid 1631] [ 7fae57804a2a] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>[pid 1323] [ 7fae577fbe40] close(9 <unfinished ...> [pid 1631] [ 7fae57804a2a] <... mmap resumed> ) = 0x7fae593d9000 [pid 1323] [ 7fae577fbe40] <... close resumed> ) = 0 [pid 1323] [ 7fae577fbe40] close(5 <unfinished ...> [pid 1631] [ 7fae577fbf00] write(10, "0\n", 2 <unfinished ...>The two servers are *EXACT* duplicates of each other, completely fresh Debian installs, with exactly the same packages installed.As you can see, the working server sends "0" into the oom_adj and the broken one sends "-17".On 30/05/2011 15:27, Cal Leeming [Simplicity Media Ltd] wrote:I FOUND THE PROBLEM!!!Explicit details can be found on the Debian kernel mailing list, but to cut short, it's caused by the firmware-bnx2 kernel module:The broken server uses 'firmware-bnx2'.. so I purged the bnx2 package, removed the bnx*.ko files from /lib/modules, ran update-initramfs, and then rebooted (i then confirmed it was removed by checking ifconfig and lsmod).And guess what.. IT WORKED.So, this problem seems to be caused by the firmware-bnx2 module being loaded.. some how, that module is causing -17 oom_adj to be set for everything..WTF?!?! Surely a bug?? Could someone please forward this to the appropriate person for the bnx2 kernel module, as I wouldn't even know where to begin :SCal On 30/05/2011 11:52, Cal Leeming [Simplicity Media Ltd] wrote:-resent due to incorrect formatting, sorry if this dups! @Kame Thanks for the reply!Both kernels used the same env/dist, but which slightly different packages.After many frustrating hours, I have pin pointed this down to a dodgy Debian package which appears to continue affecting the system, even after purging. I'm still yet to pin point the package down (I'm doing several reinstall tests, along with tripwire analysis after each reboot). @Hiroyuki Thank you for sending this to the right people! @linux-mm On a side note, would someone mind taking a few minutes to give a brief explanation as to how the default oom_adj is set, and under what conditions it is given -17 by default? Is this defined by the application? I looked through the kernel source, and noticed some of the code was defaulted to set oom_adj to OOM_DISABLE (which is defined in the headers as -17). Assuming the debian problem is resolved, this might be another call for the oom-killer to be modified so that if it encounters the unrecoverable loop, it ignores the -17 rule (with some exceptions, such as kernel processes, and other critical things). If this is going to be a relatively simple task, I wouldn't mind spending a few hours patching this? Cal On Mon, May 30, 2011 at 3:23 AM, KAMEZAWA Hiroyuki <kamezawa.hiroyu@xxxxxxxxxxxxxx> wrote:Thank you. memory cgroup and OOM troubles are handled in linux-mm. On Sun, 29 May 2011 23:24:07 +0100"Cal Leeming [Simplicity Media Ltd]"<cal.leeming@xxxxxxxxxxxxxxxxxxxxxxxx> wrote:hmm, in short, applications has -17 oom_adj in default with 2.6.32.41 ?Some further logs:./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.369927] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.369939] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.399285] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.399296] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.428690] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.428702] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.487696] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.487708] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.517023] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.517035] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.546379] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.546391] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.310789] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.310804] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.369918] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.369930] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.399284] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.399296] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.433634] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.433648] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.463947] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.463959] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.493439] redis-serverinvoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.493451] [<ffffffff810b12b7>] ? oom_kill_process+0x82/0x283AFAIK, no kernel has such crazy settings as default.. Does your 2 kernel uses the same environment/distribution ? Thanks, -KameOn 29/05/2011 22:50, Cal Leeming [Simplicity Media Ltd] wrote:First of all, my apologies if I have submitted this problem to the wrong place, spent 20 minutes trying to figure out where it needs to be sent, and was still none the wiser.The problem is related to applying memory limitations within a cgroup. If the OOM killer kicks in, it gets stuck in a loop where it tries tokill a process which has an oom_adj of -17. This causes an infinite loop, which in turn locks up the system. May 30 03:13:08 vicky kernel: [ 1578.117055] Memory cgroup out of memory: kill process 6016 (java) score 0 or a child May 30 03:13:08 vicky kernel: [ 1578.117154] Memory cgroup out of memory: kill process 6016 (java) score 0 or a child May 30 03:13:08 vicky kernel: [ 1578.117248] Memory cgroup out of memory: kill process 6016 (java) score 0 or a child May 30 03:13:08 vicky kernel: [ 1578.117343] Memory cgroup out of memory: kill process 6016 (java) score 0 or a child May 30 03:13:08 vicky kernel: [ 1578.117441] Memory cgroup out of memory: kill process 6016 (java) score 0 or a child root@vicky [/home/foxx]> uname -aLinux vicky 2.6.32.41-grsec #3 SMP Mon May 30 02:34:43 BST 2011 x86_64GNU/Linux(this happens on both the grsec patched and non patched 2.6.32.41 kernel)When this is encountered, the memory usage across the whole server isstill within limits (not even hitting swap). The memory configuration for the cgroup/lxc is: lxc.cgroup.memory.limit_in_bytes = 3000M lxc.cgroup.memory.memsw.limit_in_bytes = 3128M Now, what is even more strange, is that when running under the 2.6.32.28 kernel (both patched and unpatched), this problem doesn'thappen. However, there is a slight difference between the two kernels.The 2.6.32.28 kernel gives a default of 0 in the /proc/X/oom_adj, where as the 2.6.32.41 gives a default of -17. I suspect this is theroot cause of why it's showing in the later kernel, but not the earlier.To test this theory, I started up the lxc on both servers, and thenran a one liner which showed me all the processes with an oom_adj of -17:(the below is the older/working kernel) root@xxxxxxxxxxxxxxxxx [/mnt/encstore/lxc]> uname -aLinux courtney.internal 2.6.32.28-grsec #3 SMP Fri Feb 18 16:09:07 GMT2011 x86_64 GNU/Linux root@xxxxxxxxxxxxxxxxx [/mnt/encstore/lxc]> for x in `find /proc-iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print $3}'` ; dops -p $x --no-headers ; done grep: /proc/1411/task/1411/oom_adj: No such file or directory grep: /proc/1411/oom_adj: No such file or directory 804 ? 00:00:00 udevd 804 ? 00:00:00 udevd 25536 ? 00:00:00 sshd 25536 ? 00:00:00 sshd 31861 ? 00:00:00 sshd 31861 ? 00:00:00 sshd 32173 ? 00:00:00 udevd 32173 ? 00:00:00 udevd 32174 ? 00:00:00 udevd 32174 ? 00:00:00 udevd (the below is the newer/broken kernel) root@vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41]> uname -aLinux vicky 2.6.32.41-grsec #3 SMP Mon May 30 02:34:43 BST 2011 x86_64GNU/Linux root@vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41]> for x in`find /proc -iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print$3}'` ; do ps -p $x --no-headers ; done grep: /proc/3118/task/3118/oom_adj: No such file or directory grep: /proc/3118/oom_adj: No such file or directory 895 ? 00:00:00 udevd 895 ? 00:00:00 udevd 1091 ? 00:00:00 udevd 1091 ? 00:00:00 udevd 1092 ? 00:00:00 udevd 1092 ? 00:00:00 udevd 2596 ? 00:00:00 sshd 2596 ? 00:00:00 sshd 2608 ? 00:00:00 sshd 2608 ? 00:00:00 sshd 2613 ? 00:00:00 sshd 2613 ? 00:00:00 sshd 2614 pts/0 00:00:00 bash 2614 pts/0 00:00:00 bash 2620 pts/0 00:00:00 sudo 2620 pts/0 00:00:00 sudo 2621 pts/0 00:00:00 su 2621 pts/0 00:00:00 su 2622 pts/0 00:00:00 bash 2622 pts/0 00:00:00 bash 2685 ? 00:00:00 lxc-start 2685 ? 00:00:00 lxc-start 2699 ? 00:00:00 init 2699 ? 00:00:00 init 2939 ? 00:00:00 rc 2939 ? 00:00:00 rc 2942 ? 00:00:00 startpar 2942 ? 00:00:00 startpar 2964 ? 00:00:00 rsyslogd 2964 ? 00:00:00 rsyslogd 2964 ? 00:00:00 rsyslogd 2964 ? 00:00:00 rsyslogd 2980 ? 00:00:00 startpar 2980 ? 00:00:00 startpar 2981 ? 00:00:00 ctlscript.sh 2981 ? 00:00:00 ctlscript.sh 3016 ? 00:00:00 cron 3016 ? 00:00:00 cron 3025 ? 00:00:00 mysqld_safe 3025 ? 00:00:00 mysqld_safe 3032 ? 00:00:00 sshd 3032 ? 00:00:00 sshd 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3097 ? 00:00:00 mysqld.bin 3113 ? 00:00:00 ctl.sh 3113 ? 00:00:00 ctl.sh 3115 ? 00:00:00 sleep 3115 ? 00:00:00 sleep 3116 ? 00:00:00 .memcached.bin 3116 ? 00:00:00 .memcached.bin As you can see, it is clear that the newer kernel is setting -17 by default, which in turn is causing the OOM killer loop. So I began to try and find what may have caused this problem by comparing the two sources...I checked the code for all references to 'oom_adj' and 'oom_adjust' inboth code sets, but found no obvious differences: grep -R -e oom_adjust -e oom_adj . | sort | grep -R -e oom_adjust -e oom_adj Then I checked for references to "-17" in all .c and .h files, and found a couple of matches, but only one obvious one: grep -R "\-17" . | grep -e ".c:" -e ".h:" -e "\-17" | wc -l ./include/linux/oom.h:#define OOM_DISABLE (-17) But again, a search for OOM_DISABLE came up with nothing obvious... In a last ditch attempt, I did a search for all references to 'oom' (case-insensitive) in both code bases, then compared the two:root@annabelle [~/lol/linux-2.6.32.28]> grep -i -R "oom" . | sort -n/tmp/annabelle.oom_adjroot@vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41]> grep -i -R "oom" . | sort -n> /tmp/vicky.oom_adj and this brought back (yet again) nothing obvious.. root@vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41]> md5sum ./include/linux/oom.h 2a32622f6cd38299fc2801d10a9a3ea8 ./include/linux/oom.hroot@annabelle [~/lol/linux-2.6.32.28]> md5sum ./include/linux/oom.h2a32622f6cd38299fc2801d10a9a3ea8 ./include/linux/oom.h root@vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41]> md5sum ./mm/oom_kill.c 1ef2c2bec19868d13ec66ec22033f10a ./mm/oom_kill.c root@annabelle [~/lol/linux-2.6.32.28]> md5sum ./mm/oom_kill.c 1ef2c2bec19868d13ec66ec22033f10a ./mm/oom_kill.c Could anyone please shed some light as to why the default oom_adj is set to -17 now (and where it is actually set)? From what I can tell, the fix for this issue will either be:1. Allow OOM killer to override the decision of ignoring oom_adj ==-17 if an unrecoverable loop is encountered. 2. Change the default back to 0. Again, my apologies if this bug report is slightly unorthodox, or doesn't follow usual procedure etc. I can assure you I have tried my absolute best to give all the necessary information though. Cal--To unsubscribe from this list: send the line "unsubscribe linux-kernel" inthe body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
-- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxxx For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>