If setting mem=XXXX in kernel startup fixes the problem so should cgroup with memory.limit_in_bytes. They are both in 2.6.28 ment to control the same thing. Some older kernels you have to set memory.control_type if it exists in you cgroup folders you have it set it sorry I cannot remember the setting to cover RSS(mapped) and Page Cache(unmapped) at the same time. Should be the only factor of difference from mem=. Now if its not setable the same as mem= we have a glitch in cgroup. Now virtual memory is another issue. It going out of control has a different way of handling it. My kernel 2.6.28 cgroup manages RSS (mapped) and Page Cache (unmapped). Not swap cache and virtual memory . Now this opens a interesting question tried disabling virtual memory. http://www.ss64.com/bash/ulimit.html ulimit on the process option -v. Yes value 0 for testing flat out disallow virtual memory usage. Then latter allow the application a sane amount. Basically application could be a greedy pig of an application only stops allocating when it gets told there is no more memory. -v 0 setting from ulimit is the same as not having any virtual memory. Virtual memory going insane also happens in some native Linux programs. What ever you set in ulimit applies to all applications run in that shell after that until its changed. I don't think wine messes with ulimit directly itself.
