On Tue, Apr 02, 2013 at 11:54:03AM +0200, Cyril Hrubis wrote: > This patch fixes corner case for MAP_FIXED when requested mapping length > is larger than rlimit for virtual memory. In such case any overlapping > mappings are unmapped before we check for the limit and return ENOMEM. > > The check is moved before the loop that unmaps overlapping parts of > existing mappings. When we are about to hit the limit (currently mapped > pages + len > limit) we scan for overlapping pages and check again > accounting for them. > > This fixes situation when userspace program expects that the previous > mappings are preserved after the mmap() syscall has returned with error. > (POSIX clearly states that successfull mapping shall replace any > previous mappings.) > > This corner case was found and can be tested with LTP testcase: > > testcases/open_posix_testsuite/conformance/interfaces/mmap/24-2.c > > In this case the mmap, which is clearly over current limit, unmaps > dynamic libraries and the testcase segfaults right after returning into > userspace. > > I've also looked at the second instance of the unmapping loop in the > do_brk(). The do_brk() is called from brk() syscall and from vm_brk(). > The brk() syscall checks for overlapping mappings and bails out when > there are any (so it can't be triggered from the brk syscall). The > vm_brk() is called only from binmft handlers so it shouldn't be > triggered unless binmft handler created overlapping mappings. > > Signed-off-by: Cyril Hrubis <chrubis@xxxxxxx> Reviewed-by: Mel Gorman <mgorman@xxxxxxx> -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>