On Mon, May 21, 2018 at 5:22 PM Matthew Wilcox <willy@xxxxxxxxxxxxx> wrote: > On Mon, May 21, 2018 at 05:00:47PM -0700, Daniel Colascione wrote: > > On Mon, May 21, 2018 at 4:32 PM Dave Hansen <dave.hansen@xxxxxxxxx> wrote: > > > I think there's still a potential dead-end here. "Deallocation" does > > > not always free resources. > > > > Sure, but the general principle applies: reserve resources when you *can* > > fail so that you don't fail where you can't fail. > Umm. OK. But you want an mmap of 4TB to succeed, right? That implies > preallocating one billion * sizeof(*vma). That's, what, dozens of > gigabytes right there? That's not what I'm proposing here. I'd hoped to make that clear in the remainder of the email to which you've replied. > I'm sympathetic to wanting to keep both vma-merging and > unmap-anything-i-mapped working, but your proposal isn't going to fix it. > You need to handle the attacker writing a program which mmaps 46 bits > of address space and then munmaps alternate pages. That program needs > to be detected and stopped. Let's look at why it's bad to mmap 46 bits of address space and munmap alternate pages. It can't be that doing so would just use too much memory: you can mmap 46 bits of address space *already* and touch each page, one by one, until the kernel gets fed up and the OOM killer kills you. So it's not because we'd allocate a lot of memory that having a huge VMA tree is bad, because we already let processes allocate globs of memory in other ways. The badness comes, AIUI, from the asymptotic behavior of the address lookup algorithm in a tree that big. One approach to dealing with this badness, the one I proposed earlier, is to prevent that giant mmap from appearing in the first place (because we'd cap vsize). If that giant mmap never appears, you can't generate a huge VMA tree by splitting it. Maybe that's not a good approach. Maybe processes really need mappings that big. If they do, then maybe the right approach is to just make 8 billion VMAs not "DoS the system". What actually goes wrong if we just let the VMA tree grow that large? So what if VMA lookup ends up taking a while --- the process with the pathological allocation pattern is paying the cost, right?
