On 03/31/2014 10:26 AM, Davidlohr Bueso wrote: > On Mon, 2014-03-31 at 09:27 -0700, Dave Hansen wrote: >> On 12/17/2013 10:53 PM, Joonsoo Kim wrote: >>> * NOTE for v3 >>> - Updating patchset is so late because of other works, not issue from >>> this patchset. >> >> I've got some folks with a couple TB of RAM seeing long startup times >> with $LARGE_DATABASE_PRODUCT. It looks to be contention on >> hugetlb_instantiation_mutex because everyone is trying to zero hugepages >> under that lock in parallel. Just removing the lock sped things up >> quite a bit. > > Welcome to my world. Regarding the instantiation mutex, it is addressed, > see commit c999c05ff595 in -next. Cool stuff. That does seem to fix my parallel-fault hugetlbfs microbenchmark. I'll recommend that the $DATABASE folks check it as well. > As for the clear page overhead, I brought this up in lsfmm last week, > proposing some daemon to clear pages when we have idle cpu... but didn't > get much positive feedback. Basically (i) not worth the additional > complexity and (ii) can trigger different application startup times, > which seems to be something negative. I do have a patch that implements > huge_clear_page with non-temporal hinting but I didn't see much > difference on my environment, would you want to give it a try? I'd just be happy to see it happen outside of the locks. As it stands now, I have 1 CPU zeroing a huge page, and 159 sitting there sleeping waiting for it to release the hugetlb_instantiation_mutex. That's just nonsense. I don't think making them non-temporal will fundamentally help that. We need them parallelized. According to ftrace, a hugetlb_fault() takes ~700us. Literally 99% of that is zeroing the page. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>