I ran some test and it seems that the first time accessing memory there is about 2 minor faults per MB of data? That seems very high to me. Is there a way I can allocate more data per fault to reduce the overall fault amount? I wrote code that I can't seem to include. It reports taking 48ms to fill the array in the first loop, 31ms the second with 770 faults in the program. That's 17ms difference between initial data fill and the second fill. There seems to be a potential to save time? The real code I'm having issues with uses about 80-100MiB per thread, 6 threads (one per core, any more than that it'll get slower). The real code suffers from cache misses and perf seems to suggest my hot loop spends 1/3rd of the time in my code and the other 2/3rds in the kernel. I'd like to reduce minor-faults if possible. Using MAP_POPULATE doesn't decrease my overall time unforunately. The test code I couldn't attach takes in an argument to use multiple threads and only measure how it takes to write 1 int every 4K. The thought is I'm causing a fault every page with the least amount of work in my code as possible and execution time should be roughly how long it takes the kernel to ready up my data. Since it took 31ms to fill ram I was hoping to see <=34ms (meaning 3ms of overhead), I got 38. However overhead isn't my real problem, faults and cache misses are. Is there anything I can do to reduce the overall faults? I can increase a file pipe size with fcntl, perhaps there a setting I can change to do the same for memory in a fault? Just to reiterate, there seems to be 2 faults per MB in single threaded code and 1 or more per MB in multithreaded code. I think that's fairly high and measuring shows my hotloop spends most of the time in the kernel
