Linus Torvalds <torvalds@xxxxxxxx> writes: > Anyway, there's two possible solutions: > > - simply make sure that you can have that many open files. > > If it's a Linux system, just increase the value of the file > /proc/sys/fs/file-max, and you're done. Of course, if you're not the > admin of the box, you may need to ask somebody else to do it for you.. > > - we could try to make git not keep them mmap'ed for the whole time. > > Junio? This is your speciality, I'm not sure how painful it would be to > unmap and remap on demand.. (or switch it to some kind of "keep the last > <n> mmaps active" kind of thing to avoid having thousands and thousands of > mmaps active). 60,000 files 1kB each is 60MB which is a peanuts these days, but 10kB each would be already nontrivial burden on 32-bit (20% under 3+1 split), so even if we do the "read in small files instead of mapping" we would need diff_unpopulate_filespec() calls. I think after diffcore_rename() runs, the data in filespec is used only once during final textual diff generation. We would use once more before diff generation if diffcore_pickaxe() is in use. These codepaths begin with diff_populate_filespec(), so if we unpopulate them after diffcore_rename() runs nobody would notice (other than performance degradation and strace showing us reading the same thing twice). The diffcore_rename() matrix code expects all filespecs involved can be populated at the same time, but it should not be too hard to change it to keep one dst and all src candidates populated but others dropped if space gets tight. I need to look at the code for the details. But Nguyen's command line did not have -M; I think the filespecs are populated only during the text generation in that case, so the above would not help him while it would be a worthwhile change. Because there is _no_ processing that comes after textual diff generation that looks at the data, I think diff_flush_patch() after calling run_diff() can unpopulate the data from the filepair *p before returning without harming performance. I think diff_flush_stat() and diff_flush_checkdiff() would have the same issue, though. Ideally these should be able to do their processing while the main textual diff holds the data in memory for its processing but that is currently not the way the code is structured. - To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html