We're getting off-topic for this group except that this is *why* we're plagued with this problem, so I'll make one more observation.
Bruno Wolff III wrote:
They would allocate
100's of GB of swap space rather than taking the time to design proper data
structures. If the program crashed every week or two, no big deal -- just
run it again. Overallocating memory is a valuable technique for such
applications.
I don't think the above applies generally. Programmers need to be aware of
the working set of CPU bound apps. If the program is constantly paging,
the performance is going to be abysmal.
You're doing planetary number crunching, so you allocate an (x,y,z) space with 10^6 points on every axis, or 10^18 points, for roughly 2^60 64-bit floating-point numbers, or 2^68 bytes. But your space is mostly empty (just like real space). Where there's a planet or star, you're actually using memory, everywhere else, the memory is never referenced. So you're using just an ordinary amount of memory, but you can access your planet's data by simply referencing a three-dimensional array in FORTRAN. Just allow infinite overcommit, let the OS do the rest, and it actually works. A lot of molecular modeling works this way too.
This is how these applications were/are actually written. I'm not defending the method, just point out that it's real and it's one of the reasons virtual swap was invented.
Craig
