> How many times was the kernel tested with this much memory, for example > ? (never??) This is actually *extremely* relevant. Take a look at /proc/sys/vm/dirty_ratio and /proc/sys/vm/dirty_background_ratio if you have an older Linux system, or /proc/sys/vm/dirty_bytes, and /proc/sys/vm/dirty_background_bytes with a newer one. On older systems for instance, those are set to 40 and 20 respectively (recent kernels cut these in half). That's significant because ratio is the *percentage* of memory that can remain dirty before causing async, and background_ratio tells it when it should start writing in the background to avoid hitting that higher and much more disruptive number. This is another source of IO that can be completely independent of the checkpoint spikes that long plagued PostgreSQL versions prior to 8.3. With that much memory (1TB!), that's over 100GB of dirty memory before it starts writing that out to disk even with the newer conservative settings. We had to tweak and test for days to find good settings for these, and our servers only have 96GB of RAM. You also have to consider, as fast as the FusionIO drives are, they're still NVRAM, which has write-amplification issues. How fast do you think it can commit 100GB of dirty memory to disk? Even with a background setting of 1%, that's 10GB on your system. That means you'd need to use a very new kernel so you can utilize the dirty_bytes and dirty_background_bytes settings so you can force those settings into more sane levels to avoid unpredictable several-minute long asyncs. I'm not sure how much testing Linux sees on massive hardware like that, but that's just one hidden danger of not properly benchmarking the server and just thinking 1TB of memory and caching the entire dataset is only an improvement. -- Shaun Thomas Peak6 | 141 W. Jackson Blvd. | Suite 800 | Chicago, IL 60604 312-676-8870 sthomas@xxxxxxxxx ______________________________________________ See http://www.peak6.com/email_disclaimer.php for terms and conditions related to this email -- Sent via pgsql-performance mailing list (pgsql-performance@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
