Check your pagecache settings, when doing heavy io writes of a large file you can basically force a linux box to completely stall. At some point once the pagecache has reached it's limit it'll force all IO to go sync basically from my understanding. We are still fighting with this but lots of changes in RH6 seem to address of lot of these issues. grep -i dirty /proc/meminfo cat /proc/sys/vm/ cat /proc/sys/vm/nr_pdflush_threads Once the dirty pages reaches a really large size and the limit of pagecache your system should experience a pretty abrupt drop in performance. You should be able to avoid this by using sync writes, but we haven't had a chance to completely isolate and address this issue. -----Original Message----- From: pgsql-performance-owner@xxxxxxxxxxxxxx [mailto:pgsql-performance-owner@xxxxxxxxxxxxxx] On Behalf Of Claudio Freire Sent: Thursday, April 12, 2012 1:50 PM To: Brian Fehrle Cc: pgsql-performance@xxxxxxxxxxxxxx Subject: Re: Random performance hit, unknown cause. On Thu, Apr 12, 2012 at 3:41 PM, Brian Fehrle <brianf@xxxxxxxxxxxxxxxxxxx> wrote: > This morning, during our nightly backup process (where we grab a copy > of the data directory), we started having this same issue. The main > thing that I see in all of these is a high disk wait on the system. > When we are performing 'well', the %wa from top is usually around 30%, > and our load is around 12 - 15. This morning we saw a load 21 - 23, > and an %wa jumping between 60% and 75%. > > The top process pretty much at all times is the WAL Sender Process, is > this normal? Sounds like vacuum to me. -- Sent via pgsql-performance mailing list (pgsql-performance@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. -- Sent via pgsql-performance mailing list (pgsql-performance@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
