Search Postgresql Archives

Re: checkpoint spikes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Martijn, hi Greg,

thanks you very much for your help. We finally got rid of these annoying 
spikes.

First we tried to set 
  checkpoint_segments = 3    # before 16
  checkpoint_timeout = 5min  # before: 60min
which didn't really help.

we had the same spikes but more often. Then we tried to lower 
dirty_writeback_centisecs from 500 to 100

This helped a little bit, but we investigated this problem further.

we monitored "Dirty" memory from /proc/meminfo and we saw very long query 
durations (>5s, sometimes 10s and more) correlating with the kernel writing 
out Dirty buffer. When we saw a massive reduction in "Dirty" memory, there were 
spikes in the query duration. As long as the kernel didn't write out the dirty 
memory, everything did run fine.

So finally we tried 
echo 0 > /proc/sys/vm/dirty_background_ratio

and now everything runs very smooth. We see a few longer query durations over 
2 seconds but no more spikes of 5 or 10 seconds.

Our average response time from our our tomcat servers suddenly dropped from 
300ms to 100ms. Great!!

We know that our limitation is cheap disks, but with /dirty_background_ratio = 
0 you really have big advantages and much better performance. 

So for further reference for other people reading this thread, I really 
recommend trying this out. 

best regards
Janning

On Friday 11 June 2010 21:48:54 Martijn van Oosterhout wrote:
> On Thu, Jun 10, 2010 at 04:00:54PM -0400, Greg Smith wrote:
> >> 5. Does anybody know if I can set dirty_background_ratio to 0.5? As we
> >> have 12 GB RAM and rather slow disks 0,5% would result in a maximum of
> >> 61MB dirty pages.
> >
> > Nope.  Linux has absolutely terrible controls for this critical
> > performance parameter.   The sort of multi-second spikes you're seeing
> > are extremely common and very difficult to get rid of.
>
> Another relevent parameter is /proc/sys/vm/dirty_writeback_centisecs.
> By default linux only wakes up once every 5 seconds to check if there
> is stuff to write out. I have found that reducing this tends to smooth
> out bursty spikes. However, see:
>
> http://www.westnet.com/~gsmith/content/linux-pdflush.htm
>
> which indicates that kernel may try to defeat you here...
>
> Have a nice day,


-- 
Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Postgresql Jobs]     [Postgresql Admin]     [Postgresql Performance]     [Linux Clusters]     [PHP Home]     [PHP on Windows]     [Kernel Newbies]     [PHP Classes]     [PHP Books]     [PHP Databases]     [Postgresql & PHP]     [Yosemite]
  Powered by Linux