Thanks Bruce. Really interesting, but, I show zone reclaim is already turned off on our system. root@site-db01b:~ # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 node 0 size: 393181 MB node 0 free: 467 MB node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23 node 1 size: 393215 MB node 1 free: 319 MB node distances: node 0 1 0: 10 20 1: 20 10 root@site-db01b:~ # cat /proc/sys/vm/zone_reclaim_mode 0 Thanks, Karthik On 3/26/14 1:54 PM, "Bruce Momjian" <bruce@xxxxxxxxxx> wrote: >On Wed, Mar 26, 2014 at 08:22:01PM +0000, Anand Kumar, Karthik wrote: >> Looking a little deeper, I saw signs of memory being heavily fragmented: >> >> root@site-db01b:/var/log # cat /proc/buddyinfo >> Node 0, zone DMA 1 1 2 2 2 1 0 0 1 1 3 >> Node 0, zone DMA32 8 7 8 7 10 8 7 11 9 5 92 >> Node 0, zone Normal 13069 0 0 0 0 0 0 0 0 0 1 >> Node 1, zone Normal 652315 36885 1168 0 1 1 0 1 1 1 0 >> >> >> Node 0 has 13069 4k blocks, and zero 8k blocks available to use >> Which is likely what caused the problem, I'd think. >> >> A little while later though, buddyinfo changed and suddenly there was a >> lot more memory in 8k blocks. >> >> root@site-db01b:/proc # cat /proc/buddyinfo >> Node 0, zone DMA 1 1 2 2 2 1 0 >> 0 1 1 3 >> Node 0, zone DMA32 8 7 8 7 10 8 7 >> 11 9 5 92 >> Node 0, zone Normal 9645 5495 1115 0 0 0 0 >> 0 0 0 1 >> Node 1, zone Normal 409734 10953 1 0 1 1 0 >> 1 1 1 0 >> >> (Note the change in the Node 0 line, 8k blocks went up from 0 to 5495) >> >> Anyone have any idea why memory was so fragmented, and what causes >>memory >> to be defragged? Is it something postgres does? Are there any kernel >> specific settings that control it? > >If I had to take a guess, it is zone_reclaim; see: > > http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-recl >aim-mode.html > >The fix is this sysctl: > > vm.zone_reclaim_mode = 0 > >-- > Bruce Momjian <bruce@xxxxxxxxxx> http://momjian.us > EnterpriseDB http://enterprisedb.com > > + Everyone has their own god. + -- Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
