Re: Out Of Memory 8.1

["French, Martin" <frenchm@xxxxxxxxxxxxxx>]

Thanks for the suggestion, however; this query runs in under 2 minutes (big data set) on the server it originated from, and Iâm trying to ascertain why there is such a difference in performance between what should be two like for like server, as there are many other queries of a similar nature that have bigger sort operations to perform, and I honestly donât want to have to re-write them all.

 

Setting enable_hashagg = off causes the query to complete, however; it means the next batch queries never completes (left it > 8 hours) when ordinarily itâs done in 5 minutes.

 

Thanksâ

 

Martin.

 

From: scorpdaddy@xxxxxxxxxxx [mailto:scorpdaddy@xxxxxxxxxxx]
Sent: 07 April 2011 11:20
To: French, Martin; pgsql-admin@xxxxxxxxxxxxxx
Subject: Re: [ADMIN] Out Of Memory 8.1

 

Have you tried moving the FROM ... WHERE ... into a sub-select? GROUP BY uses HAVING, not WHERE.


----- Reply message -----
From: "French, Martin" <frenchm@xxxxxxxxxxxxxx>
Date: Wed, Apr 6, 2011 7:44 pm
Subject: [ADMIN] Out Of Memory 8.1
To: <pgsql-admin@xxxxxxxxxxxxxx>

Hi All,

I am having problems with a query on 8.1 running on

RHEL 5.4
16GB RAM
Linux pgsql3 2.6.18-164.el5PAE #1 SMP Tue Aug 18 15:59:11 EDT 2009 i686
i686 i386 GNU/Linux
2 x Xeon X5650 (2.67GHz 6 Cores)
Disks are on PERC 6 controller in RAID 10

Postgresql.conf:
# - Memory -

shared_buffers = 327680        
work_mem = 98394
maintenance_work_mem = 983040

The query:

2011-04-06 23:23:03 BST <app.charisma><26083>LOG:  statement: CREATE
TABLE medusa.cmi_group_free AS
SELECT
stkl_stockno,
(SUM(stkphys) - (SUM(stkalloc) + SUM(stkbacko))) as group_free
FROM
charisma.sk_stklfl
WHERE
stkl_stockno NOT LIKE 'ZZ%' AND
stkl_stockno ~
'^([A-z&]{2,3})([0-9][0-9][0-9])([0-9][0-9][0-9][0-9][A-z0-9]{1,3})$'
GROUP BY
stkl_stockno;

The explain (cannot explain analyze, or Postgres runs out of memory
again)
'HashAggregate  (cost=2731947.55..2731947.57 rows=1 width=38)'
'  ->  Seq Scan on stkl_rec  (cost=0.00..2731947.54 rows=1 width=38)'
'        Filter: (((stkl_comp)::text = 'A'::text) AND
((stkl_stockno)::text ~
'^(TK[A-Za-z0-9][0-9]{3}(?:[0-9]{5}(?:[0-9]{3,4})?)?|NSP[0-9]{3}([0-9]{4
})?|Z[MZ][0-9]{8,9}|LSP[0-9]{7}[A-Za-z0-9]|[A-Z][A-Z&][A-Z][0-9]{7}[A-Z0
-9])$'::text) AND ((stkl_stockno)::text !~~ 'ZZ%'::text) AND
((stkl_stockno)::text ~
'^([A-z&]{2,3})([0-9][0-9][0-9])([0-9][0-9][0-9][0-9][A-z0-9]{1,3})$'::t
ext))'

The table has approximately
6.9 million rows


It's not the OOM that's doing this as vm.overcommit_memory = 2 and
vm.overcommit_ratio = 90 nor is postgres swapping to disk.

Can anyone help me here? The same query on exactly the same data on an
older machine running the same setup except that it's RHEL 5.0 runs
fine!

Thanks.

Martin

___________________________________________________
 
This email is intended for the named recipient. The information contained
in it is confidential.  You should not copy it for any purposes, nor

___________________________________________________

This email is intended for the named recipient. The information contained
in it is confidential. You should not copy it for any purposes, nor
disclose its contents to any other party. If you received this email
in error, please notify the sender immediately via email, and delete
it from your computer.

Any views or opinions presented are solely those of the author and do not
necessarily represent those of the company.

PCI Compliancy: Please note, we do not send or wish to receive banking,
credit or debit card information by email or any other form of
communication.

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________