Thanks Martijn, I reduced a number of the buffers and connection settings, and added some more tracing and this is the result. The number (TopTransactionContext) is smaller, but still very large. Any reason why this number would suddenly go sky high - the same system was working fine just a month ago. <2006-11-14 05:48:35 EST>LOG: 00000: database system was shut down at 2006-11-14 05:48:30 EST <2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4049 <2006-11-14 05:48:35 EST>LOG: 00000: checkpoint record is at 0/19A825B8 <2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4132 <2006-11-14 05:48:35 EST>LOG: 00000: redo record is at 0/19A825B8; undo record is at 0/0; shutdown TRUE <2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4160 <2006-11-14 05:48:35 EST>LOG: 00000: next transaction ID: 626916; next OID: 54355 <2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4163 <2006-11-14 05:48:35 EST>LOG: 00000: database system is ready <2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4526 <2006-11-14 05:48:35 EST>DEBUG: 00000: proc_exit(0) <2006-11-14 05:48:35 EST>LOCATION: proc_exit, ipc.c:95 <2006-11-14 05:48:35 EST>DEBUG: 00000: shmem_exit(0) <2006-11-14 05:48:35 EST>LOCATION: shmem_exit, ipc.c:126 <2006-11-14 05:48:35 EST>DEBUG: 00000: exit(0) <2006-11-14 05:48:35 EST>LOCATION: proc_exit, ipc.c:113 <2006-11-14 05:48:35 EST>DEBUG: 00000: reaping dead processes <2006-11-14 05:48:35 EST>LOCATION: reaper, postmaster.c:1988 <2006-11-14 05:48:46 EST>DEBUG: 00000: forked new backend, pid=168246 socket=7 <2006-11-14 05:48:46 EST>LOCATION: BackendStartup, postmaster.c:2499 <2006-11-14 05:48:46 EST>DEBUG: 00000: postmaster child[168246]: starting with ( <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2829 <2006-11-14 05:48:46 EST>DEBUG: 00000: postgres <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832 <2006-11-14 05:48:46 EST>DEBUG: 00000: -v196608 <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832 <2006-11-14 05:48:46 EST>DEBUG: 00000: -p <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832 <2006-11-14 05:48:46 EST>DEBUG: 00000: files <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832 <2006-11-14 05:48:46 EST>DEBUG: 00000: ) <2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2834 <2006-11-14 05:48:46 EST>DEBUG: 00000: InitPostgres <2006-11-14 05:48:46 EST>LOCATION: PostgresMain, postgres.c:2719 <2006-11-14 05:48:46 EST>DEBUG: 00000: StartTransaction <2006-11-14 05:48:46 EST>LOCATION: ShowTransactionState, xact.c:3609 <2006-11-14 05:48:46 EST>DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, xid/subid/cid: 626916/ 1/0, nestlvl: 1, children: <> <2006-11-14 05:48:46 EST>LOCATION: ShowTransactionStateRec, xact.c:3634 TopMemoryContext: 32768 total in 3 blocks; 10760 free (3 chunks); 22008 used TopTransactionContext: 1340071936 total in 170 blocks; 928 free (14 chunks); 1340071008 used PortalMemory: 0 total in 0 blocks; 0 free (0 chunks); 0 used CacheMemoryContext: 516096 total in 6 blocks; 178752 free (10 chunks); 337344 used pg_operator_oid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used pg_amproc_opc_proc_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used pg_amop_opc_strat_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used pg_index_indexrelid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used pg_attribute_relid_attnum_index: 1024 total in 1 blocks; 744 free (0 chunks); 280 used pg_class_oid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used pg_amproc_opc_proc_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used pg_amop_opc_strat_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used pg_class_relname_nsp_index: 1024 total in 1 blocks; 744 free (0 chunks); 280 used MdSmgr: 8192 total in 1 blocks; 7808 free (0 chunks); 384 used DynaHash: 8192 total in 1 blocks; 5936 free (0 chunks); 2256 used Operator class cache: 8192 total in 1 blocks; 1968 free (0 chunks); 6224 used smgr relation table: 24576 total in 2 blocks; 16256 free (5 chunks); 8320 used Portal hash: 8192 total in 1 blocks; 4032 free (0 chunks); 4160 used Relcache by OID: 8192 total in 1 blocks; 928 free (0 chunks); 7264 used Relcache by name: 24576 total in 2 blocks; 14208 free (5 chunks); 10368 used LockTable (locallock hash): 24576 total in 2 blocks; 16272 free (6 chunks); 8304 used ErrorContext: 8192 total in 1 blocks; 8160 free (7 chunks); 32 used <2006-11-14 05:50:03 EST>FATAL: 53200: out of memory <2006-11-14 05:50:03 EST>DETAIL: Failed on request of size 24. <2006-11-14 05:50:03 EST>LOCATION: AllocSetAlloc, aset.c:702 <2006-11-14 05:50:03 EST>DEBUG: 00000: proc_exit(0) <2006-11-14 05:50:03 EST>LOCATION: proc_exit, ipc.c:95 <2006-11-14 05:50:03 EST>DEBUG: 00000: shmem_exit(0) <2006-11-14 05:50:03 EST>LOCATION: shmem_exit, ipc.c:126 <2006-11-14 05:50:03 EST>DEBUG: 00000: exit(0) <2006-11-14 05:50:03 EST>LOCATION: proc_exit, ipc.c:113 <2006-11-14 05:50:03 EST>DEBUG: 00000: reaping dead processes <2006-11-14 05:50:03 EST>LOCATION: reaper, postmaster.c:1988 <2006-11-14 05:50:03 EST>DEBUG: 00000: server process (PID 168246) exited with exit code 0 <2006-11-14 05:50:03 EST>LOCATION: LogChildExit, postmaster.c:2349 <2006-11-14 05:53:49 EST>DEBUG: 00000: postmaster received signal 15 <2006-11-14 05:53:49 EST>LOCATION: pmdie, postmaster.c:1850 <2006-11-14 05:53:49 EST>LOG: 00000: received smart shutdown request <2006-11-14 05:53:49 EST>LOCATION: pmdie, postmaster.c:1865 <2006-11-14 05:53:49 EST>LOG: 00000: shutting down <2006-11-14 05:53:49 EST>LOCATION: ShutdownXLOG, xlog.c:4706 <2006-11-14 05:53:49 EST>DEBUG: 00000: reaping dead processes <2006-11-14 05:53:49 EST>LOCATION: reaper, postmaster.c:1988 <2006-11-14 05:53:49 EST>LOG: 00000: database system is shut down <2006-11-14 05:53:49 EST>LOCATION: ShutdownXLOG, xlog.c:4715 <2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95 <2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0) <2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126 <2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113 <2006-11-14 05:53:49 EST>DEBUG: 00000: reaping dead processes <2006-11-14 05:53:49 EST>LOCATION: reaper, postmaster.c:1988 <2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95 <2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0) <2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126 <2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113 <2006-11-14 05:53:49 EST>LOG: 00000: logger shutting down <2006-11-14 05:53:49 EST>LOCATION: SysLoggerMain, syslogger.c:361 <2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95 <2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0) <2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126 <2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0) <2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113 -----Original Message----- From: Martijn van Oosterhout [mailto:kleptog@xxxxxxxxx] Sent: Tuesday, November 14, 2006 6:44 AM To: Rob Owen Cc: pgsql-general@xxxxxxxxxxxxxx Subject: Re: [GENERAL] Out of memory (Failed on request size 24) On Tue, Nov 14, 2006 at 05:53:08AM -0500, Rob Owen wrote: > PostgreSQL 8.0.3 running on AIX 5.3 (same thing happens on 5.1 though). > DBMS was running fine for some months but now one of the databases isn't accessible. Any help would be greatly appreciated. > > DBMS starts up fine, but any operation on the files database (psql > files, vaccumdb files, pgdump files) yields the same result. The > client responds with > > > psql files <snip> Something screwed up: > TopTransactionContext: 2145378304 total in 266 blocks; 928 free (14 > chunks); 2145377376 used That's a lot of memory. I thought there was a check on negative sized allocations... Did "make check" pass ok? Have a nice day, -- Martijn van Oosterhout <kleptog@xxxxxxxxx> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
