FWIW,
I can at least report the resolution of the original problem.
I went sleuthing and found some core files in the ./base/13860299 directory. Deleteing those freed up some gigabytes of space (each core was 1-2 gigs).
The server that I had tried to stop with "-m immediate" command did in fact then go offline; it came up with a few complaints; I ran a vacuum on all of the databases in that instance and our content manager was able to do his update (the update was such that reapplying it to any given row didn't hurt anything; these were massive updates changing copyright related info and the like). So far the database has passed all sanity checks and is back online.
The server that I left alone was responsive, i.e. psql could connect and do queries, but there were a few tables it refused to have anything to do with, complaining about missing xlog files. I brought it down with with "-m fast" mode, restarted it and it also seems now to be fine. (Knock on simulated woodgrain)
Lessons learned:
a) upgrade to current revisions whenever possible -- old software is a hand grenade waiting to go off.
b) look for core files and delete them if you don't need them -- I was not expecting to find them in a data directory so this was a bit of s surprise.
c) don't run out of disk space (duh)
Thanks to all who helped me. I might be able to get a server to test on with a different release of postgres if that would be useful, although we are strictly a linux shop and Dell x86 servers are what I mostly can get my hands on (running 2.4.21-0.13mdkenterprise).
Greg W.
-----Original Message-----
From: pgsql-admin-owner@xxxxxxxxxxxxxx on behalf of Tom Lane
Sent: Tue 1/3/2006 10:38 AM
To: Tomaz Borstnar
Cc: pgsql-admin@xxxxxxxxxxxxxx
Subject: Re: [ADMIN] full data disk -- any chance of recovery
Tomaz Borstnar <tomaz.borstnar@xxxxxxxx> writes:
> Jeff Frost pravi:
>> Seems like you're going to have to kill -9.
> Yeah, this is bad :( Seems like kill -9 is needed when disk is full. Tested on *BSD jails.
With what PG version? And what behavior did you see exactly?
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
!DSPAM:43bac4af260671270013900!
