Hi,
Problem environment:
Database-server:
- Fedora Core 1 under virtuozzo virtual server running on
Fujitsu-siemens hardware
- Kernel: Linux claymountain.planeetta.com
2.4.20-021stab028.5.777-enterprise #1 SMP Tue Feb 22 17:44:46 MSK 2005
i686 i686 i386 GNU/Linux
- Postgresql 7.4.6 (from rpm by provider: release: 2PGDG Build Date:
Mon 25 Oct 2004 11:25:56 PM EEST Build host gunduz.org)
- Query and update clients as php-applications
Database-update-client:
- Fedora Core 2
- Kernel Linux error.may.day 2.6.10-1.771_FC2 #1 Mon Mar 28 00:50:14 EST
2005 i686 athlon i386 GNU/Linux
- A c++-client build against libpq++-4.0 from pgfoundry. I can not
remember when the application was compiled but definitely not after auto
upgrading client version to 7.4.7
- psql (PostgreSQL) 7.4.7
Problem description:
- After running out of money and several years of operation I moved
database (7.4.6 from fedora RPM) from the client machine to the virtual
server using pg_dump and pg_restore over the Internet and modified
update client to contact remote host instead of localhost.
- After four months of operation on the virtual server the table
"raakaleimat" - the one that the client mainly does inserts into -
crashed with "Invalid page header". After some googling I decided to
drop the table, recreate it, restore data from previous backup as
inserts and restart the database
- After four and half months of operation, a week ago, this happened
again, and again I dropped the table, recreated it and restarted the
database
- Now another table "firmat" that holds keys almost every other table in
the database crashed with "Invalid page header". And what is worse
another non application related database has failed with "Invalid page
header".
It seems to me that the problem is missing log-file
"/var/lib/pgsql/data/pg_clog/0000" which I failed to spot on 21st of
April when
the raakaleimat-table crashed second time.
What can I do to recreate the missing file or is that what I should do?
There are 13 databases on the system. Should I back them all up, drop
everything. Remove every file, do initdb and restore everything? Should
I kick the providers database rpm out of the box and fetch another one
from somewhere else? Or should I downgrade client environment and
recompile ?
Logs:
Apr 21 14:07:45 claymountain postgres[6453]: [8-1] FATAL: the database
system is starting up
Apr 21 14:07:45 claymountain postgres[5912]: [16-1] LOG: database
system is ready
Apr 21 14:08:35 claymountain postgres[7155]: [8-2] DETAIL: could not
open file
"/var/lib/pgsql/data/pg_clog/0295": No such file or directory
Apr 21 14:10:02 claymountain postgres[11079]: [9-2] DETAIL: could not
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory
Apr 21 14:11:11 claymountain postgres[22476]: [8-1] LOG: received fast
shutdown request
Apr 21 14:11:11 claymountain postgres[22476]: [9-1] LOG: aborting any
active transactions
Apr 21 14:11:11 claymountain postgres[12042]: [10-1] FATAL: terminating
connection due to administrator command
Apr 21 14:11:11 claymountain postgres[12016]: [10-1] FATAL: terminating
connection due to administrator command
Apr 21 14:11:21 claymountain postgres[13074]: [11-1] LOG: database
system is shut down
Apr 21 14:11:42 claymountain postgres[13614]: [1-1] LOG: could not
create IPv6
socket: Address family not supported by protocol
Apr 21 14:11:42 claymountain postgres[13618]: [2-1] LOG: database
system was shut down at 2005-04-21 14:11:21 EEST
Apr 21 14:11:42 claymountain postgres[13618]: [3-1] LOG: checkpoint
record is at 6/350B0FDC
Apr 21 14:11:42 claymountain postgres[13618]: [4-1] LOG: redo record is
at 6/350B0FDC; undo record is at 0/0; shutdown TRUE
Apr 21 14:11:42 claymountain postgres[13618]: [5-1] LOG: next
transaction ID: 50072169; next OID: 3492415
Apr 21 14:11:42 claymountain postgres[13618]: [6-1] LOG: database
system is ready
Apr 21 14:13:38 claymountain postgres[15582]: [3-2] DETAIL: could not
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory
...
Apr 28 08:08:03 claymountain postgres[3227]: [3-2] DETAIL: could not
open file
"/var/lib/pgsql/data/pg_clog/0000": No such file or directory
Apr 28 07:59:33 claymountain postgres[25623]: [2-1] ERROR: invalid page
header
in block 1 of relation "firmat"
This is on database pikkusisko, the one that has crashed twice before
Apr 28 08:12:40 claymountain postgres[10835]: [2-1] LOG: recycled
transaction log file "00000006000000B1"
Apr 28 08:41:05 claymountain postgres[7602]: [4-1] ERROR: invalid page
header in block 3 of relation "serendipity_suppress"
This one is one of the four serendipity databases we have, I do not yet
know which one
Apr 28 08:46:45 claymountain postgres[14998]: [3-2] DETAIL: could not
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory
Regards,
Mauri Sahlberg
