Hi Chris, > I think it is important to figure out why this is happening. I would > not want to run any production databases on systems that were failing > like this. You and me both :) (in our application though, it is not a total disaster to lose the last 5 minutes of transactions, it is a disaster if the database is unusable when it comes up) > 1) Any other computers suffer random application crashes, power downs, > etc. in your building? No, but more importantly, we have seen this failure happen in different buildings (in different cities), on same spec but different hardware (at least three motherboards and power supplies, six disks). That's why it really feels like a bug or configuration error. > 2) I take it there are no Raid controllers involved? No. But we get this error with and without software RAID, FWIW. > 3) RAM is non-ECC? I'll have to double-check, but I think it is. > 4) Are the systems on UPS's? Yes. > If I could make a wild (and probably wrong) guess, I would wonder if > something external to the system (like electrical supply) was > introducing glitches into memory, causing bad data to be written. I am > only mentioning it because I have implicated electrical supply in other > cases where rare computer failurres weer affecting many systems... I would tend to agree, but this occurs on multiple systems in multiple locations (but, oddly enough, we are having trouble reproducing it in our lab). And we have run memtest. However, it is true that all the systems on which this has been seen have the same spec power supply/UPS. I would think though, that this could cause error at any time -- all of these failures occur after reboot (that is, no corruption, reboot, immediate corruption). I have stopped/started Postgres while the application is running, without corruption. (smells like a dirty buffer not being written to disk, which is why we focused on the filesystem). Here are some further details: - 865PE/G Neo2-P (MS-6728) ATX motherboard (and similar for IDE) - 2x 512MB/400MHz DIMM RAM - Intel Pentium 4/3.2GHz/1MB/800MHz CPU (hyperthreading enabled) - 2x WD 250GB/7200RPM/8MB/SATA-150 on ICH5 SATA ports (also tested similar IDE drives), writethrough - XFS and JFS (not seen on ext3, but not fully tested) - either software RAID 0 on both drives, or one drive alone without RAID - SuSE 9.1 - 2.6.6 kernel - Postgres 7.4.2 - 300 TPS against DB containing 5-50GB data, no more than a dozen concurrent connections. - fsync (or not) and fdatasync - Postgres may be taken down (via init script) with connections open to it (in fact the application may aggressively try to re-establish the connection as it goes down). - we have put syncs, sleeps and large dd to the disk in the shutdown scripts, none of which work. At this point, I'm really looking for fresh ideas. Thanks, --Ian
