On 6/5/07, Scott Marlowe <smarlowe@xxxxxxxxxxxxxxxxx> wrote:
Greg Smith wrote: > On Tue, 5 Jun 2007, Paolo Bizzarri wrote: > >> On 6/4/07, Scott Marlowe <smarlowe@xxxxxxxxxxxxxxxxx> wrote: >>> http://lwn.net/Articles/215868/ >>> documents a bug in the 2.6 linux kernel that can result in corrupted >>> files if there are a lot of processes accessing it at once. >> >> in fact, we were using a 2.6.12 kernel. Can this be a problem? > > That particular problem appears to be specific to newer kernels so I > wouldn't think it's related to your issue. That is not entirely correct. The problem was present all the way back to the 2.5 kernels, before the 2.6 kernels were released. However, there was an update to the 2.6.18/19 kernels that made this problem much more likely to bite. There were reports of data loss for many people running on older 2.6 kernels that mysteriously went away after updating to post 2.6.19 kernels (or in the case of redhat, the updated 2.6.9-44 or so kernels, which backported the fix.)
I understand this. At the same time, the system was under quite heavy load, so it is possible that some peculiar, rather subtle bug was biting us. There were many files manipulated all in the same way, but only some (really little of them) were truncated. I would like to remove all possible known cases of bugs. BTW, as ou Postgresql was recompiled from sources, do you suggest to recompile the whole after upgrading the kernel?
So, it IS possible that it's the kernel, but not likely. I'm still betting on a bad RAID controller or something like that. But updating the kernel probably wouldn't be a bad idea.
The deployed configuration is quite large (two servers using a shared SCSI-to-IDE large disk array), and it would be quite difficult to replicate a different configuration. At the same time, problems were visible only under heavy load, so using a simpler system would not really help. Ciao Paolo Bizzarri Icube S.r.l.
