Hm.. didn't notice it before. Anyway, i didn't notice that dlm was doing any more job than usually. The most CPU-consuming processes on the alive nodes was "top" itself (although the load was around 600 because of the hang Apache procs). Janar Shawn Hood wrote: > See my thread from yesterday. Same general thing, but the dlm kernel > threads were eating cycles. > > Sent from my iPhone > > On Oct 8, 2008, at 7:24 PM, Janar Kartau <janar.kartau@xxxxxxxxx> wrote: > >> Hi, >> Recently our three-node webserver cluster started randomly crashing. I >> never had time to investigate what the problem was, cause i needed to >> bring them back online again. But it seemed like alla Apache processes >> just hang (couldn't even kill them).. waiting for something. The only >> thing that helped, was a reboot for all or couple of the nodes. Anyway, >> today i encountered this problem at night and i could look into it a >> little more. I noticed that some of the GFS filesystems were >> unaccessable (we have 5 of them, mounted on every nide) and of the nodes >> was completely unaccessable. So i guessed that this half-dead node was >> holding locks on the filesystems or sth. Did a hard reset on this dead >> node and all stabilized. >> Absolutely no cluster/GFS errors in the logs (besides the ones which >> tell that the half-dead node was leaving the cluster when i reset it). >> Nodes have CentOS 4.6 installed (2.6.9-67.0.7.ELsmp, dlm-1.0.7-1, >> GFS-6.1.15-1, cman-1.0.17-0.el4_6.5). We use EMC CX3-10c for GFS storage >> (over iSCSI) and EMC PowerPath for multipathing. Separate VLAN is used >> for CMAN/DLM traffic. >> Please give me ideas how to solve this or atleast some debugging tips as >> it's happening twice a day now and seems i simply can't help it. :( >> >> Janar Kartau >> >> -- >> Linux-cluster mailing list >> Linux-cluster@xxxxxxxxxx >> https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
