On Mon, Dec 06, 2004 at 07:25:32PM -0800, Duncan Morgan wrote: > Unfortunately we don't see the load decrease at all and have to reboot > all servers in the cluster - causing mass chaos :)) so I'm not sure we > are talking about the same issue. Then you have enough time to examine the problem more thoroughly. Use lsof to check which resources are accessed by the processes in D-state. You will probably find that a larger share is accessing the same files. > Thanks for your input. > > Duncan > > From: linux-cluster-bounces@xxxxxxxxxx > [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Axel Thimm > Sent: Monday, December 06, 2004 7:15 PM > To: linux clistering > Subject: [Linux-cluster] Re: D Apache processes > > On Mon, Dec 06, 2004 at 06:52:02PM -0800, Duncan Morgan wrote: > > Hello, > > > > We have apache running on 14 GFS nodes where the web roots are shared > > via GFS. Occasionally we see that the load on all nodes rises > > dramatically (to 150+) and all httpd processes become dead (D). I know > > this is a little lacking in details but does anyone have any insight > > into this? We suspected perhaps a cron job was running simultaneously > > against the GFS file system on all nodes but have virtually ruled this > > out. > > I am getting the same behaviour on a non-clustered, non GFS system > (dual opteron on FC2/x86_64). There is a peak of almost all httpd > processes in D-state (that's not really dead, but "uninterruptible > sleep", e.g. when the kernel does IO for the userland process). A few > seconds later the number of D-processes fall down to less than a dozen > and you can watch your load exponentially decrease. > > Note that 150 is the default MaxClient setting for apache, that's why > you get slightly more than 150 load. > > I guess that's a kernel issue with too many processes accessing the > same files. Nothing to do directly with GFS. > > > Please help - this is very alarming. > > > > Thanks in advance, > > Duncan Morgan > > > -- Axel.Thimm at ATrpms.net
Attachment:
pgpdoisLUW4l5.pgp
Description: PGP signature
