Re: High load, low CPU and processes in state "devfs"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks Mohit. Shortly after you sent this last night I enabled system accounting. There's now a fair amount of data in the output but I'm not really sure where to find the disk wait times...

root@harold:~# sa
   41065    44351.99re      428.71cp       11avio      823k
    8678    33375.24re      418.23cp       34avio      496k   httpd*
    2469       51.19re        4.79cp        2avio      767k   convert
    1513     1180.51re        2.69cp       48avio     4929k   php
    2469       56.01re        2.59cp        2avio     1312k   composite
     590     3945.74re        0.09cp        5avio     6182k   smtp
     624      152.45re        0.09cp        0avio   196079k   perl*
      46      468.10re        0.08cp      691avio      574k   cleanup
    1170       10.05re        0.08cp       11avio   107305k   rateup
      47      388.52re        0.03cp      190avio     1054k   pickup
743 240.80re 0.02cp 0avio 27976k trivial- rewrite
   10443     1319.75re        0.00cp        0avio 26333351k   sh
     154      305.90re        0.00cp       24avio    44750k   bounce
    4454      101.21re        0.00cp        0avio  1753042k   sendmail
    4454       52.77re        0.00cp        7avio  2748826k   postdrop
       2        1.00re        0.00cp        0avio     3136k   top
       9     1278.02re        0.00cp        2avio    32944k   ***other
     389       13.73re        0.00cp        0avio   908587k   ps
       2        0.01re        0.00cp        6avio      981k   sa
      24       80.90re        0.00cp        0avio    60224k   scache
       9       35.12re        0.00cp        0avio    12700k   anvil
      78        0.28re        0.00cp        0avio   198900k   atrun
     623     1043.23re        0.00cp        0avio   959100k   cron*
      39        1.24re        0.00cp        2avio   112100k   dd
      27       52.78re        0.00cp        0avio   109600k   error
      65      114.24re        0.00cp        0avio   298000k   flush
    1165       41.51re        0.00cp        0avio  4475200k   grep
      40        0.14re        0.00cp        0avio    60000k   jot
     279        1.16re        0.00cp        3avio   971600k   mv
       7        0.31re        0.00cp        3avio     7200k   newsyslog
      13        1.68re        0.00cp        0avio    42500k   proxymap
      12       23.78re        0.00cp        0avio    78800k   smtpd
      40        0.16re        0.00cp        2avio   194800k   unlink
     388       14.47re        0.00cp        0avio  1219700k   wc

Any help would be appreciated. Thanks.

-Stut

On 14 Jun 2008, at 00:43, Mohit Anchlia wrote:

try "sa"

On 6/13/08, Stut <stuttle@xxxxxxxxx> wrote: On 13 Jun 2008, at 23:38, Mohit Anchlia wrote: look at "sar" output and see the wait times on the disk is more than the service time.

That command doesn't seem to exist on FreeBSD. Do you know if there's an equivalent command?

-Stut


On 6/13/08, Stut <stuttle@xxxxxxxxx> wrote:

On 13 Jun 2008, at 22:56, Dragon wrote:

Stut wrote:
On 13 Jun 2008, at 22:37, Dragon wrote:
Stut wrote:
Hi,

I have a problem with one of the web servers I manage. It runs
FreeBSD
6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre- fork
processes (usually around 240).

What basically happens is that during our peak hours in the evening
the site becomes very slow as does everything on the server. The top
output shows nearly all httpd processes in the state "devfs" and the
load is jumping up over 10 but the CPU is 70% idle. There's plenty of
free memory, over 4GB, and lots of available disk space.

While it's in this state all disk access is painfully slow. I've
checked all the drives and the RAID card status and everything
appears
to be fine. It also recovers itself after a few hours when the
traffic
dies down again.

I've spent a lot of today Googling but can't find any reference to
this particular combination of symptoms. Does anyone have any ideas?
---------------- End original message. ---------------------

It sounds to me like your system is waiting on disk resources (I am
assuming that is what the devfs state is), it is very likely trying
to use swap space.

Now much memory does this system have?

If it has insufficient memory to keep most things in RAM, it is
going to use a lot of swap and that will slow things down
considerably.

Oh how I wish it were that simple...

Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
Free
Swap: 8192M Total, 8192M Free

No swap in use, plenty of free memory.
---------------- End original message. ---------------------

Well, I took my best shot on the info presented. I still think there is something in the disk I/O path that is acting as a bottleneck.

Sorry it isn't so easy, I have nothing else here. Wish I could help.

No worries, thanks for trying.

I agree that it's probably disk-related, but other than these symptoms I can't find anything wrong.

I'll see if the FreeBSD list can help.

-Stut

--
http://stut.net/


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
 "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
 "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx


[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux