Re: High load, low CPU and processes in state "devfs"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



You'll need to analyze this data with what you are seeing on this system. I suggest you get sar like tool. Look at http://www.googlebit.com/bsdsar/ or see this post too http://groups.google.com/group/sol.lists.freebsd.questions/browse_thread/thread/b999f095612e59bd/257ddbadf0c9ec71?hl=en&lnk=st&q=FreeBSD+sar#257ddbadf0c9ec71

Or ask you system admin. You need to look at it very closely.

On Sat, Jun 14, 2008 at 10:23 AM, Stut <stuttle@xxxxxxxxx> wrote:
Thanks Mohit. Shortly after you sent this last night I enabled system accounting. There's now a fair amount of data in the output but I'm not really sure where to find the disk wait times...

root@harold:~# sa
  41065    44351.99re      428.71cp       11avio      823k
   8678    33375.24re      418.23cp       34avio      496k   httpd*
   2469       51.19re        4.79cp        2avio      767k   convert
   1513     1180.51re        2.69cp       48avio     4929k   php
   2469       56.01re        2.59cp        2avio     1312k   composite
    590     3945.74re        0.09cp        5avio     6182k   smtp
    624      152.45re        0.09cp        0avio   196079k   perl*
     46      468.10re        0.08cp      691avio      574k   cleanup
   1170       10.05re        0.08cp       11avio   107305k   rateup
     47      388.52re        0.03cp      190avio     1054k   pickup
    743      240.80re        0.02cp        0avio    27976k   trivial-rewrite
  10443     1319.75re        0.00cp        0avio 26333351k   sh
    154      305.90re        0.00cp       24avio    44750k   bounce
   4454      101.21re        0.00cp        0avio  1753042k   sendmail
   4454       52.77re        0.00cp        7avio  2748826k   postdrop
      2        1.00re        0.00cp        0avio     3136k   top
      9     1278.02re        0.00cp        2avio    32944k   ***other
    389       13.73re        0.00cp        0avio   908587k   ps
      2        0.01re        0.00cp        6avio      981k   sa
     24       80.90re        0.00cp        0avio    60224k   scache
      9       35.12re        0.00cp        0avio    12700k   anvil
     78        0.28re        0.00cp        0avio   198900k   atrun
    623     1043.23re        0.00cp        0avio   959100k   cron*
     39        1.24re        0.00cp        2avio   112100k   dd
     27       52.78re        0.00cp        0avio   109600k   error
     65      114.24re        0.00cp        0avio   298000k   flush
   1165       41.51re        0.00cp        0avio  4475200k   grep
     40        0.14re        0.00cp        0avio    60000k   jot
    279        1.16re        0.00cp        3avio   971600k   mv
      7        0.31re        0.00cp        3avio     7200k   newsyslog
     13        1.68re        0.00cp        0avio    42500k   proxymap
     12       23.78re        0.00cp        0avio    78800k   smtpd
     40        0.16re        0.00cp        2avio   194800k   unlink
    388       14.47re        0.00cp        0avio  1219700k   wc

Any help would be appreciated. Thanks.

-Stut


On 14 Jun 2008, at 00:43, Mohit Anchlia wrote:

try "sa"

On 6/13/08, Stut <stuttle@xxxxxxxxx> wrote: On 13 Jun 2008, at 23:38, Mohit Anchlia wrote:
look at "sar" output and see the wait times on the disk is more than the service time.

That command doesn't seem to exist on FreeBSD. Do you know if there's an equivalent command?

-Stut


On 6/13/08, Stut <stuttle@xxxxxxxxx> wrote:

On 13 Jun 2008, at 22:56, Dragon wrote:

Stut wrote:
On 13 Jun 2008, at 22:37, Dragon wrote:
Stut wrote:
Hi,

I have a problem with one of the web servers I manage. It runs
FreeBSD
6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre- fork
processes (usually around 240).

What basically happens is that during our peak hours in the evening
the site becomes very slow as does everything on the server. The top
output shows nearly all httpd processes in the state "devfs" and the
load is jumping up over 10 but the CPU is 70% idle. There's plenty of
free memory, over 4GB, and lots of available disk space.

While it's in this state all disk access is painfully slow. I've
checked all the drives and the RAID card status and everything
appears
to be fine. It also recovers itself after a few hours when the
traffic
dies down again.

I've spent a lot of today Googling but can't find any reference to
this particular combination of symptoms. Does anyone have any ideas?
---------------- End original message. ---------------------

It sounds to me like your system is waiting on disk resources (I am
assuming that is what the devfs state is), it is very likely trying
to use swap space.

Now much memory does this system have?

If it has insufficient memory to keep most things in RAM, it is
going to use a lot of swap and that will slow things down
considerably.

Oh how I wish it were that simple...

Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
Free
Swap: 8192M Total, 8192M Free

No swap in use, plenty of free memory.
---------------- End original message. ---------------------

Well, I took my best shot on the info presented. I still think there is something in the disk I/O path that is acting as a bottleneck.

Sorry it isn't so easy, I have nothing else here. Wish I could help.

No worries, thanks for trying.

I agree that it's probably disk-related, but other than these symptoms I can't find anything wrong.

I'll see if the FreeBSD list can help.

-Stut

--
http://stut.net/


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
 "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
 "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
 "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx



[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux