All, This is the current status map on me server: Total accesses: 14118 - Total Traffic: 3.9 GB CPU Usage: u404.04 s9.82 cu0 cs0 - .956% CPU load .326 requests/sec - 94.4 kB/second - 289.5 kB/request 6 requests currently being processed, 15 idle workers .__..._.._.R_R_......_...__.._._R_.R...R..W_..__................ ................................................................ ................................................................ ................................................................ top reports CPU load average "4.71, 10.49, 10.43" so things are settling down a bit right now. Here's what top shows for the httpd processes (at the top of the report, so these are the highest-CPU processes this particular moment): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23225 apache 20 0 27772 6860 2444 R 8.6 1.1 0:02.94 httpd 23337 apache 20 0 27764 6724 2324 R 6.3 1.1 0:00.80 httpd 23224 apache 20 0 27820 7112 2624 R 4.6 1.2 0:03.02 httpd 23023 apache 20 0 27824 7204 2692 R 4.3 1.2 0:12.15 httpd 21815 apache 20 0 27820 7420 2916 R 4.0 1.2 0:58.10 httpd 22478 apache 20 0 27992 8412 3856 R 4.0 1.4 0:28.18 httpd I wasn't able to copy/paste all of that stuff at exactly the same time, so the numbers are a bit off from each other. But the server is simply not handling very much traffic. I don't understand the spikes in CPU load -- which is what lows everything down. Even doing something like attaching strace to a running process took something like 30 seconds before it started actually logging anything to the terminal. I checked the status of entropy availability using this command: $ cat /proc/sys/kernel/random/entropy_avail I did this a few tens of times and the lowest number I ever got was 133. On systems that have truly run out of randomness, I have seen this number drop to zero (it usually happens during OpenSSL server key generation :). I'm at a loss for what to check next. Any suggestions? Thanks, -chris On 4/3/14, 10:38 AM, Christopher Schultz wrote: > Oscar, > > On 4/3/14, 10:33 AM, Oscar Knorn wrote: >> Sounds like apache is waiting for a response or a means to forward the >> request via stunnel. >> Is /dev/random or /dev/urandom providing sufficient random to the process? > > I'm not sure how to check that. Any ideas? > > In either case above, wouldn't the processes be sitting idle waiting for > blocking IO to return? I'm not surprised that the bytes are moving > slowly, actually. I'm surprised that I'm pegging my CPU waiting for > bytes to move... > > Thanks, > -chris > >> On 4/3/14, 40:23 PM, Christopher Schultz wrote: >>> All, >>> >>> I forgot to mention that most of our traffic is over SSL. OpenSSL >>> version is OpenSSL 1.0.1e-fips 11 Feb 2013. >>> >>> Thanks, >>> -chris >>> >>> On 4/3/14, 10:04 AM, Christopher Schultz wrote: >>>> All, >>>> >>>> I'm having a problem in production I've never seem before. We are >>>> running a pair of AWS EC2 m1.micro web servers where only one of them in >>>> really in service at any given time. The httpd instance serves some >>>> static content and forwards a great deal of traffic via stunnel to a >>>> single back-end Tomcat server using mod_jk 1.2.37. We have been running >>>> under this configuration for several years with no problems. >>>> >>>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from >>>> Amazon's previous version (I can't remember which one), including the >>>> package-refresh that comes with it for httpd. The current kernel version >>>> is 3.10.34. The current httpd version is 2.2.26. The package name is >>>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a >>>> prefork MPM with the following (default) settings: >>>> >>>> StartServers 8 >>>> MinSpareServers 5 >>>> MaxSpareServers 20 >>>> ServerLimit 256 >>>> MaxClients 256 >>>> MaxRequestsPerChild 4000 >>>> >>>> What I can observe is that the CPU load average is rising from the usual >>>> sub-2.0 value to sometimes as high as 70. That's seventy, not >>>> seven-point-oh. >>>> >>>> I see no errors in the log, and httpd doesn't seem to be dropping any >>>> requests... just running very very slowly. >>>> >>>> It seems to come in waves: the load will go up, and everything will slow >>>> down, and then we'll get a reprieve. >>>> >>>> I can see 22 server processes running right this moment, but the load >>>> average has dropped back to 0.05. >>>> >>>> I've enabled ExtendedStatus and it really doesn't look like there is a >>>> huge number of requests being served. Less than 1 req/sec. This is *not* >>>> a high-load server. I can see some of the httpd child processes using >>>> 20% or more of the CPU as reported by 'top'. >>>> >>>> Is there a good way for me to determine what those processes are doing? >>>> As this is a modestly-used server, I can probably enable additional >>>> logging without too much trouble. >>>> >>>> Any help anyone can provide would be very much appreciated. >>>> >>>> Thanks, >>>> -chris >>>> >>> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx >> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx >> >
Attachment:
signature.asc
Description: OpenPGP digital signature
|