Igor, On 4/5/14, 4:57 PM, Igor Cicimov wrote: > > On 06/04/2014 5:51 AM, "Christopher Schultz" > <chris@xxxxxxxxxxxxxxxxxxxxxx <mailto:chris@xxxxxxxxxxxxxxxxxxxxxx>> wrote: >> >> Igor, >> >> On 4/4/14, 5:39 AM, Igor Cicimov wrote: >> > >> > On 04/04/2014 1:05 AM, "Christopher Schultz" >> > <chris@xxxxxxxxxxxxxxxxxxxxxx <mailto:chris@xxxxxxxxxxxxxxxxxxxxxx> > <mailto:chris@xxxxxxxxxxxxxxxxxxxxxx > <mailto:chris@xxxxxxxxxxxxxxxxxxxxxx>>> wrote: >> >> >> >> All, >> >> >> >> I'm having a problem in production I've never seem before. We are >> >> running a pair of AWS EC2 m1.micro web servers where only one of > them in >> >> really in service at any given time. The httpd instance serves some >> >> static content and forwards a great deal of traffic via stunnel to a >> >> single back-end Tomcat server using mod_jk 1.2.37. We have been running >> >> under this configuration for several years with no problems. >> >> >> > Enable the stunnel logs maybe they will reveal something? >> >> I don't think stunnel has changed much. Besides, the stunnel processes >> aren't eating up the CPU: it's the httpd processes that are. >> > Yes but apache actually connects to stunnel and can be affected by it in > case of any issues it has. > > I would also try to run pstack again some of the pids consuming the cpu > to find out what are they doing. > > Basically since the kernel got upgraded any of the parts used here might > be possibly impacted by some new or existing bug. I have some more data. I have launched two EC2 t1.micro instances based upon the same old snapshot that I had available. I upgraded 3 packages on one of them (httpd and two dependencies: mod_ssl and httpd-tools) and ran a load test using JMeter. I have these aggregate numbers for the two machines: Linux kernel 3.2.38 / httpd 2.2.23 Samples: 9942 Average: 715 Median: 347 90% Line: 1500 Min: 159 Max: 47157 Throughput: 35.1 k/sec KB/sec: 120.4 Linux 3.2.38 / httpd 2.2.26: Samples: 9941 Average: 1124 Median: 344 90% line: 2687 Min: 159 Max: 117918 Throughput: 35.1 KB/sec: 120.4 Otherwise the configuration is essentially the same: same modules loaded, etc. The numbers from above represent essentially a trivial request to my Tomcat backend. I'm not sure I can chalk-up the above difference to simply network noise (specifically, average, max, 90% line). What I'm observing here is certainly not the extent of the problems we are experiencing, but I have noticed a difference between these two versions as provided by Amazon's yum repo. I'll see what happens if I start to update other components as well. -chris
Attachment:
signature.asc
Description: OpenPGP digital signature
|