Performance problems

Thomas Moyer <tmmoyer@xxxxxxxxxxx> · Thu, 11 Sep 2008 08:23:41 -0400

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am having some problems with performance when using mod_python to  

serve some dynamic content.  The request handler gathers information  

being generated by background processes on the system, and then  

creates a simple XML document that is sent to the client. Here is the  

code I am using with some explanation.

    requestedFilePath = "/" + req.filename.split("/")[-1] 

[:-4].replace("_", "/").replace("-",".")

    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    sock.connect(ATTService)
    Attest = message.recvMsg(sock)[0]
    sock.close()

    fmt = "%ds" % len(requestedFilePath)
    msg = struct.pack(fmt, requestedFilePath)
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    sock.connect(MHTService)
    message.sendMsg(sock, msg, fmt)
    MHTList = message.recvMsg(sock)[0]
    sock.close()

    req.content_type = 'text/xml'
    req.write("<ACA>%s%s</ACA>" % (Attest,MHTList))
    return apache.OK

There are 3 different pieces here.  The first block connects to a  

daemon running on the system that is monitoring the system. I connect  

to a Unix domain socket and receive the most recent information it  

has.  I then do the same thing with another daemon that is gathering  

information about files that are available on the system. Finally I  

combine these two blocks of information and send it to the client.

My problem is, when I start using benchmarking tools like JMeter, and  

I put the system under any sort of load (100 clients for example), the  

first recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12  

seconds to complete per client.  I watch top during the run and see  

that the apache processes are completely occupied (usually showing 120+ 

% CPU usage).  The system running apache is an 8-core machine with  

16GB of RAM so I'm pretty sure there isn't a resource problem for  

either memory or processor.

I'm not really sure where to look next to understand why a simple  

socket recv() call is taking 12 seconds to complete.  The amount of  

data being recv()'d is ~80K of text if that makes a difference.

Below is the config file for Apache:
ServerRoot "/etc/apache2"
LockFile /var/lock/apache2/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
AccessFileName .htaccess
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>
DefaultType text/plain
HostnameLookups Off
ErrorLog /var/log/apache2/error.log
LogLevel warn
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Include /etc/apache2/httpd.conf
Include /etc/apache2/ports.conf

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i 

\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
ServerTokens Full
ServerSignature On
PythonOption mod_python.mutex_locks 32
PythonOptimize On
Include /etc/apache2/conf.d/
Include /etc/apache2/sites-enabled/

Enabled Modules:
lias.conf
alias.load
auth_basic.load
authn_file.load
authz_default.load
authz_groupfile.load
authz_host.load
authz_user.load
autoindex.load
cgid.load
dir.load
env.load
mime.load
mod_python.load
negotiation.load
setenvif.load
status.load

And finally the site configuration itself:
NameVirtualHost *
<VirtualHost *>
	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/
	<Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
		AddHandler mod_python .aca
		PythonHandler serveACA | .aca
		PythonPath "sys.path + ['/var/shamon/']"
		PythonDebug On
	</Directory>
	ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
	<Directory "/usr/lib/cgi-bin">
		AllowOverride None
		Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
		Order allow,deny
		Allow from all
	</Directory>
	ErrorLog /var/log/apache2/error.log
	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel warn
	CustomLog /var/log/apache2/access.log combined
	ServerSignature On
    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>
</VirtualHost>

Not sure what other information is needed.

~tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkjJDc0ACgkQa7Yypxfw9TdvngCgio05p2p9rHXhI3qB2EC8LgPV
uJEAn0w60bAQJ6QiMsZDrd4Z0c+uTzjr
=4TUr
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
  "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx