What is the best way to handle "too many open files" errors?

[Ben Welsh <ben.welsh@xxxxxxxxx>]

Hello,

I'm managing a pgsql --> django --> mod_python --> apache machine that serves no media, but handles the postgres memcached and apache part of the stack all by itself. It's a dedicated virtual server with Red Hat EL and 1GB RAM. 

As traffic to the site has ramped up over time, I've begun to encounter "too many open files in the system" errors via Apache that look something like this:

OperationalError: FATAL: could not open file <filename>: Too many open files in system

This has become something of a debilitating problem and I'm struggling to come up with the best strategy for handling it. Some people have told me that I can solve it by better configuring my existing machine -- while others have insisted that I must upgrade my server rig by expanding it beyond one DV. 

My hopes in posting here is that y'all might give me some insight into how I can answer the question about the better approach. 

I've run the commands you see below and the conclusion that I'm drawing is that the I simply have way too many httpd file objects open -- but I'm unsure about the wisest way of reducing them -- or whether I can reasonably expected them to go down much. 

 I'd very much appreciate any insight the list could provide into the situation, even if it's just point me to a reference. I've trolled the lists and the web and still am a Iittle unsure of the best approach, so I thought I'd post. If I'm overlooking the most awesome
tutorial or previous lister post ever, please forgive my ignorance. I'd
appreciate even just the link.

Thank you,

Ben.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 137215
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

postgres.conf (max_files_per_process is commented out)
max_connections = 100
shared_buffers = 1000

$ httpd -l

Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

httpd.conf...

Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5

<IfModule prefork.c>
 StartServers       8
 MinSpareServers    5
 MaxSpareServers   20
 ServerLimit      256
 MaxClients       256
 MaxRequestsPerChild  4000
 </IfModule>

$ cat /proc/sys/fs/file-nr
73320   0       262144

$ lsof | awk '{print $1}' | sort | uniq -c
      1 COMMAND
     10 awk
     49 bash
     17 crond
   5154 httpd
      9 init
     25 lsof
     39 memcached
    807 postmaste
     62 saslauthd
     69 sendmail
     28 sh
      7 sort
    183 sshd
     13 syslogd
      8 uniq
     19 xinetd