not sure if we qualify as big enough, but here goes: we typically have
3000 concurrent TLS/SSL connections on each Perdition server during peak
hours (although we occasionally see 5000), but the CPU impact is
negligible[1]. at peak, 8% system and 12% user out of 400% CPU
available (this is Dell PE2650 with dual Xeon 1.8 GHz HT) and just 15
Mbps of network traffic in and out -- quite symmetric as you would
expect for a proxy server. Perdition's connection to Cyrus is
unencrypted, of course.
[1] the main issue is RAM for all those processes. with just 2 GiB RAM
it ran sluggishly once in while during lunchtime, they now have 4 GiB
each.
We used to use perdition. The problem we found was that even with Linux 2.6
and the O(1) scheduler, when you get to 7000+ processes, the entire box and
all the connections feel sluggish and the machine has a load of around 2 or
so, even though it's mostly CPU idle. RAM usage is also huge (3G or so). It
seems that just handling the context switching and management of all those
processes is a bottleneck.
Now by coincidence, about a year ago we were in contact with a guy called
Igor Sysoev (who developed a module called mod_accel for apache) and he had
started building a new http proxy system based on a non-blocking
architecture using epoll/kqueue called nginx. We spoke with him about
extending the proxy to work with POP/IMAP. We negotiated, came up with a
price, and he built the appropriate code for us. It's now incorporated into
the public release of nginx itself (http://sysoev.ru/nginx/). As an aside,
we've done a similar thing with cyrus, sponsoring Ken to write various
extensions that made it into 2.3 before he moved to CMU. I really think this
is a great way to support open source projects and the people involved with
them.
Anyway the good news:
Before: 2 frontend servers with 7000+ connections (eg 14,000+ total) using
6G of RAM with a load on each of about 2
After: 1 frontend server with 14,000+ connections, less than 1G of RAM
usage, load of under 0.5
The only problem is that all the docs are in Russian. I should get around to
writing up a document on setting it up, it's not that hard. The main issue
is that you have to write your own authentication daemon. It doesn't connect
to anything standard like mysql or LDAP, but it does use a simple protocol
you can write your own adaptor for quickly in something like perl.
Rob
----
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
