On Dec 11, 2006, at 1:27 PM, Andrew Morgan wrote:
On Mon, 11 Dec 2006, Nik Conwell wrote:
On Dec 11, 2006, at 12:29 PM, Andrew Morgan wrote:
On Mon, 11 Dec 2006, Nik Conwell wrote:
I'm using the UW mailutil to transfer mailboxes from UW to Cyrus
(2.3.7). It uses APPEND, specifically multiappend (single APPEND
with multiple messages being appended). Cyrus-imapd handles
this multiappend by creating stage files for each appended
message and leaving the file descriptor open. The problem is
that after 240 messages, we run out of file descriptors and so
an open() of the next stage file fails with EMFILE. I updated /
etc/cyrus.conf to make the max fds be 1024 (AFAICT kernel MAX)
which helped somewhat but not for larger mailboxes with > 1008
messages.
Shouldn't the multiappend/append be closing the FD for each
stage file and then reopening it later as it needs it?
Do people just tweak their kernels to have some insane number of
FDs available in order to compensate for this?
Or, do people not use mailutil and instead use something that
issues multiple append commands rather than a single append with
multiple e-mails?
We run with a much, much larger number of file descriptors here.
I've increased the system limit to around 200k (/proc/sys/fs/file-
max on linux). This is for the day-to-day running of Cyrus, so I
don't know if you would need a higher limit for running mailutil
(but I doubt it).
In practice, each of my backends has only used a maximum of
around 12k file descriptors, but I'd hate to run out! :)
That's for the entire system though, right? I'm running into a
1024 limit per process, namely the cyrus imap server process has
all the appended stage files open. Am I missing something
fundamental here? (I probably am because I would have figured
people would have run into this issue already...)
BTW - my /proc/sys/fs/file-max has 406572.
Ah, then you just need to up the ulimits in your cyrus init script.
Something like:
# Crank up the limits
ulimit -n 209702
ulimit -u 2048
ulimit -c 102400
That did the trick. Thanks a lot.
-nik
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
