|
I set the cyrus.service script to print our ulimit -a. This is
what it shows. Is there anything here that should limit cyrus?
May 25 02:36:51 MAILSERVER systemd[1]: Starting LSB: The
cyrus-imapd mail system...
May 25 02:36:51 MAILSERVER cyrus[6762]: Starting IMAP/POP3 service
(cyrus-imapd)core file size (blocks, -c) 102400
May 25 02:36:51 MAILSERVER cyrus[6762]: data seg size
(kbytes, -d) unlimited
May 25 02:36:51 MAILSERVER cyrus[6762]: scheduling
priority (-e) 0
May 25 02:36:51 MAILSERVER cyrus[6762]: file size
(blocks, -f) unlimited
May 25 02:36:51 MAILSERVER cyrus[6762]: pending
signals (-i) 63615
May 25 02:36:51 MAILSERVER cyrus[6762]: max locked memory
(kbytes, -l) 64
May 25 02:36:51 MAILSERVER cyrus[6762]: max memory size
(kbytes, -m) unlimited
May 25 02:36:51 MAILSERVER cyrus[6762]: open
files (-n) 209702
May 25 02:36:51 MAILSERVER cyrus[6762]: pipe size (512
bytes, -p) 8
May 25 02:36:51 MAILSERVER cyrus[6762]: POSIX message queues
(bytes, -q) 819200
May 25 02:36:51 MAILSERVER cyrus[6762]: real-time
priority (-r) 50000
May 25 02:36:51 MAILSERVER cyrus[6762]: stack size
(kbytes, -s) 8192
May 25 02:36:51 MAILSERVER cyrus[6762]: cpu time
(seconds, -t) unlimited
May 25 02:36:51 MAILSERVER cyrus[6762]: max user
processes (-u) 4096
May 25 02:36:51 MAILSERVER cyrus[6762]: virtual memory
(kbytes, -v) unlimited
May 25 02:36:51 MAILSERVER cyrus[6762]: file
locks (-x) unlimited
May 25 02:36:52 MAILSERVER master[6773]: setrlimit: Unable to set
file descriptors limit to -1: Operation not permitted
May 25 02:36:52 MAILSERVER master[6773]: retrying with 209702
(current max)
May 25 02:36:52 MAILSERVER master[6773]: process started
May 25 02:36:52 MAILSERVER cyrus[6762]: ..done
May 25 02:36:52 MAILSERVER systemd[1]: Started LSB: The
cyrus-imapd mail system.
May 25 02:36:52 MAILSERVER master[6776]: about to exec
/usr/lib/cyrus/bin/ctl_cyrusdb
May 25 02:36:52 MAILSERVER ctl_cyrusdb[6776]: recovering cyrus
databases
May 25 02:36:52 MAILSERVER ctl_cyrusdb[6776]: skiplist:
checkpointed /var/lib/imap/mailboxes.db (1446 records, 186620
bytes) in 0 seconds
May 25 02:36:52 MAILSERVER ctl_cyrusdb[6776]: skiplist:
checkpointed /var/lib/imap/annotations.db (0 records, 144 bytes)
in 0 seconds
May 25 02:36:52 MAILSERVER ctl_cyrusdb[6776]: done recovering
cyrus databases
May 25 02:36:52 MAILSERVER master[6778]: about to exec
/usr/lib/cyrus/bin/idled
May 25 02:36:52 MAILSERVER master[6773]: ready for work
On 05/24/2017 10:59 PM, Michael Hieb
wrote:
I did
try to change the service names. I posted that result to the
group. It did not make a difference.
I am not a low-level programmer so maybe have the wrong idea here,
but it seems locked on reading the imapd executable. Doesn't that
suggest a system limit of either open files or open processes? I
have tried to open up those limits as it was my first angle of
investigation. What confuses me is that the problem goes away when
I switch to a single-threaded Cyrus service listening on all ports
simultaneously.
Any thoughts, suggestions welcome.
Michael
On May 24, 2017 10:42:30 PM GMT+08:00,
Michael Ulitskiy <mulitskiy@xxxxxxxxxx> wrote:
well, it tells us that it has to do with locking during connection accept.
have you tried to use different service name per server as was suggested by
others?
Michael
On Wednesday, May 24, 2017 12:00:58 PM Michael Hieb wrote:
I have had good success connecting to the right process and can
replicate the result. Here is what I found;
MAILSERVER:~ # ps auwwx|grep imapd
cyrus 1738 0.0 0.0 119572 10140 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1745 0.0 0.0 119580 9812 ? S 03:32 0:00 imapd
-C /etc/imapd.domain2.com.conf -s
cyrus 1749 0.0 0.0 119596 9856 ? S 03:32 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1750 0.0 0.0 129424 11500 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1751 0.0 0.0 119696 9988 ? S 03:32 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1752 0.0 0.0 119596 9816 ? S 03:32 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 1753 0.0 0.0 119604 9964 ? S 03:32 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1754 0.0 0.0 119596 9652 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1755 0.0 0.0 119596 9800 ? S 03:32 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1756 0.0 0.0 119604 10056 ? S 03:32 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 1757 0.0 0.0 129564 11724 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1758 0.0 0.0 119512 9896 ? S 03:33 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1766 0.0 0.0 119588 9904 ? S 03:33 0:00 imapd
-C /etc/imapd.domain3.com.conf -s
cyrus 1767 0.0 0.0 119576 9804 ? S 03:33 0:00 imapd
-C /etc/imapd.domain3.com.conf -s
cyrus 1778 0.0 0.0 119480 9936 ? S 03:33 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1786 0.0 0.0 119768 10144 ? S 03:33 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1881 0.0 0.0 119808 10080 ? S 03:34 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1912 0.0 0.0 119688 9868 ? S 03:36 0:00 imapd
-C /etc/imapd.domain2.com.conf
cyrus 1937 0.0 0.0 119784 10216 ? S 03:38 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1947 0.0 0.0 125872 10660 ? S 03:39 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1993 0.0 0.0 119604 9808 ? S 03:42 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 2078 0.0 0.0 117044 7628 ? S 03:48 0:00 imapd
-C /etc/imapd.domain4.com.conf
root 2080 0.0 0.0 10548 1656 pts/2 S+ 03:48 0:00 grep
--color=auto imapd
Then try to connect:
user@somesystem:~> telnet imap.domain2.com 143
Trying 192.168.110.175...
Connected to imap.domain2.com.
Escape character is '^]'.
Then mailserver is the same except one new process at the end:
MAILSERVER:~ # ps auwwx|grep imapd
cyrus 1738 0.0 0.0 119572 10140 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1745 0.0 0.0 119580 9812 ? S 03:32 0:00 imapd
-C /etc/imapd.domain2.com.conf -s
cyrus 1749 0.0 0.0 119596 9856 ? S 03:32 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1750 0.0 0.0 129424 11500 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1751 0.0 0.0 119696 9988 ? S 03:32 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1752 0.0 0.0 119596 9816 ? S 03:32 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 1753 0.0 0.0 119604 9964 ? S 03:32 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1754 0.0 0.0 119596 9652 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1755 0.0 0.0 119596 9800 ? S 03:32 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1756 0.0 0.0 119604 10056 ? S 03:32 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 1757 0.0 0.0 129564 11724 ? S 03:32 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1758 0.0 0.0 119512 9896 ? S 03:33 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1766 0.0 0.0 119588 9904 ? S 03:33 0:00 imapd
-C /etc/imapd.domain3.com.conf -s
cyrus 1767 0.0 0.0 119576 9804 ? S 03:33 0:00 imapd
-C /etc/imapd.domain3.com.conf -s
cyrus 1778 0.0 0.0 119480 9936 ? S 03:33 0:00 imapd
-C /etc/imapd.domain1.com.conf -s
cyrus 1786 0.0 0.0 119768 10144 ? S 03:33 0:00 imapd
-C /etc/imapd.domain5.com.conf
cyrus 1881 0.0 0.0 119808 10080 ? S 03:34 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1912 0.0 0.0 119688 9868 ? S 03:36 0:00 imapd
-C /etc/imapd.domain2.com.conf
cyrus 1937 0.0 0.0 119784 10216 ? S 03:38 0:00 imapd
-C /etc/imapd.domain3.com.conf
cyrus 1947 0.0 0.0 125872 10660 ? S 03:39 0:00 imapd
-C /etc/imapd.domain1.com.conf
cyrus 1993 0.0 0.0 119604 9808 ? S 03:42 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 2078 0.0 0.0 117044 7628 ? S 03:48 0:00 imapd
-C /etc/imapd.domain4.com.conf
cyrus 2120 0.0 0.0 117044 7628 ? S 03:48 0:00 imapd
-C /etc/imapd.domain2.com.conf
root 2122 0.0 0.0 10548 1656 pts/2 S+ 03:48 0:00 grep
--color=auto imapd
MAILSERVER:~ # strace -p 2120
Process 2120 attached
fcntl(13, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}
Process hangs, nothing happening... then about 45 seconds later (in this
case - time to wait varies each time I try this) it frees itself and
strace shows this:
MAILSERVER:~ # strace -p 2120
Process 2120 attached
fcntl(13, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
stat("/usr/lib/cyrus/bin/imapd", {st_mode=S_IFREG|0755, st_size=1758976,
...}) = 0
accept(4, 0, NULL) = 14
fcntl(13, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
alarm(0) = 0
getpeername(14, {sa_family=AF_INET, sin_port=htons(43778),
sin_addr=inet_addr("192.168.110.27")}, [16]) = 0
getpeername(14, {sa_family=AF_INET, sin_port=htons(43778),
sin_addr=inet_addr("192.168.110.27")}, [16]) = 0
getsockname(14, {sa_family=AF_INET, sin_port=htons(143),
sin_addr=inet_addr("192.168.110.174")}, [16]) = 0
open("/etc/hosts.allow", O_RDONLY|O_CLOEXEC) = 15
fstat(15, {st_mode=S_IFREG|0644, st_size=2639, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7ff634095000
read(15, "# /etc/hosts.allow\n# See 'man tc"..., 4096) = 2639
read(15, "", 4096) = 0
close(15) = 0
munmap(0x7ff634095000, 4096) = 0
open("/etc/hosts.deny", O_RDONLY|O_CLOEXEC) = 15
fstat(15, {st_mode=S_IFREG|0644, st_size=149, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7ff634095000
read(15, "# /etc/hosts.deny\n# See 'man tcp"..., 4096) = 149
read(15, "", 4096) = 0
close(15) = 0
munmap(0x7ff634095000, 4096) = 0
write(3, "\2\0\0\0H\10\0\0", 8) = 8
sendto(6, "<31>May 24 03:56:21 imap[2120]: "..., 51, MSG_NOSIGNAL, NULL,
0) = 51
dup2(14, 0) = 0
dup2(14, 1) = 1
dup2(14, 2) = 2
close(14) = 0
write(3, "\3\0\0\0H\10\0\0", 8) = 8
getpeername(0, {sa_family=AF_INET, sin_port=htons(43778),
sin_addr=inet_addr("192.168.110.27")}, [16]) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 14
connect(14, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(14, "\2\0\0\0\r\0\0\0\6\0\0\0hosts\0", 18, MSG_NOSIGNAL, NULL, 0)
= 18
poll([{fd=14, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=14,
revents=POLLIN|POLLHUP}])
recvmsg(14, {msg_name(0)=NULL, msg_iov(2)=[{"hosts\0", 6},
{"\310O\3\0\0\0\0\0", 8}], msg_controllen=20, [{cmsg_len=20,
cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, [15]}],
msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 14
mmap(NULL, 217032, PROT_READ, MAP_SHARED, 15, 0) = 0x7ff634021000
close(15) = 0
close(14) = 0
getsockname(0, {sa_family=AF_INET, sin_port=htons(143),
sin_addr=inet_addr("192.168.110.174")}, [16]) = 0
open("/var/lib/imap/proc/2120", O_RDWR|O_CREAT|O_TRUNC, 0666) = 14
fstat(14, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7ff634095000
lseek(14, 0, SEEK_SET) = 0
write(14, "somehost.domain1.com [192.168.110.2"..., 35) = 35
lseek(14, 0, SEEK_CUR) = 35
ftruncate(14, 35) = 0
open("/var/lib/imap/msg/motd", O_RDONLY) = -1 ENOENT (No such file or
directory)
write(1, "* OK [CAPABILITY IMAP4rev1 LITER"..., 199) = 199
select(1, [0], NULL, NULL, {1800, 0}
and telnet session shows this:
user@somesystem:~> telnet imap.domain2.com 143
Trying 192.168.110.175...
Connected to imap.domain2.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=GSS-SPNEGO
AUTH=GSSAPI AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=LOGIN AUTH=PLAIN SASL-IR]
imap.domain2.com Cyrus IMAP v2.4.18 server ready
Thereafter telnet to imap.domain2.com has immediate response and other
domains hang..
I can replicate this at will.
What does it mean?
Michael
On 05/23/2017 10:38 PM, Michael Ulitskiy wrote:
you can use 'strace -p <pid>', specifying the pid of process you want to
trace. you will have to "catch" the process that blocks. it may not be
easy, but I have no other suggestions.
On Tuesday, May 23, 2017 06:19:17 PM Michael Hieb wrote:
The SASL is cyrus-sasl 2.1.26-9.5. I believe it does use /dev/urandom.
I am not sure how to run strace on the process that the Master listener
forks. Here is the strace run on the imapd process at a command prompt
(which is not quite the same as the master listener forking it to a port
where another process is waiting to connect to it).
MAILSERVER:~ # strace /usr/lib/cyrus/bin/imapd -C
/etc/imapd.domain1.com.conf
execve("/usr/lib/cyrus/bin/imapd", ["/usr/lib/cyrus/bin/imapd", "-C",
"/etc/imapd.domain1.com.conf"], [/* 56 vars */]) = 0
brk(0) = 0x5573deaf5000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb770c000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=80522, ...}) = 0
mmap(NULL, 80522, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fdeb76f8000
close(3) = 0
open("/usr/lib64/libsasl2.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300N\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=118552, ...}) = 0
mmap(NULL, 2213800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb72d0000
mprotect(0x7fdeb72ec000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb74eb000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b000) = 0x7fdeb74eb000
close(3) = 0
open("/usr/lib64/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200Y\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=855280, ...}) = 0
mmap(NULL, 2951008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb6fff000
mprotect(0x7fdeb70c1000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb72c0000, 65536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc1000) = 0x7fdeb72c0000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f7000
open("/lib64/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\26\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14760, ...}) = 0
mmap(NULL, 2109928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb6dfb000
mprotect(0x7fdeb6dfe000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb6ffd000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fdeb6ffd000
close(3) = 0
open("/lib64/libssl.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pr\1\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0555, st_size=440632, ...}) = 0
mmap(NULL, 2535888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb6b8f000
mprotect(0x7fdeb6bf0000, 2097152, PROT_NONE) = 0
mmap(0x7fdeb6df0000, 45056, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x61000) = 0x7fdeb6df0000
close(3) = 0
open("/lib64/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\313\6\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0555, st_size=2447744, ...}) = 0
mmap(NULL, 4559184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb6735000
mprotect(0x7fdeb6962000, 2097152, PROT_NONE) = 0
mmap(0x7fdeb6b62000, 167936, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22d000) = 0x7fdeb6b62000
mmap(0x7fdeb6b8b000, 12624, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb6b8b000
close(3) = 0
open("/usr/lib64/libdb-4.8.so", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@X\2\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1560248, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f6000
mmap(NULL, 3655304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb63b8000
mprotect(0x7fdeb6530000, 2097152, PROT_NONE) = 0
mmap(0x7fdeb6730000, 20480, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x178000) = 0x7fdeb6730000
close(3) = 0
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P#\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=88216, ...}) = 0
mmap(NULL, 2183304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb61a2000
mprotect(0x7fdeb61b7000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb63b6000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7fdeb63b6000
close(3) = 0
open("/lib64/libwrap.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p8\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=40952, ...}) = 0
mmap(NULL, 2139520, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb5f97000
mprotect(0x7fdeb5fa0000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb619f000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7fdeb619f000
mmap(0x7fdeb61a1000, 1408, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb61a1000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\10\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1925280, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f5000
mmap(NULL, 3811872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb5bf4000
mprotect(0x7fdeb5d8e000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb5f8d000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x199000) = 0x7fdeb5f8d000
mmap(0x7fdeb5f93000, 14880, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb5f93000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=18712, ...}) = 0
mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb59f0000
mprotect(0x7fdeb59f2000, 2097152, PROT_NONE) = 0
mmap(0x7fdeb5bf2000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fdeb5bf2000
close(3) = 0
open("/usr/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340K\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=191424, ...}) = 0
mmap(NULL, 2290168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb57c0000
mprotect(0x7fdeb57ed000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb59ec000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2c000) = 0x7fdeb59ec000
mmap(0x7fdeb59ef000, 504, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb59ef000
close(3) = 0
open("/usr/lib64/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20:\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=52416, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f4000
mmap(NULL, 2147816, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb55b3000
mprotect(0x7fdeb55bf000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb57be000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb000) = 0x7fdeb57be000
close(3) = 0
open("/lib64/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\26\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14688, ...}) = 0
mmap(NULL, 2109704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb53af000
mprotect(0x7fdeb53b2000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb55b1000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fdeb55b1000
close(3) = 0
open("/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2209\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=97152, ...}) = 0
mmap(NULL, 2189896, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb5198000
mprotect(0x7fdeb51ac000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb53ab000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) = 0x7fdeb53ab000
mmap(0x7fdeb53ad000, 6728, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb53ad000
close(3) = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360r\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=135952, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f3000
mmap(NULL, 2213008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb4f7b000
mprotect(0x7fdeb4f93000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb5192000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fdeb5192000
mmap(0x7fdeb5194000, 13456, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb5194000
close(3) = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0ps\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=147208, ...}) = 0
mmap(NULL, 2251056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb4d55000
mprotect(0x7fdeb4d78000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb4f77000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fdeb4f77000
mmap(0x7fdeb4f79000, 6448, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fdeb4f79000
close(3) = 0
open("/usr/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\30\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=452976, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f2000
mmap(NULL, 2548232, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fdeb4ae6000
mprotect(0x7fdeb4b54000, 2093056, PROT_NONE) = 0
mmap(0x7fdeb4d53000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6d000) = 0x7fdeb4d53000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76f0000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb76ee000
arch_prctl(ARCH_SET_FS, 0x7fdeb76ee840) = 0
mprotect(0x7fdeb5f8d000, 16384, PROT_READ) = 0
mprotect(0x7fdeb5192000, 4096, PROT_READ) = 0
mprotect(0x7fdeb4d53000, 4096, PROT_READ) = 0
mprotect(0x7fdeb5bf2000, 4096, PROT_READ) = 0
mprotect(0x7fdeb4f77000, 4096, PROT_READ) = 0
mprotect(0x7fdeb53ab000, 4096, PROT_READ) = 0
mprotect(0x7fdeb55b1000, 4096, PROT_READ) = 0
mprotect(0x7fdeb57be000, 4096, PROT_READ) = 0
mprotect(0x7fdeb59ec000, 8192, PROT_READ) = 0
mprotect(0x7fdeb619f000, 4096, PROT_READ) = 0
mprotect(0x7fdeb63b6000, 4096, PROT_READ) = 0
mprotect(0x7fdeb6730000, 8192, PROT_READ) = 0
mprotect(0x7fdeb6b62000, 110592, PROT_READ) = 0
mprotect(0x7fdeb6df0000, 16384, PROT_READ) = 0
mprotect(0x7fdeb6ffd000, 4096, PROT_READ) = 0
mprotect(0x7fdeb72c0000, 53248, PROT_READ) = 0
mprotect(0x7fdeb74eb000, 4096, PROT_READ) = 0
mprotect(0x5573dd954000, 8192, PROT_READ) = 0
mprotect(0x7fdeb770d000, 4096, PROT_READ) = 0
munmap(0x7fdeb76f8000, 80522) = 0
set_tid_address(0x7fdeb76eeb10) = 18042
set_robust_list(0x7fdeb76eeb20, 24) = 0
rt_sigaction(SIGRTMIN, {0x7fdeb4f81d80, [], SA_RESTORER|SA_SIGINFO,
0x7fdeb4f8bb10}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fdeb4f81e10, [],
SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdeb4f8bb10}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) =
0
statfs("/sys/fs/selinux", 0x7ffe1b4e8b00) = -1 ENOENT (No such file or
directory)
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096,
f_blocks=474903111, f_bfree=452714842, f_bavail=452228240,
f_files=120627200, f_ffree=120300880, f_fsi\
d={352889743, -1913047647}, f_namelen=255, f_frsize=4096}) = 0
brk(0) = 0x5573deaf5000
brk(0x5573deb16000) = 0x5573deb16000
open("/proc/filesystems", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb770b000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 297
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7fdeb770b000, 4096) = 0
access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or
directory)
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3, "\355\207\254Q\253\17d\266\22x\324\257
\236\205H\327b\32\221nzmx\251\216D\257J\362\301\357"..., 60) = 60
close(3) = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
poll([{fd=3, events=POLLIN}], 1, 10) = 1 ([{fd=3, revents=POLLIN}])
read(3,
"r\324\5\243C\224\n\222d\333\25\25\2738\7\ts\351\3759\265\207&\n\251}\316
\20 5\224v\17\253"..., 1024) = 1024
close(3) = 0
getuid() = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3, "\3323i\236\226J\353\241\201\213\20\30\201b<\371\5\246\234\230",
20) = 20
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3,
"u2V%3\367\276\251\202$\376_\1\365r`N^\5?\303H\330\3173i\335\247]A\222\37
6". .., 256) = 256
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\333_\2\335]:\7\"\373s]\352\375\305\312t\244.@;\245\345\245^(\262d\350\2
73[ \327-"..., 216) = 216
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3, "\207", 1) = 1
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3, "\332", 1) = 1
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\22\335\242$\210`\376\376d\200\215m\213-\0162\r\257\206\335\nY\343\3376/
\22 4\352\360\312\336\360", 32) = 32
close(3) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...})
= 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
read(3,
"U\2\251M\21fm\345\334\215Q1(\253\1M!\270\226\212s&A\263}\351\16\271",
28) = 28
close(3) = 0
futex(0x7fdeb5bf30a8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("/lib64/.libcrypto.so.1.0.0.hmac", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("/proc/sys/crypto/fips_enabled", O_RDONLY) = -1 ENOENT (No such
file or directory)
access("/lib64/.libcrypto.so.1.0.0.hmac", F_OK) = -1 ENOENT (No such
file or directory)
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=127, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=127, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fdeb770b000
read(3,
"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096)
= 127
lseek(3, -71, SEEK_CUR) = 56
read(3,
"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) =
71 close(3) = 0
munmap(0x7fdeb770b000, 4096) = 0
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = 0
sendto(3, "<11>May 23 10:11:18 imapd: could"..., 67, MSG_NOSIGNAL, NULL,
0) = 67
exit_group(70) = ?
+++ exited with 70 +++
Michael
On 05/19/2017 05:55 PM, Michael Ulitskiy wrote:
Msg # 57050 Previous in Sequence
<http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.info-cyrus
&ms g=57374>
Date Fri, 19 May 2017 11:32:38 -0400
To info-cyrus@xxxxxxxxxxxxxxxxxxxx
From Michael Ulitskiy <mulitskiy@xxxxxxxxxx>
Reply-To: Michael Ulitskiy <mulitskiy@xxxxxxxxxx>
Subject Re: Multi-thread cyrus delayed forking of imapd processes after
connecting to master listener
My immediate guess is it blocks at reading /dev/random.
strace should show you exactly where it blocks.
What version of SASL are you using? Newer versions have switched to
/dev/urandom AFAIK
Michael
On Friday, May 19, 2017 05:48:36 PM Michael Hieb wrote:
Apologies for long post, most of it is configuration details.
I have recently changed from a single threaded cyrus server in which
multiple domains were overloaded on one listener as multiple virtual
domains to a multiple threaded cyrus server in which multiple domains
are listened for on separate ip addresses with separate configurations.
The primary motivation was to have separate tls certificates for each
domain. The problem is that I can connect to cyrus listener on all
ip/ports for all domains, but while one of them will fork immediately
and respond with a banner, the others will delay for a period of time
from a few seconds to a minute or so before forking and responding with
a banner. With the same setup, I change to a single threaded cyrus
server to listen on all ip/ports then I get an immediate response on
all, but of course I do not get the separate tls certificate or
configuration. It is replicable that switching between the single
threaded and multiple threaded configuration triggers the problem. I
have searched the logs and the mail-lists and found nothing that seems
related.
Question: why does switching to multiple threaded cyrus server trigger
delayed forking of imapd processes after connecting to master listener?
On listener which responds immediately (as expected) I get this:
user@somehost:~> telnet imap.domain1.com 143
Trying 192.168.110.171...
Connected to imap.domain1.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=GSS-SPNEGO
AUTH=GSSAPI AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=LOGIN AUTH=PLAIN SASL-IR]
imap.domain1.com Cyrus IMAP v2.4.18 server ready
On listener with delay, I get this (and process hangs for many seconds
or even a minute or two):
user@somehost:~> telnet imap.domain2.com 143
Trying 192.168.110.171...
Connected to imap.domain2.com.
Escape character is '^]'.
The domain for which the listener responds and the one for which it
delays seems to change and be random as far as I can tell.
There is nothing
--
Sent from my Android device with K-9 Mail. Please excuse my
brevity.
|
