Re: Problem with large twoskip mailboxes.db after upgrade

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I've made another test. I was wondering if problem exists in mailbox/folder name. I generated file for ctl_mboxlist -u with script:

for i in {1..375000} ; do printf "domainname.pl"'!'"user.imap${i}-test\t0 default imap${i}-test@xxxxxxxxxxxxx\tlrswipkxtecdan\tcyrusadmin\tlrswipxtecda\t\n" ; done

And tested varius scenarios on stopped cyrus

50k rec : import 2min : reconstruct on one mailbox 0,4s
100k rec : import 3,31min : reconstruct on one mailbox 0,66s
250k rec : import 9min : reconstruct on one mailbox 1,63s
375k rec : import 13min : reconstruct on one mailbox 2,55s
550k rec : import 19min : reconstruct on one mailbox 3,4s

It shows that amount of records in mailboxes.db
have coincidence with time needed for operations.


Regards,
AK

W dniu 05.11.2021 o 09:12, Andrzej Kwiatkowski pisze:
Hi Bron,

Thanks for your attention.
All is on Centos 8.4. For better tests on the same server we have
installed 2.5.15 (we are using compiled version), and with 2.5
reconstruct lasts:
real    0m0.206s
with full 550k records in mailboxes.db

With 3.4.2 and with 3.2.8(we also installed this version) about 4s.

I think that problem is with slow searching in twoskip, or some
misconfiguration. The only difference between 2.5 and 3.x we have found
is that in 2.5 version inside config/db directory are some DB files :
__db.001  __db.002  __db.003

and with 3.x this folder is empty.

I'm not sure if this is cause, but we were cleaning this folder , and
running reconstruct without starting master, and this files appeared.

Our compilation options are:

/configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus
--with-cyrus-group=cyrus  --disable-gssapi
--with-cyrus-prefix=xxxxxxxxxxxx/cyrus
--with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx
--sysconfdir=xxxxxxxxxxxx/etc --disable-murder
--with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid
--with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled

(nothing special)

and config file (imapd.conf)

configdirectory:        xxxxxxxxxxxxxxxxx/config
defaultpartition: default
partition-default:     xxxxxxxxxxxxxxx/mailbox
sievedir:               xxxxxxxxxxxxxxxxx/sieve
annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf
servername: xxxxxxxxxxxxxxx
hashimapspool: true
fulldirhash: 1
altnamespace: 1
allowusermoves: 1
virtdomains: 1
unixhierarchysep: 1
allowplaintext: 1
sasl_mech_list: plain login
sasl_pwcheck_method: auxprop saslauthd
sasl_auxprop_plugin: sasldb
admins: cyrusadmin
defaultdomain: xxxxxxxxxxxxxxxxx
autocreate_quota: 10485760
autocreate_sieve_folders: Junk
syslog_prefix: xxxxxxxxxxxxxxxxxxxxx
lmtp_over_quota_perm_failure: on
quotawarn: 85
proxyservers: xxxxxxxxxxxxxxxxxxxxxxx
proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx
lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx

Also nothing special,
reverseacls: true didn't helped.

This is standalone instalation without murder, replication and other
features.

Regards
AK


W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:
Hi Andrzej,

What operating system is this on?  I suspect maybe it's that the locks or mmaps are significantly more expensive on this platform than on the places I've used it.

Also if you can give specifics of your config that might help a lot in tracking down what's going on.

I wonder if setting |reverseacls: true| in your imapd.conf might fix this.  If it's locking and unlocking that frequently then it seems like maybe it's doing a mailbox scan that we could be avoiding.

Cheers,

Bron.

On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:

Before Upgrade on  old 2.5.x cyrus we have about 250k mailboxes with
about 500k folders.
We made upgrade with imapsync , because of problem with path
compatibility with unixhierarchysep set on yes.

After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db
are very long. For tests we have made copy of one production servers,
and run without load, reconstruct for empty mailbox lasts 4-5 seconds,
and strace shows us:

fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
st_size=104625064, ...}) = 0
fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
l_len=0}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0

For reconstruct single mailbox there is sometimes 27k operation for RDLCK.

One of our server have database with only 50k folders, and there is
everythink ok.We also made dump, restore mailboxes.db, conversion to
skipstamp and back for twoskip.
Without progress. With zeroskip is much worse.

We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk
performance issue..

Any idea ?

Regards

AK

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f> Delivery options: https://cyrus.topicbox.com/groups/info/subscription <https://cyrus.topicbox.com/groups/info/subscription>


--
   Bron Gondwana, CEO, Fastmail Pty Ltd
   brong@xxxxxxxxxxxxxxxx


*Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions <https://cyrus.topicbox.com/groups/info> + participants <https://cyrus.topicbox.com/groups/info/members> + delivery options <https://cyrus.topicbox.com/groups/info/subscription> Permalink <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755>

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M099f301bbd68ec13d55942d0
Delivery options: https://cyrus.topicbox.com/groups/info/subscription




[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux