Re: Problem with large twoskip mailboxes.db after upgrade

[Andrzej Kwiatkowski <lists@xxxxxxxxxxxxxx>]

Hi Bron,

Thanks for your attention.
All is on Centos 8.4. For better tests on the same server we have
installed 2.5.15 (we are using compiled version), and with 2.5
reconstruct lasts:
real    0m0.206s
with full 550k records in mailboxes.db

With 3.4.2 and with 3.2.8(we also installed this version) about 4s.

I think that problem is with slow searching in twoskip, or some
misconfiguration. The only difference between 2.5 and 3.x we have found
is that in 2.5 version inside config/db directory are some DB files :
__db.001  __db.002  __db.003

and with 3.x this folder is empty.

I'm not sure if this is cause, but we were cleaning this folder , and
running reconstruct without starting master, and this files appeared.

Our compilation options are:

/configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus
--with-cyrus-group=cyrus  --disable-gssapi
--with-cyrus-prefix=xxxxxxxxxxxx/cyrus
--with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx
--sysconfdir=xxxxxxxxxxxx/etc --disable-murder
--with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid
--with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled

(nothing special)

and config file (imapd.conf)

configdirectory:        xxxxxxxxxxxxxxxxx/config
defaultpartition: default
partition-default:     xxxxxxxxxxxxxxx/mailbox
sievedir:               xxxxxxxxxxxxxxxxx/sieve
annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf
servername: xxxxxxxxxxxxxxx
hashimapspool: true
fulldirhash: 1
altnamespace: 1
allowusermoves: 1
virtdomains: 1
unixhierarchysep: 1
allowplaintext: 1
sasl_mech_list: plain login
sasl_pwcheck_method: auxprop saslauthd
sasl_auxprop_plugin: sasldb
admins: cyrusadmin
defaultdomain: xxxxxxxxxxxxxxxxx
autocreate_quota: 10485760
autocreate_sieve_folders: Junk
syslog_prefix: xxxxxxxxxxxxxxxxxxxxx
lmtp_over_quota_perm_failure: on
quotawarn: 85
proxyservers: xxxxxxxxxxxxxxxxxxxxxxx
proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx
lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx

Also nothing special,
reverseacls: true didn't helped.

This is standalone instalation without murder, replication and other
features.

Regards
AK


W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:
> Hi Andrzej,
>
> What operating system is this on?  I suspect maybe it's that the locks 
> or mmaps are significantly more expensive on this platform than on the 
> places I've used it.
>
> Also if you can give specifics of your config that might help a lot in 
> tracking down what's going on.
>
> I wonder if setting |reverseacls: true| in your imapd.conf might fix 
> this.  If it's locking and unlocking that frequently then it seems like 
> maybe it's doing a mailbox scan that we could be avoiding.
>
> Cheers,
>
> Bron.
>
> On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:
> > Before Upgrade on  old 2.5.x cyrus we have about 250k mailboxes with
> > about 500k folders.
> > We made upgrade with imapsync , because of problem with path
> > compatibility with unixhierarchysep set on yes.
> >
> > After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db
> > are very long. For tests we have made copy of one production servers,
> > and run without load, reconstruct for empty mailbox lasts 4-5 seconds,
> > and strace shows us:
> >
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> > stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
> > st_size=104625064, ...}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
> > l_len=0}) = 0
> > fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
> >
> > For reconstruct single mailbox there is sometimes 27k operation for RDLCK.
> >
> > One of our server have database with only 50k folders, and there is
> > everythink ok.We also made dump, restore mailboxes.db, conversion to
> > skipstamp and back for twoskip.
> > Without progress. With zeroskip is much worse.
> >
> > We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk
> > performance issue..
> >
> > Any idea ?
> >
> > Regards
> >
> > AK
> >
> > ------------------------------------------
> > Cyrus: Info
> > Permalink: 
> > https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f 
> > <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f>
> > Delivery options: https://cyrus.topicbox.com/groups/info/subscription 
> > <https://cyrus.topicbox.com/groups/info/subscription>
>
> --
>    Bron Gondwana, CEO, Fastmail Pty Ltd
>    brong@xxxxxxxxxxxxxxxx
>
>
> *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions 
> <https://cyrus.topicbox.com/groups/info> + participants 
> <https://cyrus.topicbox.com/groups/info/members> + delivery options 
> <https://cyrus.topicbox.com/groups/info/subscription> Permalink 
> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755> 

------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M2f58fb436c7d065a07a74af0
Delivery options: https://cyrus.topicbox.com/groups/info/subscription