Hi,I've made another test. I was wondering if problem exists in mailbox/folder name. I generated file for ctl_mboxlist -u with script:
for i in {1..375000} ; do printf "domainname.pl"'!'"user.imap${i}-test\t0 default imap${i}-test@xxxxxxxxxxxxx\tlrswipkxtecdan\tcyrusadmin\tlrswipxtecda\t\n" ; done
And tested varius scenarios on stopped cyrus 50k rec : import 2min : reconstruct on one mailbox 0,4s 100k rec : import 3,31min : reconstruct on one mailbox 0,66s 250k rec : import 9min : reconstruct on one mailbox 1,63s 375k rec : import 13min : reconstruct on one mailbox 2,55s 550k rec : import 19min : reconstruct on one mailbox 3,4s It shows that amount of records in mailboxes.db have coincidence with time needed for operations. Regards, AK W dniu 05.11.2021 o 09:12, Andrzej Kwiatkowski pisze:
Hi Bron, Thanks for your attention. All is on Centos 8.4. For better tests on the same server we have installed 2.5.15 (we are using compiled version), and with 2.5 reconstruct lasts: real 0m0.206s with full 550k records in mailboxes.db With 3.4.2 and with 3.2.8(we also installed this version) about 4s. I think that problem is with slow searching in twoskip, or some misconfiguration. The only difference between 2.5 and 3.x we have found is that in 2.5 version inside config/db directory are some DB files : __db.001 __db.002 __db.003 and with 3.x this folder is empty. I'm not sure if this is cause, but we were cleaning this folder , and running reconstruct without starting master, and this files appeared. Our compilation options are: /configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus --with-cyrus-group=cyrus --disable-gssapi --with-cyrus-prefix=xxxxxxxxxxxx/cyrus --with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx --sysconfdir=xxxxxxxxxxxx/etc --disable-murder --with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid --with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled (nothing special) and config file (imapd.conf) configdirectory: xxxxxxxxxxxxxxxxx/config defaultpartition: default partition-default: xxxxxxxxxxxxxxx/mailbox sievedir: xxxxxxxxxxxxxxxxx/sieve annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf servername: xxxxxxxxxxxxxxx hashimapspool: true fulldirhash: 1 altnamespace: 1 allowusermoves: 1 virtdomains: 1 unixhierarchysep: 1 allowplaintext: 1 sasl_mech_list: plain login sasl_pwcheck_method: auxprop saslauthd sasl_auxprop_plugin: sasldb admins: cyrusadmin defaultdomain: xxxxxxxxxxxxxxxxx autocreate_quota: 10485760 autocreate_sieve_folders: Junk syslog_prefix: xxxxxxxxxxxxxxxxxxxxx lmtp_over_quota_perm_failure: on quotawarn: 85 proxyservers: xxxxxxxxxxxxxxxxxxxxxxx proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx Also nothing special, reverseacls: true didn't helped. This is standalone instalation without murder, replication and other features. Regards AK W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:Hi Andrzej,What operating system is this on? I suspect maybe it's that the locks or mmaps are significantly more expensive on this platform than on the places I've used it.Also if you can give specifics of your config that might help a lot in tracking down what's going on.I wonder if setting |reverseacls: true| in your imapd.conf might fix this. If it's locking and unlocking that frequently then it seems like maybe it's doing a mailbox scan that we could be avoiding.Cheers, Bron. On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:Before Upgrade on old 2.5.x cyrus we have about 250k mailboxes with about 500k folders. We made upgrade with imapsync , because of problem with path compatibility with unixhierarchysep set on yes. After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db are very long. For tests we have made copy of one production servers, and run without load, reconstruct for empty mailbox lasts 4-5 seconds, and strace shows us: fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0 For reconstruct single mailbox there is sometimes 27k operation for RDLCK. One of our server have database with only 50k folders, and there is everythink ok.We also made dump, restore mailboxes.db, conversion to skipstamp and back for twoskip. Without progress. With zeroskip is much worse. We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk performance issue.. Any idea ? Regards AK ------------------------------------------ Cyrus: InfoPermalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f> Delivery options: https://cyrus.topicbox.com/groups/info/subscription <https://cyrus.topicbox.com/groups/info/subscription>-- Bron Gondwana, CEO, Fastmail Pty Ltd brong@xxxxxxxxxxxxxxxx*Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions <https://cyrus.topicbox.com/groups/info> + participants <https://cyrus.topicbox.com/groups/info/members> + delivery options <https://cyrus.topicbox.com/groups/info/subscription> Permalink <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755>
------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M099f301bbd68ec13d55942d0 Delivery options: https://cyrus.topicbox.com/groups/info/subscription
