Re: Huge performance problems after updating from 2.4 to 2.5.8

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 07/15/2016 08:11 AM, Bron Gondwana via Info-cyrus wrote:
So we've discussed this at length in #cyrus on freenode, and concluded that the issue is that twoskip is doing far too many munmap / mmap calls during an unlocked foreach (which is what the LIST command uses).

I've filed a bug:

https://github.com/cyrusimap/cyrus-imapd/issues/5

And I'm looking at what it would take to fix this behaviour in twoskip now (yes, I know I should be sleeping, but I'm not going to be able to sleep until I understand how this got broken!)

Bron.

On Fri, Jul 15, 2016, at 20:41, Hynek Schlawack via Info-cyrus wrote:
Hello,

we’ve updated one of our Cyrus IMAP backends from 2.4 to 2.5.8 on FreeBSD 10.3 with ZFS and now we have an operational emergency.

Cyrus IMAPd starts fine and keeps working for about 5 to 20 minutes (rather sluggishly tho).  At some point the server load starts growing and explodes eventually until we have to restart the IMAP daemons which gives us another 5 to 20 minutes.

It doesn’t really matter if we run `reconstruct` in the background or not.


# Observations:

1. While healthy, the imapd daemons’s states are mostly `select` or `RUN`.  Once things get critical they all are mostly in `zfs` (but do occasionally switch).
2. Customers report that their mail clients are downloading all e-mails.  That’s obviously extra bad given we seem to run in some kind of I/O problems.  Running `truss` on busy imapd processes seem to confirm that.
3. Once hell breaks loose, IO collapses even on other file systems/hard disks.
4. `top` mentions processes in `lock` state – sometimes even more than 200.  That’s nothing we see on our other backends.
5. There seems to be a correlation between processes hanging in `zfs` state and `truss` showing them accessing mailboxes.db.  Don’t know if it’s related, but soon after the upgrade, mailboxes.db broke and we had to reconstruct it.


# Additional key data:

- 25,000 accounts
- 4.5 TB data
- 64 GB RAM, no apparent swapping
- 16 cores CPU
- nginx in front of it.

## zpool iostat 5

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        4.52T   697G    144  2.03K  1.87M  84.2M
tank        4.52T   697G     84    730  2.13M  3.94M
tank        4.52T   697G    106    904  2.78M  4.52M
tank        4.52T   697G    115    917  3.07M  5.11M
tank        4.52T   697G    101   1016  4.04M  5.06M
tank        4.52T   697G    124  1.03K  3.27M  6.59M

Which doesn’t look special.

The data used to be on HDDs and worked fine with an SSD ZIL.  After the upgrade and ensuing problems we tried a Hail Mary by replacing the HDDs thru SSDs to no avail (migrated a ZFS snapshot for that).

So we do *not* believe it’s really a traditional I/O bottleneck since it only started *after* the upgrade to 2.5 and did not go away by adding SSDs.  The change notes led us to believe that there shouldn’t be any I/O storm due to mailbox conversions but is it true in any case?  How could we double check?  Observation #2 from above leads us to believe that there are in fact some meta data problems.  We’re reconstructing in the background but that’s going to take days; which is sadly time we don’t really have.

## procstat -w 1 of an active imapd

  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     -         FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     -         FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     -         FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     *vm objec FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     -         FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     zfs       FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     -         FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd
45016 43150 43150 43150     0   1 toor     select    FreeBSD ELF64 imapd


Had anyone similar problems (and got them solved, ideally!)?

Are there any known incompatibilities between Cyrus 2.5.8 and FreeBSD/ZFS?

Has anyone ever successfully downgraded from 2.5.8 back to 2.4?

Do we have any other options?

Any help would be *very much* appreciated!
—h
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


So if the issue apparently lies with twoskip, can we keep our dbs using skiplist and do the 2.4 -> 2.5 upgrade? Is it possible -h could revert back to skiplist?

If it could help we could test upgrading to 2.5.8 on our dev server while leaving our database(s) as skiplist.

--
Andy Dorman
Ironic Design, Inc.

CONFIDENTIALITY NOTICE: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any erroneous transmission. If you receive this message in error, please immediately destroy it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, or copy any part of this message if you are not the intended recipient.

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus




[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux