Hi,
Quoting Bron Gondwana <brong@xxxxxxxxxxxxxxxx>:
Hi Michael,
Sorry about the delay in looking at this - I was mad crazy busy
getting ready to go overseas. At Fosdem now, about to give a talk
about JMAP!
OK, let's start with the things that give me a little bit of hives...
configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part
Ouch. There's a couple of things I wouldn't do there - having the
partition be the same as the config directory, and having a separate
partition be a subdirectory of a partition. They're both asking for
trouble. I would probably lay my system out like:
configdirectory: /srv/cyrus-be/conf
partition-default: /srv/cyrus-be/default-part
partition-ssd: /srv/cyrus-be/ssd-part
partition-default isn't used any more. To use the metapartition we moved
all accounts form the default partition to the ssd partition which is the
the new defaultpartition ("defaultpartition: ssd")
And then each tree would only have one type of thing in it.
Anyway, I don't think that would break anything.
metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations
lock dav archivecache
Ooh, I haven't tested having cache and archivecache on the same
location. That's really interesting. Again, I'd be in favour of
separation here, give them different paths. That might be tricky
with ssd though, the way this is laid out. I assume you have some
kind of symlink farm going on?
I didn't know that there could be a problem with cache and archivecache.
At the time we decided on the configuration for cyrus 3.0 I looked at the
imapd.conf man page and for metapartition_files decided that I want all
meta files on the ssd storage. There was no indication in the man page
that there could be a problem.
How do I separate location of archivecache from the other metapartition path?
And fix the cache and archivecache files?
No there is no sysmlink farm. We have mounted different iSCSI volumes to
/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
Otherwise it all looks OK. Are you getting other IOERRORs in your
log files which could show things aborting? It really looks like
your conversations DB is getting out of sync due to other failures.
I found a few instances of 3 related errors.
Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
user.XXXX 2185 failed to copyfile
(/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>
/srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code
____ 255
The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.
stat 2185.
File: '2185.'
Size: 424542 Blocks: 832 IO Block: 4096 regular file
Device: 860h/2144d Inode: 4677438462 Links: 1
Access: (0600/-rw-------) Uid: ( 96/ cyrus) Gid: ( 12/ mail)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-11-27 01:12:46.585864239 +0100
Modify: 2018-11-27 01:12:46.585864239 +0100
Change: 2018-11-27 01:12:46.585864239 +0100
Birth: -
Cheers,
Bron.
On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote:
Quoting Bron Gondwana <brong@xxxxxxxxxxx>:
> On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:
>> Hi Bron
>>
>>
>> Quoting Bron Gondwana <brong@xxxxxxxxxxx>:
>>
>> > Sorry I haven't been following along with this earlier. Can you post
>> > your imapd.conf and cyrus.conf as well as let her know if you run
>> > anything which directly messes with files on disk.
>>
>> Thanks for looking into it. I have attached the backend and replica
>> configs. There should be nothing messing with the files on disk
>> directly. Some times we need to restore mails from file based
>> backup (bacula) followed by a reconstruct but this was not the
>> case this time.
>
> Do you always rebuild the conversations db after doing the
> reconstruct? That will be necessary now. We switched to doing all
> our restores using IMAP append a while back so we're never fiddling
> the file system under Cyrus.
>
>> >
>> > Also, what operating system is this on and what Cyrus version?
>> >
>>
>> We are running a cyrus 3.0.8 compiled with the following Options
>> (./configure --enable-murder --enable-http --enable-calalarmd
>> --enable-replication --enable-backup --enable-idled
>> --enable-autocreate CFLAGS="-fPIC -g")
>> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.
>
> They should be fine. I'll have a read of the config when I'm at a
> real computer.
>
did you find anything in the config that would explain the problems?
>>
>> > Bron.
>> >
>> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
>> >> Hi,
>> >>
>> >> I have discovered an other problem with the conversations db:
>> >>
>> >> Thousends of lines with "IOERROR: conversations_audit on load:" and
>> >> "IOERROR: conversations_audit on store:"
>> >> A look at the source code shows that these errors are logged after
>> >> "_sanity_check_counts" is called.
>> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a
>> >> serious problem. Do I?
>> >>
>> >> This problem occurred for accounts where the rebuild of the
>> >> conversations db was successful.
>> >>
>> >> I don't want to rebuild the conversations db every few days.
>> >>
>> >> Any help is appreciated.
>> >>
>> >> Kind regards
>> >>
>> >>
>> >> Michael Menge
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>>
--------------------------------------------------------------------------------
>> >> M.Menge Tel.: (49) 7071/29-70316
>> >> Universität Tübingen Fax.: (49) 7071/29-5912
>> >> Zentrum für Datenverarbeitung mail:
>> >> michael.menge@xxxxxxxxxxxxxxxxxxxx
>> >> Wächterstraße 76
>> >> 72074 Tübingen
>> >>
>> >> ----
>> >> Cyrus Home Page: http://www.cyrusimap.org/
>> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> >> To Unsubscribe:
>> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> >
>> > --
>> > Bron Gondwana
>> > brong@xxxxxxxxxxx
>>
>>
>>
>>
--------------------------------------------------------------------------------
>> M.Menge Tel.: (49) 7071/29-70316
>> Universität Tübingen Fax.: (49) 7071/29-5912
>> Zentrum für Datenverarbeitung mail:
>> michael.menge@xxxxxxxxxxxxxxxxxxxx
>> Wächterstraße 76
>> 72074 Tübingen
>>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>> *Attachments:*
>> * imapd_be.conf
>> * cyrus_be.conf
>> * cyrus_re.conf
>> * imapd_re.conf
>
> --
> Bron Gondwana
> brong@xxxxxxxxxxx
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.menge@xxxxxxxxxxxxxxxxxxxx
Wächterstraße 76
72074 Tübingen
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana, CEO, FastMail Pty Ltd
brong@xxxxxxxxxxxxxxxx
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.menge@xxxxxxxxxxxxxxxxxxxx
Wächterstraße 76
72074 Tübingen
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
