On Mon, Jul 25, 2016 at 03:53:43PM +1000, ellie timoney wrote: > I don't have a 2.3.x environment handy to try to reproduce this from the > outside in, so I'm starting at the error code and working outwards... > > > > <1469384361<a0080 UID COPY 1162729 "Drafts" > > > >1469384362>a0080 NO Mailbox format corruption detected > > "Mailbox format corruption detected" corresponds with > IMAP_MAILBOX_CHECKSUM > > This error code is exclusively returned by functions in imap/mailbox.c, > for index/record crc mismatches. Given that the mailbox is selectable > and such, I'm supposing that the whole-file/whole-record crc checks are > okay... > > It looks like 2.3.16 had MAILBOX_MINOR_VERSION 10, and in particular > doesn't have the record->cache_crc field, which many of the > IMAP_MAILBOX_CHECKSUM checks are for. When reading mailboxes < 12 in > newer Cyrus versions, this field is initialised to zero. > > Some of the places that want to do a crc check against record->cache_crc > first make sure record->cache_crc is non-zero before doing so: > cache_append_record(), mailbox_cacherecord(). > > But some don't, or at least don't appear to: mailbox_append_cache() > initially uses cache_append_record() (good), but then reads it straight > back in to ensure freshness -- and then having done so, appears to do an > unconditional check against record->cache_crc (dodgy?) > > Actually that seems like all the spots this check happens -- so the fix > might be as simple as the attached patch? > > Kenneth, are you able to try this out? > > Cheers, > > ellie > Hi Ellie, I will try it this morning. Regards, Ken > On Mon, Jul 25, 2016, at 02:02 PM, Bron Gondwana wrote: > > Thanks for reporting this. Ellie, if you get a chance can you look at > > it? I'm in the middle of the the FastMail security rollout right now, so > > I can't do anything today. > > > > Bron. > > > > On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote: > > > Hi, > > > > > > I accidentally hijacked another thread. So here is a new message. I am > > > testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems > > > moving messages between folders before the 'reconstruct -V max' has been > > > run on the mailboxes. Here is the error from the IMAP client: > > > > > > imap_copy_messages [a0080 NO Mailbox format corruption detected]? > > > > > > and the details from the IMAP logs: > > > > > > <1469384341<a0077 NOOP > > > >1469384341>a0077 OK Completed > > > <1469384354<a0078 LIST "" "%" > > > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX > > > * LIST (\HasChildren) "/" DSPAM_notspam > > > * LIST (\HasChildren) "/" DSPAM_spam > > > * LIST (\HasNoChildren) "/" Drafts > > > * LIST (\HasChildren) "/" Mail > > > * LIST (\HasNoChildren) "/" Sent > > > * LIST (\HasNoChildren) "/" Spam > > > * LIST (\HasNoChildren) "/" Trash > > > a0078 OK Completed (0.240 secs 356 calls) > > > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY) > > > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846) > > > a0079 OK Completed > > > <1469384361<a0080 UID COPY 1162729 "Drafts" > > > >1469384362>a0080 NO Mailbox format corruption detected > > > <1469384370<a0081 LIST "" "%" > > > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX > > > * LIST (\HasChildren) "/" DSPAM_notspam > > > * LIST (\HasChildren) "/" DSPAM_spam > > > * LIST (\HasNoChildren) "/" Drafts > > > * LIST (\HasChildren) "/" Mail > > > * LIST (\HasNoChildren) "/" Sent > > > * LIST (\HasNoChildren) "/" Spam > > > * LIST (\HasNoChildren) "/" Trash > > > a0081 OK Completed (0.230 secs 356 calls) > > > <1469384372<a0082 CLOSE > > > a0083 MYRIGHTS "Drafts" > > > a0084 SELECT "Drafts" > > > >1469384372>a0082 OK Completed > > > >1469384372>* MYRIGHTS Drafts lrswipkxtecda > > > a0083 OK Completed > > > >1469384372>* 0 EXISTS > > > * 0 RECENT > > > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old) > > > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok > > > * OK [UIDVALIDITY 1345619846] Ok > > > * OK [UIDNEXT 6] Ok > > > * OK [HIGHESTMODSEQ 1] Ok > > > * OK [URLMECH INTERNAL] Ok > > > * OK [ANNOTATIONS 65536] Ok > > > a0084 OK [READ-WRITE] Completed > > > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX > > > * LIST (\HasChildren) "/" DSPAM_notspam > > > * LIST (\HasChildren) "/" DSPAM_spam > > > * LIST (\HasNoChildren) "/" Drafts > > > * LIST (\HasChildren) "/" Mail > > > * LIST (\HasNoChildren) "/" Sent > > > * LIST (\HasNoChildren) "/" Spam > > > * LIST (\HasNoChildren) "/" Trash > > > a0078 OK Completed (0.240 secs 356 calls) > > > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY) > > > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846) > > > a0079 OK Completed > > > <1469384361<a0080 UID COPY 1162729 "Drafts" > > > >1469384362>a0080 NO Mailbox format corruption detected > > > <1469384370<a0081 LIST "" "%" > > > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX > > > * LIST (\HasChildren) "/" DSPAM_notspam > > > * LIST (\HasChildren) "/" DSPAM_spam > > > * LIST (\HasNoChildren) "/" Drafts > > > * LIST (\HasChildren) "/" Mail > > > * LIST (\HasNoChildren) "/" Sent > > > * LIST (\HasNoChildren) "/" Spam > > > * LIST (\HasNoChildren) "/" Trash > > > a0081 OK Completed (0.230 secs 356 calls) > > > <1469384372<a0082 CLOSE > > > a0083 MYRIGHTS "Drafts" > > > a0084 SELECT "Drafts" > > > >1469384372>a0082 OK Completed > > > >1469384372>* MYRIGHTS Drafts lrswipkxtecda > > > a0083 OK Completed > > > >1469384372>* 0 EXISTS > > > * 0 RECENT > > > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old) > > > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok > > > * OK [UIDVALIDITY 1345619846] Ok > > > * OK [UIDNEXT 6] Ok > > > * OK [HIGHESTMODSEQ 1] Ok > > > * OK [URLMECH INTERNAL] Ok > > > * OK [ANNOTATIONS 65536] Ok > > > a0084 OK [READ-WRITE] Completed > > > <1469384392<a0085 NOOP > > > >1469384392>a0085 OK Completed > > > <1469384412<a0086 NOOP > > > >1469384412>a0086 OK Completed > > > > > > Is this a known issue with the upgrade process? Messages can be delivered to > > > the folders via a sieve script, but both message moves and copies fail. If there > > > is any fix for this issue, it would be greatly appreciated. I have been waiting > > > since the release of 2.4.x for a version that would handle the I/O storm caused > > > by automatically upgrading the mailbox format on first access, and 2.5.x does > > > manage that successfully. Thank you for any assistance. > > > > > > Regards, > > > Ken > > > > > > On Fri, Jul 22, 2016 at 02:27:01PM -0500, Kenneth Marshall wrote: > > > > On Sat, Jul 16, 2016 at 12:04:26AM +1000, Bron Gondwana via Info-cyrus wrote: > > > > > On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote: > > > > > > On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote: > > > > > > > So if the issue apparently lies with twoskip, can we keep our dbs using > > > > > > > skiplist and do the 2.4 -> 2.5 upgrade? Is it possible -h could revert > > > > > > > back to skiplist? > > > > > > > > > > > > You can convert a db back just by changing the setting in imapd.conf and restarting - it will convert itself. > > > > > > > > > > > > mboxlist_db: skiplist > > > > > > > > > > > > > If it could help we could test upgrading to 2.5.8 on our dev server > > > > > > > while leaving our database(s) as skiplist. > > > > > > > > > > > > Twoskip should be fixed in 2.5.9. I've worked out what went wrong, and am working on patches now :) > > > > > > > > > > Patches have passed all the tests - they're on master and 2.5. I'm pushing to FastMail's testing stores and then going to bed :) > > > > > > > > > > -- > > > > > Bron Gondwana > > > > > brong@xxxxxxxxxxx > > > > > ---- > > > > > > > > Hi Bron and Ellie, > > > > > > > > Thank you for the new release. I am currently testing my upgrade process and while > > > > I can access my mailboxes after the upgrade to 2.5.x but before the 'reconstruct -V max', > > > > I cannot move messages between mailboxes. When I try I get the following error: > > > > > > > > imap_copy_messages [a0012 NO Mailbox format corruption detected]? > > > > > > > > and the move fails. Once I have updated the folder formats with the reconstruct, > > > > it works, but this causes a lot of issues with the upgrade. Have you seen this > > > > before and is there anything that can be done to make it work before the > > > > reconstruct? > > > > > > > > Regards, > > > > Ken > > > *** HEADER EXTRACTED deferred/C/CB16F4219066 *** > > > *** MESSAGE FILE END deferred/C/CB16F4219066 *** > > > > > > -- > > Bron Gondwana > > brong@xxxxxxxxxxx > diff --git a/imap/mailbox.c b/imap/mailbox.c > index aa933b9..98a6c35 100644 > --- a/imap/mailbox.c > +++ b/imap/mailbox.c > @@ -610,6 +610,10 @@ int mailbox_append_cache(struct mailbox *mailbox, > record->cache_offset, &record->crec); > if (r) return r; > > + /* old-style record */ > + if (!record->cache_crc) > + return 0; > + > if (record->cache_crc != crc32_buf(cache_buf(record))) > return IMAP_MAILBOX_CHECKSUM; > ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
