I don't have a 2.3.x environment handy to try to reproduce this from the
outside in, so I'm starting at the error code and working outwards...
> > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > >1469384362>a0080 NO Mailbox format corruption detected
"Mailbox format corruption detected" corresponds with
IMAP_MAILBOX_CHECKSUM
This error code is exclusively returned by functions in imap/mailbox.c,
for index/record crc mismatches. Given that the mailbox is selectable
and such, I'm supposing that the whole-file/whole-record crc checks are
okay...
It looks like 2.3.16 had MAILBOX_MINOR_VERSION 10, and in particular
doesn't have the record->cache_crc field, which many of the
IMAP_MAILBOX_CHECKSUM checks are for. When reading mailboxes < 12 in
newer Cyrus versions, this field is initialised to zero.
Some of the places that want to do a crc check against record->cache_crc
first make sure record->cache_crc is non-zero before doing so:
cache_append_record(), mailbox_cacherecord().
But some don't, or at least don't appear to: mailbox_append_cache()
initially uses cache_append_record() (good), but then reads it straight
back in to ensure freshness -- and then having done so, appears to do an
unconditional check against record->cache_crc (dodgy?)
Actually that seems like all the spots this check happens -- so the fix
might be as simple as the attached patch?
Kenneth, are you able to try this out?
Cheers,
ellie
On Mon, Jul 25, 2016, at 02:02 PM, Bron Gondwana wrote:
> Thanks for reporting this. Ellie, if you get a chance can you look at
> it? I'm in the middle of the the FastMail security rollout right now, so
> I can't do anything today.
>
> Bron.
>
> On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote:
> > Hi,
> >
> > I accidentally hijacked another thread. So here is a new message. I am
> > testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
> > moving messages between folders before the 'reconstruct -V max' has been
> > run on the mailboxes. Here is the error from the IMAP client:
> >
> > imap_copy_messages [a0080 NO Mailbox format corruption detected]?
> >
> > and the details from the IMAP logs:
> >
> > <1469384341<a0077 NOOP
> > >1469384341>a0077 OK Completed
> > <1469384354<a0078 LIST "" "%"
> > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0078 OK Completed (0.240 secs 356 calls)
> > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > a0079 OK Completed
> > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > >1469384362>a0080 NO Mailbox format corruption detected
> > <1469384370<a0081 LIST "" "%"
> > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0081 OK Completed (0.230 secs 356 calls)
> > <1469384372<a0082 CLOSE
> > a0083 MYRIGHTS "Drafts"
> > a0084 SELECT "Drafts"
> > >1469384372>a0082 OK Completed
> > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > a0083 OK Completed
> > >1469384372>* 0 EXISTS
> > * 0 RECENT
> > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> > * OK [UIDVALIDITY 1345619846] Ok
> > * OK [UIDNEXT 6] Ok
> > * OK [HIGHESTMODSEQ 1] Ok
> > * OK [URLMECH INTERNAL] Ok
> > * OK [ANNOTATIONS 65536] Ok
> > a0084 OK [READ-WRITE] Completed
> > >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0078 OK Completed (0.240 secs 356 calls)
> > <1469384361<a0079 STATUS "Drafts" (UIDVALIDITY)
> > >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> > a0079 OK Completed
> > <1469384361<a0080 UID COPY 1162729 "Drafts"
> > >1469384362>a0080 NO Mailbox format corruption detected
> > <1469384370<a0081 LIST "" "%"
> > >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> > * LIST (\HasChildren) "/" DSPAM_notspam
> > * LIST (\HasChildren) "/" DSPAM_spam
> > * LIST (\HasNoChildren) "/" Drafts
> > * LIST (\HasChildren) "/" Mail
> > * LIST (\HasNoChildren) "/" Sent
> > * LIST (\HasNoChildren) "/" Spam
> > * LIST (\HasNoChildren) "/" Trash
> > a0081 OK Completed (0.230 secs 356 calls)
> > <1469384372<a0082 CLOSE
> > a0083 MYRIGHTS "Drafts"
> > a0084 SELECT "Drafts"
> > >1469384372>a0082 OK Completed
> > >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> > a0083 OK Completed
> > >1469384372>* 0 EXISTS
> > * 0 RECENT
> > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old \*)] Ok
> > * OK [UIDVALIDITY 1345619846] Ok
> > * OK [UIDNEXT 6] Ok
> > * OK [HIGHESTMODSEQ 1] Ok
> > * OK [URLMECH INTERNAL] Ok
> > * OK [ANNOTATIONS 65536] Ok
> > a0084 OK [READ-WRITE] Completed
> > <1469384392<a0085 NOOP
> > >1469384392>a0085 OK Completed
> > <1469384412<a0086 NOOP
> > >1469384412>a0086 OK Completed
> >
> > Is this a known issue with the upgrade process? Messages can be delivered to
> > the folders via a sieve script, but both message moves and copies fail. If there
> > is any fix for this issue, it would be greatly appreciated. I have been waiting
> > since the release of 2.4.x for a version that would handle the I/O storm caused
> > by automatically upgrading the mailbox format on first access, and 2.5.x does
> > manage that successfully. Thank you for any assistance.
> >
> > Regards,
> > Ken
> >
> > On Fri, Jul 22, 2016 at 02:27:01PM -0500, Kenneth Marshall wrote:
> > > On Sat, Jul 16, 2016 at 12:04:26AM +1000, Bron Gondwana via Info-cyrus wrote:
> > > > On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote:
> > > > > On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> > > > > > So if the issue apparently lies with twoskip, can we keep our dbs using
> > > > > > skiplist and do the 2.4 -> 2.5 upgrade? Is it possible -h could revert
> > > > > > back to skiplist?
> > > > >
> > > > > You can convert a db back just by changing the setting in imapd.conf and restarting - it will convert itself.
> > > > >
> > > > > mboxlist_db: skiplist
> > > > >
> > > > > > If it could help we could test upgrading to 2.5.8 on our dev server
> > > > > > while leaving our database(s) as skiplist.
> > > > >
> > > > > Twoskip should be fixed in 2.5.9. I've worked out what went wrong, and am working on patches now :)
> > > >
> > > > Patches have passed all the tests - they're on master and 2.5. I'm pushing to FastMail's testing stores and then going to bed :)
> > > >
> > > > --
> > > > Bron Gondwana
> > > > brong@xxxxxxxxxxx
> > > > ----
> > >
> > > Hi Bron and Ellie,
> > >
> > > Thank you for the new release. I am currently testing my upgrade process and while
> > > I can access my mailboxes after the upgrade to 2.5.x but before the 'reconstruct -V max',
> > > I cannot move messages between mailboxes. When I try I get the following error:
> > >
> > > imap_copy_messages [a0012 NO Mailbox format corruption detected]?
> > >
> > > and the move fails. Once I have updated the folder formats with the reconstruct,
> > > it works, but this causes a lot of issues with the upgrade. Have you seen this
> > > before and is there anything that can be done to make it work before the
> > > reconstruct?
> > >
> > > Regards,
> > > Ken
> > *** HEADER EXTRACTED deferred/C/CB16F4219066 ***
> > *** MESSAGE FILE END deferred/C/CB16F4219066 ***
>
>
> --
> Bron Gondwana
> brong@xxxxxxxxxxx
diff --git a/imap/mailbox.c b/imap/mailbox.c
index aa933b9..98a6c35 100644
--- a/imap/mailbox.c
+++ b/imap/mailbox.c
@@ -610,6 +610,10 @@ int mailbox_append_cache(struct mailbox *mailbox,
record->cache_offset, &record->crec);
if (r) return r;
+ /* old-style record */
+ if (!record->cache_crc)
+ return 0;
+
if (record->cache_crc != crc32_buf(cache_buf(record)))
return IMAP_MAILBOX_CHECKSUM;
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
