On Fri, 21 Jan 2022, at 1:56 PM, Chis Walker wrote:
I've looked a bit more closely at the telemetry from Thunderbird and it never does a "UID SEARCH". Instead, it does a "UID FETCH 1:*" and Cyrus successfully returns all of the messages that are in the user's inbox.
That's not a search, that's just the normal way of fetching messages over IMAP. It doesn't require there to be search indexes, and everything a client needs to use this style of interaction is in the SELECT response a few lines up.
SOGo/Android/Apple are all executing a "UID SEARCH beginning_msg_index:ending_msg_index" to get a list of message IDs.
Yeah, these clients are unusual in that they seem to be ignoring the information in the SELECT response, and just using a SEARCH to find messages instead. Presumably, having a list of messages, they would then FETCH them, but they don't get that far because their search comes back empty.
For some strange reason, Cyrus just blindly fires back a "* SEARCH"; telling the client that there's nothing there. I'm thinking these message indexes should map to actual file names on disk.
It sounds like you still don't have search indexes; or maybe you do now, but they're empty.
I now have something to grep on. I'll start looking for "* SEARCH" in the source and see what Cyrus is doing. Maybe there are some hints in the code on what settings I need in order for UID SEARCHes to succeed.
You might be getting confused because there are a couple of kinds of indexes. The "cyrus.index" file you see on disk in each mailbox, and which reconstruct deals with, is NOT a search index. Well, it is, but it's not one that's useful to the SEARCH command specifically.
The keyword you want to look for is "xapian", though I'm having trouble finding any single comprehensive "how to set up search" document. I thought something like that existed, but maybe it doesn't, and it really should. It might be written up somewhere on this mailing list and just never made it to the actual documentation.
http://www.cyrusimap.org/imap/reference/admin/locations/searchtiers.html talks a bit about how to set up "multiple search tiers" -- I believe you have to set up _at least one_ search tier; whether you take that further is up to you. These options are described as "this option MUST be specified for xapian search" -- but note that these days xapian search is _the only search_.
There's also http://www.cyrusimap.org/imap/reference/admin/sop/squatter.html which is a bit vague, but contains the detail that, having configured where your indexes are stored, you need to run squatter to actually build them once, and need to rerun squatter regularly to keep them up to date. The squatter(8) man page suggests either running it periodically from the EVENTS block (in cyrus.conf), or setting it up in rolling mode. The rolling mode uses the same mechanism as rolling replication, listening on a sync_log channel for notification of mailboxes that have had updates so it can re-index them in close to real time.
Given you have clients that bypass the normal IMAP workflow and rely entirely on SEARCH instead, you probably want the rolling updates. If you only ran it periodically, they would only see new mail appear at the same interval.
Cheers,
ellie
