Hi
Quoting ellie timoney <ellie@xxxxxxxxxxxx>:
...
On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote:
...
Quoting ellie timoney <ellie@xxxxxxxxxxxx>:
...
> On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
>> Hi,
>>
>> because conversations db seems to be required for search indexes, I
>> enabled this option
>> on our production servers today and tried to rebuild the conversations
>> db for all users with
>>
>> ctl_conversationsdb -v -b UID
>>
>> For most users this did take less than a second. But for some users
>> this process would
>> not finish. I did kill one process after about 30 Minutes (most others
>> after 3 Minutes).
>> The UserID.conversations has grown over 2 GB (the mailbox itself has
>> only ~700 MB of mails,
>> and the conversations files from finnished rebuilds are less then 20 MB)
...
I did run ctl_conversationsd -b with strace and discovered that the inbox
(cyrus.index O_RDWR, cyrus.header O_RDONLY, cache cyrus.O_RDWR )
of that user
was opened over and over again, no other folder for that user was accessed.
For other users the inbox was only opened once and then the other
folders followed.
So not stopping "ctl_conversationsd -b", it would have run till my
filesystem was
full.
I guess something about that inbox is confusing it, and making it
redo it over and over? Very curious
...
> Though, that raises the question: was the user accessing the mailbox
> while the rebuild was in progress? I wonder if their client is
> doing something funny and tripping things up?
We did this while the server was accessible (didn't want to have
several hour downtime for ~44000 users)
Fair enough!
But as I did my recent testing on a test-server without user access,
only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during
the conversation rebuild mentioned in this mail. So I don't think that
user access is part of the problem.
Thanks for testing that, good to rule it out as a culprit
So for me it looks like we have two problems:
1. multiple entries for the same key in skiplist db files.
As skiplist is a Key/Value store this should not be possible
to have duplicate keys?
I don't know the internals of our database formats well, but I
_think_ maybe it writes new key/value pairs by appending to the end
of the database, and then updating the skiplist structure to point
to the newer version instead of the older version. If this is
correct, then if it's rewriting the keys over and over for some
reason, I would expect the raw file to contain a lot of old,
unlinked records for the same key, but which aren't visible over the
API. I believe there's a process that runs occasionally that
rewrites these databases with only the "live" keys and gets back the
wasted space?
I did discover, because of the loop (see below) the point of
database commit was not reached, so I guess that could explain
that there was no duplicated key cleanup done.
Is twoskip a better alternative?
This is before my time, but twoskip was written to solve problems
FastMail experienced with using skiplist with conversations
("post-crash recovery was too slow"). So if there's a pathological
case in the more recent conversations code that trips a bug in
skiplist, we probably wouldn't see it. So, I'd recommend using
twoskip for conversationsdb just based on that :)
Same problem with twoskip, but as I said above, the commit point
was not reached.
2. Why didn't ctl_conversationsdb continue to process
the next mailbox but re-did the INBOX / the last mailbox
It kinda sounds to me like it considered the operation to be
unsuccessful, and tried again? Not sure what would cause this. Is
there anything interesting in syslog?
I will try to run a debugger on ctl_conversationsdb and
will test with twoskip as conversations db format
The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)
For the problematic mailbox that I tested, for every message
record->cid was NULLCONVERSATION, so mailbox_cacherecord,
message_update_conversations and mailbox_rewrite_index_record
where called, each returned 0.
After iterating trough all messages in the mailbox count was > 0, and r==0,
so the while condition (!r && count) was true for the next run.
At this point record->cid was still NULLCONVERSATION for every message,
which I guess should not be the case.
Thanks, it'd be very interesting to see if this issue reproduces
with the twoskip format!
Yes, it did.
I did notice one other problem, recreating the conversations db
on the replica confused the sync protocol as the rebuild did increase
the modseq.
Do you mean to say that this user's conversations db could be
rebuilt successfully on the replica? Or was this with one of the
"good" users?
I did activated the conversation db first on the replica (to watch the
performance
impact and time needed) and did the rebuild with the -r flag.
I didn't notice an endless loop on the replic, but remember the creation of
the conversation db failed for some users and I had no good way to know for
which user the process was successful and for which it failed I decided to
run a single process for each user and check the return value.
I will test if the -r flag changes the rebuild on my test-server.
...
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.menge@xxxxxxxxxxxxxxxxxxxx
Wächterstraße 76
72074 Tübingen
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
