Hi,
Sorry, I will update the issue on github, as the situation is more weird, than expected.
I removed that message directly, reconstructed, squatter -> it dies with assertion failed, but the message earlier.
Maybe the next one is problematic?
To be continue on github :)
István
2022. 11. 28, hétfő keltezéssel 16.02-kor Pongrácz István ezt írta:
Hi,I just found a specific message, which cause the assertion failed issue for me.It seems, this kind of issue cannot be solved using the reconstruct, because technically it supposed to be valid email.I know, because I sent that email some days ago from outside to test something else.I reported here:Cheers,István2022. 11. 28, hétfő keltezéssel 14.58-kor Pongrácz István ezt írta:Hi,Just a quick update about the progress, regarding to this topic and give hint to others in similat situation.The standard squatter invocation still stopped, due to unknown issues, as described earlier.It seems I am able to fix this, using reconstruct for the problematic mailboxes, like this, as user cyrus:/usr/lib/cyrus/bin/reconstruct -r -x -u pongracziUsing the formula shown above, it seems I was able to fix 3 user's mailboxes already.The reported problems were, until now:
- user.pongraczi.Archives.2013 uid 3197 not found (there were several emails - 8 - where uid not found)
- some mails just missing (I assume I deleted them some weeks ago, but I forgot and I did not reconstruct yet)
- one had wrong owner and permission (my bad, as root I moved around that file, so, the reason was me in one specific case)
I used the following procedure to catch problematic mailboxes:
- I used the strace to get logs about opened files (mails):
strace -o squat2.log /usr/lib/cyrus/bin/squatter- when squattering stopped, due to problem, I was able to get the last message and maybe got a hint about the issue by grepping the log like this:
cat squat2.log | grep "openat(AT_FDCWD" | tail
the last line in my case: openat(AT_FDCWD, "/var/spool/cyrus/mail/uuid/j/9/j9x6ytfm3ain3r60yd9zcemw/469.", O_RDONLY) = 89
So, I got the file exactly the squattering died.
Some cases I got ACCESS DENIED (was root:root instead of cyrus:mail) or no such file or directory messages- After reconstructing the user's mailbox using -r -x the squatter was able to squat these and move to the next mailbox instead of dying.
Probably just reconstructing the full mail hierarchy using -r -x without any squatter testing could work and takes minimum time.If you need more information, please let me know.Cheers,István2022. 11. 14, hétfő keltezéssel 07.39-kor Pongrácz István ezt írta:Hi,Unfortunately not, but I cannot locate the specific emails, which cause this.I cannot provide too much useful information.It seems, when I put a lot of v in the options of squatter, it can squatter all the messages, at least there is no assertion and it did not stop.# incremental index changed mailboxes (fulltext) approximately every hoursquatter1 cmd="/usr/sbin/cyrus squatter -vvvvv -p -i" period=180# reindex all mailboxes (fulltext) dailysquattera cmd="/usr/sbin/cyrus squatter -vvvvv -p" at=0130In the other hand, the log contains additional info, around the problematic folders.Like this:2022-11-13T01:32:11.937642+01:00 cyrus36 cyrus/squatter[19126]: indexing mailbox user/csaba/Sent...2022-11-13T01:32:12.071267+01:00 cyrus36 cyrus/squatter[19126]: squat: opening document part ''2022-11-13T01:32:12.071595+01:00 cyrus36 cyrus/squatter[19126]: squat: writing 10 bytes into message 320912022-11-13T01:32:12.078024+01:00 cyrus36 cyrus/squatter[19126]: squat: opening document part 'h32091'2022-11-13T01:32:12.078185+01:00 cyrus36 cyrus/squatter[19126]: squat: writing 27 bytes into message 320912022-11-13T01:32:12.083676+01:00 cyrus36 cyrus/squatter[19126]: squat: opening document part 'f32092'2022-11-13T01:32:12.083837+01:00 cyrus36 cyrus/squatter[19126]: squat: writing 44 bytes into message 320922022-11-13T01:32:12.087780+01:00 cyrus36 cyrus/squatter[19126]: squat: opening document part 't32092'2022-11-13T01:32:12.088028+01:00 cyrus36 cyrus/squatter[19126]: squat: writing 48 bytes into message 320922022-11-13T01:32:12.090078+01:00 cyrus36 cyrus/squatter[19126]: squat: opening document part 's32092'(red part is suspicious for me)But I do not find 1:1 between the additional logs and assertion problems: seems more logs than assertion. (Maybe all of them could cause assertion, but squatter normally dies at the first one/account, so, the rest is invisible at this point.)And unfortunately I was not able to find the exact mails, which caused that.Using mbexamine, I thought I found some lead, but the theory failed when I tried to prove (emails with similar symptoms could be squattered).I had no time to modify the squatter sourcecode with some more debug (print the message id/filename) and compiling, yet.I have the idea to print all filenames which opened for squattering and where the assertion happened, I got the problematic email.It is still a theory.Do you think, I could any useful information at this moment?Thanks,István2022. 11. 14, hétfő keltezéssel 10.43-kor ellie timoney ezt írta:Hi István,I haven't seen a bug report about this come through GitHub yet, did you end up solving the problem yourself?Cheers,ellieOn Tue, 8 Nov 2022, at 11:51 PM, Pongrácz István wrote:Thank you, I check it!cheers,István2022. 11. 8, kedd keltezéssel 12.01-kor Nic Bernstein ezt írta:I would suggest exploring a bit with 'mbexamine(8)' to see what it shows. See the man page here:Specifically, if you have either the UID or the sequence number of a message, you may use the "-u uid" or "-s seqnum" options to narrow down the output.If you then decide to create a bug report, the output from mbexamine(8) will be of use.Cheers,-nicOn 11/8/22 11:55, Sebastian Hagedorn wrote:My suggestion would be to create a bug report on GitHub.
Cheers,SebastianOn 8 Nov 2022, at 12:47, Pongracz Istvan wrote:
Hi,I tried to reconstruct one of the problematic account with this:/usr/lib/cyrus/bin/reconstruct -r -f -R -u pongracziDidn't work, squatter still assert failured:cyrus/squatter[12044]: ERROR: message has more than 1000 header lines not caching any morefatal error: Internal error: assertion failed: imap/squat_internal.c: 134: v64 >= 0In case of 2 out of 7 assertion issues, that more than 1000 header lines exists, but the rest (5) there is no other information, only the folder name available.Any idea?Thank you!IStvánPongrácz István <pongracz.istvan@xxxxxxxxx> ezt írta (időpont: 2022. nov. 7., H, 16:13):Hi,We use the cyrus version 3.6.0-beta3-1+b1 and I found an issue with squatter.In some accounts there are a folder, where the squatter dies with fatal error and the whole squatter process cannot move on.I mean, there are about 40 accounts, called a.... to v....The squatter starts and can finish about 4 accounts, up to user 'foo' where it run into a folder, I assume it found an email and id dies with the following message:process type:EVENT name:squatter1 path:/usr/sbin/cyrus age:138.664s pid:31781 exited, status 70When I issue the command in command line, as cyrus user/usr/lib/cyrus/bin/squatter -v -p -u fooI got the following, more specific error message:Indexing mailbox user/foo/Archives/2013... fatal error: Internal error: assertion failed: imap/squat_internal.c: 134: v64 >= 0As I checked the squatter_internal.c it did not change for years (github).The problem with this, the periodic squatter just dies in the very beginning and other accounts never will be squattered.This server populated using recent imapsync from old cyrus server.The used squatter is: squat (not xapian).Now I have to run squatter "manually", directly for every user to get search function usable, more or less.(using the formula: /usr/lib/cyrus/bin/squatter -v -p -u foo )Do you know any kind of trick or method, how to eliminate/solve this issue?Thank you!István
