Hi, do you have any idea of the possible next steps?
The only reason I'm stuck with squatter, is that on another cloned test machine I upgraded to latest cyrus with xapian, but while indexing seems to run fine, I actually found the cyrus imapd to be much slower on full text search with respect to squatter alone (almost as with a cyrus imapd with no squatter at all).
Is this possible? Is there any way I could test that the full text search was actually using the xapian index?
Thanks, I'm stuck upgrading our cloud to Cyrus 3, and still running hours of squatter indexing often segfaulting.... :(
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
Da: Gabriele Bulfon via Info <info@xxxxxxxxxxxxxxxxxx>
A: Info <info@xxxxxxxxxxxxxxxxxx>
Data: 13 maggio 2021 9.25.47 CEST
Oggetto: Re: Squatter core dump
Here is another valgrind log, with binary rebuilt with commented out free, infact the free calls are not present here.
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
Da: Robert Stepanek <rsto@xxxxxxxxxxxxxxxx>
A: Gabriele Bulfon <gbulfon@xxxxxxxxxxx>Gabriele Bulfon via Info <info@xxxxxxxxxxxxxxxxxx>
Data: 11 maggio 2021 13.54.40 CEST
Oggetto: Re: Squatter core dump
Yes, but unfortunately I didn't find time to do more than that. The log pretty much confirms that there's memory corruption, with invalid reads and writes all over the place.This line looks like a potential start to debug:==17293== Address 0xff28149a0 is 0 bytes inside a block of size 2,056 free'd==17293== at 0xFFFF64A29: free (vg_replace_malloc.c:549)==17293== by 0x407A28: write_trie_word_data (squat_build.c:1364)(interestingly, in my code the free happens at line 1363).As a very crude attempt, you could just comment out this free() call. It will leak memory, but if that omits the segfault then we have narrowed down what's causing the corruption.On Tue, May 11, 2021, at 1:38 PM, Gabriele Bulfon wrote:Hight, did you have any chance to check the valgrind log I've sent?Sonicle S.r.l. : http://www.sonicle.comeXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanetsDa: Robert Stepanek <rsto@xxxxxxxxxxxxxxxx>
A: Gabriele Bulfon via Info <info@xxxxxxxxxxxxxxxxxx>
Data: 7 maggio 2021 17.45.19 CEST
Oggetto: Re: Squatter core dumpOn Fri, May 7, 2021, at 5:31 PM, Gabriele Bulfon via Info wrote:Never did, it's our illumos distro, but I may have valgrind.Can you help on this?Just running "valgrind --leak-check=full <path-to-squatter> <your-args>" should do the trick.Cheers,Robert
==21038== Memcheck, a memory error detector ==21038== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==21038== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==21038== Command: squatter -v -i -s -a -r user/test.big@xxxxxxxxxxx ==21038== ==21038== Warning: invalid file descriptor -1 in syscall fstat() ==21038== Warning: invalid file descriptor -1 in syscall fstat() ==21038== Syscall param putmsg(ctrlptr->buf) points to uninitialised byte(s) ==21038== at 0xFFFEDD57A: __putmsg (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76AEA: vsyslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76702: syslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFCCFB6D: config_read (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0xFFFD46558: cyrus_init (in /sonicle/lib/amd64/libcyrus_imap.so.0.0.0) ==21038== by 0x4054D4: main (squatter.c:706) ==21038== Address 0x57fed660 is on thread 1's stack ==21038== in frame #1, created by vsyslog (???:) ==21038== ==21038== Syscall param putmsg(ctrlptr->buf) points to uninitialised byte(s) ==21038== at 0xFFFEDD57A: __putmsg (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76AEA: vsyslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76702: syslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFCCF8AC: config_read (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0xFFFD46558: cyrus_init (in /sonicle/lib/amd64/libcyrus_imap.so.0.0.0) ==21038== by 0x4054D4: main (squatter.c:706) ==21038== Address 0x57fed660 is on thread 1's stack ==21038== in frame #1, created by vsyslog (???:) ==21038== ==21038== Syscall param putmsg(ctrlptr->buf) points to uninitialised byte(s) ==21038== at 0xFFFEDD57A: __putmsg (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76AEA: vsyslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76702: syslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFCCF1C9: config_read (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0xFFFD46558: cyrus_init (in /sonicle/lib/amd64/libcyrus_imap.so.0.0.0) ==21038== by 0x4054D4: main (squatter.c:706) ==21038== Address 0x57fed660 is on thread 1's stack ==21038== in frame #1, created by vsyslog (???:) ==21038== ==21038== Syscall param putmsg(ctrlptr->buf) points to uninitialised byte(s) ==21038== at 0xFFFEDD57A: __putmsg (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76AEA: vsyslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76702: syslog (in /lib/amd64/libc.so.1) ==21038== by 0x4054E8: main (squatter.c:708) ==21038== Address 0x57fee710 is on thread 1's stack ==21038== in frame #1, created by vsyslog (???:) ==21038== ==21038== Syscall param putmsg(ctrlptr->buf) points to uninitialised byte(s) ==21038== at 0xFFFEDD57A: __putmsg (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76AEA: vsyslog (in /lib/amd64/libc.so.1) ==21038== by 0xFFFE76702: syslog (in /lib/amd64/libc.so.1) ==21038== by 0x4052F2: index_me (squatter.c:631) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0x57fede10 is on thread 1's stack ==21038== in frame #1, created by vsyslog (???:) ==21038== Indexing mailbox user/test.big@xxxxxxxxxxx... Doing incremental for mailbox sonicle.com!user.test^big... ==21038== Warning: set address range perms: large range [0xfa3a38040, 0xfca7c8ea0) (undefined) ==21038== Warning: set address range perms: large range [0xfa3a38028, 0xfca7c8eb8) (noaccess) ==21038== Warning: set address range perms: large range [0xf882ca040, 0xfa006bad0) (undefined) ==21038== Warning: set address range perms: large range [0xf882ca028, 0xfa006bae8) (noaccess) ==21038== Warning: set address range perms: large range [0xf76a42040, 0xf882c97c0) (undefined) ==21038== Warning: set address range perms: large range [0xf76a42028, 0xf882c97d8) (noaccess) ==21038== Warning: set address range perms: large range [0xf97053040, 0xfaa71eec0) (undefined) ==21038== Warning: set address range perms: large range [0xf97053028, 0xfaa71eed8) (noaccess) ==21038== Warning: set address range perms: large range [0xf82fbe040, 0xf97052510) (undefined) ==21038== Warning: set address range perms: large range [0xf82fbe028, 0xf97052528) (noaccess) ==21038== Warning: set address range perms: large range [0xf68e45040, 0xf82fbdeb0) (undefined) ==21038== Warning: set address range perms: large range [0xf68e45028, 0xf82fbdec8) (noaccess) ==21038== Warning: set address range perms: large range [0xf93451040, 0xfa542a1e0) (undefined) ==21038== Warning: set address range perms: large range [0xf93451028, 0xfa542a1f8) (noaccess) ==21038== Warning: set address range perms: large range [0xf81825040, 0xf93450e10) (undefined) ==21038== Warning: set address range perms: large range [0xf81825028, 0xf93450e28) (noaccess) ==21038== Warning: set address range perms: large range [0xf6fa6b040, 0xf81824870) (undefined) ==21038== Warning: set address range perms: large range [0xf6fa6b028, 0xf81824888) (noaccess) ==21038== Warning: set address range perms: large range [0xf8a679040, 0xfa302a930) (undefined) ==21038== Warning: set address range perms: large range [0xf8a679028, 0xfa302a948) (noaccess) ==21038== Warning: set address range perms: large range [0xf7a5a0040, 0xf8a678a20) (undefined) ==21038== Warning: set address range perms: large range [0xf7a5a0028, 0xf8a678a38) (noaccess) ==21038== Warning: set address range perms: large range [0xf928a6040, 0xfa302a5f0) (undefined) ==21038== Warning: set address range perms: large range [0xf928a6028, 0xfa302a608) (noaccess) ==21038== Warning: set address range perms: large range [0xf7a5d6040, 0xf8f2eea50) (undefined) ==21038== Warning: set address range perms: large range [0xf7a5d6028, 0xf8f2eea68) (noaccess) ==21038== Warning: set address range perms: large range [0xf6637b040, 0xf7a5d5950) (undefined) ==21038== Warning: set address range perms: large range [0xf6637b028, 0xf7a5d5968) (noaccess) ==21038== Warning: set address range perms: large range [0xf50068040, 0xf6637a4a0) (undefined) ==21038== Warning: set address range perms: large range [0xf50068028, 0xf6637a4b8) (noaccess) ==21038== Warning: set address range perms: large range [0xf78934040, 0xf8f2ee260) (undefined) ==21038== Warning: set address range perms: large range [0xf78934028, 0xf8f2ee278) (noaccess) ==21038== Warning: set address range perms: large range [0xf65027040, 0xf78933f00) (undefined) ==21038== Warning: set address range perms: large range [0xf65027028, 0xf78933f18) (noaccess) ==21038== Warning: set address range perms: large range [0xf6878f040, 0xf7f6cd6b0) (undefined) ==21038== Warning: set address range perms: large range [0xf6878f028, 0xf7f6cd6c8) (noaccess) ==21038== Warning: set address range perms: large range [0xf54249040, 0xf6878e020) (undefined) ==21038== Warning: set address range perms: large range [0xf54249028, 0xf6878e038) (noaccess) ==21038== Warning: set address range perms: large range [0xf77493040, 0xf8f2eea30) (undefined) ==21038== Warning: set address range perms: large range [0xf77493028, 0xf8f2eea48) (noaccess) ==21038== Warning: set address range perms: large range [0xf64cb7040, 0xf774925e0) (undefined) ==21038== Warning: set address range perms: large range [0xf64cb7028, 0xf774925f8) (noaccess) ==21038== Warning: set address range perms: large range [0xf7dd66040, 0xf8f2eeae0) (undefined) ==21038== Warning: set address range perms: large range [0xf7dd66028, 0xf8f2eeaf8) (noaccess) ==21038== Warning: set address range perms: large range [0xf6d008040, 0xf7dd65060) (undefined) ==21038== Warning: set address range perms: large range [0xf6d008028, 0xf7dd65078) (noaccess) ==21038== Warning: set address range perms: large range [0xf7eca4040, 0xf8f2ee310) (undefined) ==21038== Warning: set address range perms: large range [0xf7eca4028, 0xf8f2ee328) (noaccess) ==21038== Warning: set address range perms: large range [0xf6dfc6040, 0xf7eca36d0) (undefined) ==21038== Warning: set address range perms: large range [0xf6dfc6028, 0xf7eca36e8) (noaccess) ==21038== Warning: set address range perms: large range [0xf5da88040, 0xf6dfc5a70) (undefined) ==21038== Warning: set address range perms: large range [0xf5da88028, 0xf6dfc5a88) (noaccess) ==21038== Warning: set address range perms: large range [0xf4e66a000, 0xf7e82a000) (defined) ==21038== Invalid write of size 4 ==21038== at 0x4067A7: add_word_to_trie (squat_build.c:790) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0xf98d86fc8 is 8 bytes after a block of size 9,060,224 alloc'd ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x407BDC: dump_index_trie_words (squat_build.c:1415) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== Invalid write of size 8 ==21038== at 0x406752: add_to_table (squat_build.c:770) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0xf98d86fc0 is 0 bytes after a block of size 9,060,224 alloc'd ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x407BDC: dump_index_trie_words (squat_build.c:1415) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== Invalid write of size 8 ==21038== at 0x40672F: add_to_table (squat_build.c:763) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0xf98d87020 is 32 bytes before a block of size 2,064 in arena "client" ==21038== ==21038== Invalid read of size 8 ==21038== at 0x40674B: add_to_table (squat_build.c:770) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0xf98d86fc0 is 0 bytes after a block of size 9,060,224 alloc'd ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x407BDC: dump_index_trie_words (squat_build.c:1415) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== Invalid write of size 8 ==21038== at 0x406760: add_to_table (squat_build.c:773) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== Address 0xf98d86fc0 is 0 bytes after a block of size 9,060,224 alloc'd ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x407BDC: dump_index_trie_words (squat_build.c:1415) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== ==21038== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==21038== Access not within mapped region at address 0xF99D87008 ==21038== at 0x4067A7: add_word_to_trie (squat_build.c:790) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== If you believe this happened as a result of a stack ==21038== overflow in your program's main thread (unlikely but ==21038== possible), you can try to increase the size of the ==21038== main thread stack using the --main-stacksize= flag. ==21038== The main thread stack size used in this run was 10485760. ==21038== ==21038== HEAP SUMMARY: ==21038== in use at exit: 1,874,622,708 bytes in 904,867 blocks ==21038== total heap usage: 175,985,562 allocs, 175,080,695 frees, 26,363,703,442 bytes allocated ==21038== ==21038== 1,054,728 bytes in 513 blocks are possibly lost in loss record 51 of 57 ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x40668E: add_to_table (squat_build.c:744) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== 1,824,843,920 (178,783,592 direct, 1,646,060,328 indirect) bytes in 86,957 blocks are definitely lost in loss record 57 of 57 ==21038== at 0xFFFF6529F: malloc (vg_replace_malloc.c:319) ==21038== by 0xFFFCD69FE: xmalloc (in /sonicle/lib/amd64/libcyrus_min.so.0.0.0) ==21038== by 0x40668E: add_to_table (squat_build.c:744) ==21038== by 0x4067C5: add_word_to_trie (squat_build.c:791) ==21038== by 0x407CBD: dump_index_trie_words (squat_build.c:1449) ==21038== by 0x408153: index_close_internal (squat_build.c:1589) ==21038== by 0x4085A4: squat_index_finish (squat_build.c:1695) ==21038== by 0x404D73: squat_single (squatter.c:484) ==21038== by 0x405330: index_me (squatter.c:636) ==21038== by 0x4056ED: main (squatter.c:746) ==21038== ==21038== LEAK SUMMARY: ==21038== definitely lost: 178,783,592 bytes in 86,957 blocks ==21038== indirectly lost: 1,646,060,328 bytes in 800,613 blocks ==21038== possibly lost: 1,054,728 bytes in 513 blocks ==21038== still reachable: 48,724,060 bytes in 16,784 blocks ==21038== suppressed: 0 bytes in 0 blocks ==21038== Reachable blocks (those to which a pointer was found) are not shown. ==21038== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==21038== ==21038== Use --track-origins=yes to see where uninitialised values come from ==21038== For lists of detected and suppressed errors, rerun with: -s ==21038== ERROR SUMMARY: 147418 errors from 12 contexts (suppressed: 0 from 0)
