Hi,
I cloned a production server (zfs + lxc) to test the xapian search and started to squatter.
What I am experiencing is a very slow progress, especially comparing to the search_engine:squat.
Besides of the very slow indexing, the whole cyrus seems slowed down for the user, which is under indexing (squattering).
When I set the batch to 8192, I got longlock for around 120 seconds, with 1024 now I got about 18 seconds (numbers are not constants of course).
The full indexing supposed to slow with xapian engine?
Details:
Config file:
# Sync log - rolling squat
sync_log: on
sync_log_channels: indexer
search_engine: xapian
search_index_headers: no
search_batchsize: 1024
#search_batchsize: 8192
#search_indexed_db: twoskip
defaultsearchtier: t1
t1searchpartition-default: /var/spool/cyrus/search
As I checked the size of the search partition, I got 300GB, even squatter only started with 2 users (never finished any of them).
I have about 1.1 TByte email messages in total with about 50 users, so about 300 GB for 2 users seems very strange to me.
I guess, I really missed something really important here.
The steps were:
- cyrus-imapd stopped
- changing config files to meet the xapian requirements (config like above, rolling invocation of squatter in cyrus.conf: cmd="squatter -R" )
- (the squatter in daemon section was squatter -R -n indexer, but as I read in one email, -R and -n are not really friends
- start cyrus-imapd
- start an initial squatter for everybody: /usr/lib/cyrus/bin/squatter -v -p
- stopping this squatter
- starting squatter for specific users
- stoppping after about 24 hours
- starting again with another user
- realizing, something really not ok
I read the emails and manual before I wrote this email, because I still in doubt, how can I put this into production.
At this moment it is not possible to start to convert search engine to xapian, due to that, the initial indexing (which I guess necessary for old emails) takes a lot of time and technically block the email server to serve at least the client, whos emails are under squattering.
Could you provide me a link to a recent documentation and/or give me some hints?
Thanks,
István
