Xapian initial index really that slow? - 3.6.beta3-2+

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I cloned a production server (zfs + lxc) to test the xapian search and started to squatter.

What I am experiencing is a very slow progress, especially comparing to the search_engine:squat.

Besides of the very slow indexing, the whole cyrus seems slowed down for the user, which is under indexing (squattering).

When I set the batch to 8192, I got longlock for around 120 seconds, with 1024 now I got about 18 seconds (numbers are not constants of course).

The full indexing supposed to slow with xapian engine?

Details:
Config file:
# Sync log - rolling squat
sync_log: on
sync_log_channels: indexer
search_engine: xapian
search_index_headers: no
search_batchsize: 1024
#search_batchsize: 8192
#search_indexed_db: twoskip
defaultsearchtier: t1
t1searchpartition-default: /var/spool/cyrus/search


As I checked the size of the search partition, I got 300GB, even squatter only started with 2 users (never finished any of them).
I have about 1.1 TByte email messages in total with about 50 users, so about 300 GB for 2 users seems very strange to me.

I guess, I really missed something really important here.

The steps were:
- cyrus-imapd stopped
- changing config files to meet the xapian requirements (config like above, rolling invocation of squatter in cyrus.conf:  cmd="squatter -R" )
  - (the squatter in daemon section was squatter -R -n indexer, but as I read in one email, -R and -n are not really friends
- start cyrus-imapd
- start an initial squatter for everybody:  /usr/lib/cyrus/bin/squatter -v -p 
  - stopping this squatter
- starting squatter for specific users
  - stoppping after about 24 hours
- starting again with another user
- realizing, something really not ok

I read the emails and manual before I wrote this email, because I still in doubt, how can I put this into production.
At this moment it is not possible to start to convert search engine to xapian, due to that, the initial indexing (which I guess necessary for old emails) takes a lot of time and technically block the email server to serve at least the client, whos emails are under squattering.

Could you provide me a link to a recent documentation and/or give me some hints?

Thanks,
István

[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux