Re: Recomendations for a Migration of a Cyrus mailStore with 70K users.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We just finished migrating using 12 spools ext4 RAID5 for much less users (6000) with quotas up to 3Gb.
It tooks a few days on line just rename mailboxes to another spool.
my script looks like this. Just take care no to abort it in the middle of a rename. It would be better to add a handler for SIGUP/SIGTERM to exit properly.
New spool are sp0, sp1 etc.. I check if it's an old spool then rename mailboxes.

Hope this helps

Dom

#!/usr/bin/perl

use Cyrus::IMAP::Admin;
my $DEBUG=0;
my $quota=3072000; # 3 000 Mo selon horde

my %Hspools=(
   'a' => 'sp0',
   'b' => 'sp1',
   'c' => 'sp2',
   'd' => 'sp3',
   'e' => 'sp4',
   'f' => 'sp5',
   'g' => 'sp6',
   'h' => 'sp0',
   'i' => 'sp11',
   'j' => 'sp3',
   'k' => 'sp5',
   'l' => 'sp7',
   'm' => 'sp8',
   'n' => 'sp5',
   'o' => 'sp4',
   'p' => 'sp9',
   'q' => 'sp11',
   'r' => 'sp10',
   's' => 'sp4',
   't' => 'sp10',
   'u' => 'sp11',
   'v' => 'sp9',
   'w' => 'sp11',
   'x' => 'sp11',
   'y' => 'sp11',
   'z' => 'sp11',
);


### Connect IMAP
my $client = Cyrus::IMAP::Admin->new('127.0.0.1');
$client->authenticate( -user => 'cyrus', -password => 'xxxxxxx',-mechanism => "LOGIN") or die $client->error;

@res=`/usr/cyrus/bin/ctl_mboxlist -d /var/lib/imap/mailboxes.db`;
open (LOG,'>>/root/movebal.txt');
$i=0;
foreach (@res){
   if (/^user\.(\S+)\s+\d+\s+(\S+)\s+/){
      $user=$1;
      $spool=$2;
      next if $user=~/\./;
      next if $spool=~/^sp\d+/;
      if (IsUserOnline($user)){
         print "$user en ligne\n";
         next;
      }
      $i++;
      $lettre=substr($user,0,1);
      $client->setacl("user/$user", 'cyrus' => 'all');
      $client->setacl("user/$user/*", 'cyrus' => 'all');
      $client->rename("user/$user","user/$user",$Hspools{$lettre});
      $now=localtime;
      printf LOG "$now $user $spool $lettre $Hspools{$lettre}\n";
      if (defined($client->error)){
         printf LOG "Pb pour déplacer la bal $uid $Hspools{$lettre}: ",$client->error,"\n";
      }
      $client->setquota("user/$user",'STORAGE' => $quota);
   }
}
close LOG;

sub IsUserOnline{
   my ($user)=@_;
   @ret=`grep "$user\tuser.$user" /var/lib/imap/proc/*`;
   foreach (@ret){
      return 1 if /$user/;
   }
}



2010/6/16 Nestor A Diaz <nestor@xxxxxxxxxxxxxxx>
Hello Cyrus People.

I have been always a proud user of the cyrus email system, currently my
larger cyrus installation is about 70K users, which have been working
fine, however i need to make some adjustems to improve the resposiviness
of the mailstore.

At the begining i use only a default mailstore, on a ext3 formatted
filesystem over a raid 10 / LVM, it works fine, however when it reaches
the 32768 directories, (a limitation of ext3) i have to create another
partition, and then another one, so i have currently 3 partitions in total.

Having defined some criteria in order to group every user into a
partition i will like some recomendations, taking in mind the current
cyrus server tools which i currently don't know what are their current
capabilities, my plan is to forget using cyrus partitions, and instead
using LVM partitions and the propper links to the physical partitions
from the logical one.

So i currently have:

partition-default : 30K users
partition-alt1: 20K users
partition-alt2 : 20K users.

and then i will like to have just one partition:

partition-default ext4 formatted with links to physical partitions based
on the first letter [a-z] of the user.

I think LVM is a great advantage, and in some way aliviate the need for
cyrus partitions, i it will lead me deal with the grow of a partition
and data administration so easy without too much downtime.

So, it means that accordingly to this stragegy which seems to be the
most simple for me, what would be the best technical path to accomplish
a good migration ? allowusermoves: yes option and renamemailbox is
enoguth to do this task ? what if i want to do this online ? can i ? how
? is there any script that will help me do that ?

Any suggestions or migration histories are welcome !

What about murder / perdition ? they make any sense in this configuration ?

Keep this good work !

Slds.

--
Nestor A. Diaz
Ingeniero de Sistemas
Tel. +57 1-485-3020 x 211
Cel. +57 316-227-3593
Tel. SIP: sip:211@xxxxxxxxxxxxxxx
Email/MSN: nestor@xxxxxxxxxxxxxxx
http://www.tiendalinux.com/
Bogota, Colombia

----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html



--
Dominique LALOT
Ingénieur Systèmes et Réseaux
http://annuaire.univmed.fr/showuser.php?uid=lalot
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux