Re: cyrus-imapd with lmtpd + postfix slow delivery with group email ids

jayesh shinde <jayesh.shinde@xxxxxxxxxxxxx> · Mon, 18 May 2015 13:51:52 +0530



    Hi Stephen , 

          
          I am handling via TCP  i.e from smtp server to mailbox server.
          

          on smtp server postfix's main.cf  transport_maps is maintain ,
          which deliver the emails to lmtp. 

          
          Example like below. 

          /@mydomain.com/        lmtp:[192.168.1.1]

          
          Request you to please guide here or any fine tuning etc ....

          
          Regards

          Jayesh Shinde

        
      On 05/15/2015 10:11 PM, Stephen Ulmer wrote:

    
      Are you handing off to LMTP via TCP or a pipe?
      

            -- 
            Stephen
            

            On May 15, 2015, at 2:18 AM, jayesh shinde
              <jayesh.shinde@xxxxxxxxxxxxx>
              wrote:
            

                Thanks Dan for your
                    feedback. 

                    I verified the suggested points , its working as
                    require.  Please find below inline reply for each. 

                    
                    I found that  when big group email is getting
                    deliver ( i.e more than 500 - 4K members ) at that
                    time other emails are not getting deliver
                    simultaneously to user's mailbox. 

                    I verified this from live maillog  i.e  by doing
                    grep key words like "to mailbox:" /  "Delivered:" 
                    etc ...

                    
                    My mail queue issue is getting critical in peak
                    hours !!  Kindly suggest. 

                    
                    On 05/14/2015 08:22 PM, Dan White wrote:

                  
                On 05/14/15 05:42 +0000,
                    jayesh.shinde wrote: 

                  
                  Please
                      suggest , for faster delivery  what could be the
                      best config  in 

                      postfix + cyrus-imapd . 

                     
                      I am using the cyrus-imapd-2.4.17-6.el6.x86_64 for
                      mailbox server and for 

                      smtp server postfix-2.10.0-1.el6.x86_64 Both
                      server have SAS hdd , 16 core 

                      , 16 GB RAM .  There is no i/o issues 

                    
                  Mailbox
                      server have  21K email ids.  Cyrus running with 
                      tcp socket on 

                      port 24 i.e lmtpd .  cyrus singleinstance is
                      maintained 

                    
                  For group id
                      mapping is under virtual_alias_maps of smtp's
                      main.cf  i.e no 

                      /etc/aliases. From smtp server emails are getting
                      deliver by 

                      transport_maps to mailbox server . 

                    
                  My problem
                      observe :-- 

                      -------------------------- 

                      1) When HOD send email to 3-5 big group  email ids
                      then , then its postfix 

                      delivery get slow. What I observe is postfix
                      getting delivered the traffic 

                      per email and not parallel.   Each group contain
                      3k or 7k email ids. 

                    
                    Verify that single instance store is working. If
                    not, you may have a 

                    Postfix configuration issue. Find one of these mass
                    emails in a mailbox and 

                    stat it to find out: 

                   
                    stat -c %h <file> 

                   
                    A returned value of 1 means single instance store
                    isn't working. 

                   
                    If that's not the issue, you can determine if this
                    is a postfix issue or a 

                    cyrus issue by sending a mass email directly to
                    lmtpd, with lmtptest. 

                  
                Cyrus
                  Singleinstance is working , I have verified like
                  below. In imapd.conf man page its define as default 1
                  ( i.e default enable ) 

                  
                  [root@Mailbox ~]# ls -lirth
                  /var/spool/imap/domain/v/vdomain/m/user/muser1/ | tail
                  -n 20  

                  
                  163326970 -rw-------  198 cyrus mail
                  413K May 15 10:07 64828.

                  163396232 -rw-------  145 cyrus mail 
                  92K May 15 10:13 64829.

                  163379367 -rw-------  151 cyrus mail 
                  96K May 15 10:29 64830.

                  163323993 -rw-------  496 cyrus mail 
                  13K May 15 10:30 64831.

                  163398621 -rw-------  516 cyrus mail
                  850K May 15 10:42 64832.

                  163398568 -rw-------  523 cyrus mail
                  455K May 15 10:43 64833.

                  163398933 -rw-------  617 cyrus mail
                  3.7M May 15 10:47 64834.

                  163398962 -rw-------  522 cyrus mail
                  1.3M May 15 10:48 64835.

                  163398990 -rw-------  617 cyrus mail 
                  58K May 15 10:49 64836.

                  163396678 -rw-------  465 cyrus mail
                  207K May 15 10:56 64837.

                  163399529 -rw-------  576 cyrus mail
                  332K May 15 10:57 64838.

                  163399554 -rw-------    2 cyrus mail 
                  54K May 15 10:57 64839.

                  163346523 -rw-------   36 cyrus mail
                  767K May 15 11:01 64840.

                  
                  [root@Mailbox muser1]# stat -c %h 64829.

                  145

                  [root@Mailbox muser1]# stat -c %h
                  64828.

                  198

                  
                  In peak hours
                      the queue on postfix get high and other normal
                      emails also 

                      getting stuck in queue. Once the group email get
                      clear after that other 

                      emails also get clear. 

                    
                  in mailbox
                      cyrus.conf :-- lmtp  cmd="lmtpd -a" listen="lmtp"
                      prefork=0 

                      in main.cf of smtp server :-- 

                      lmtp_destination_concurrency_limit = 100 

                      lmtp_destination_recipient_limit = 0 

                    
                    How many lmtp processes do you see spawned in this
                    scenario? 

                   
                [root@Mailbox
                  ~]# ps auxwww| grep lmtp 

                  
                  cyrus     3224  0.0  0.0  73376  4648 ?        S   
                  10:55   0:00 lmtpd

                  cyrus     4043  0.0  0.0  73376  4156 ?        S   
                  10:59   0:00 lmtpd

                  postfix   6972  0.0  0.0  75708  4336 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6974  0.0  0.0  75708  4340 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6975  0.0  0.0  75708  4336 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6977  0.0  0.0  75708  4344 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6979  0.0  0.0  75708  4336 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6981  0.0  0.0  75708  4336 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  postfix   6985  0.0  0.0  75708  4348 ?        S   
                  11:10   0:00 lmtp -t unix -u

                  cyrus     6998  0.0  0.0  73376  4052 ?        S   
                  11:10   0:00 lmtpd

                  cyrus     7001  0.0  0.0  73376  4044 ?        S   
                  11:10   0:00 lmtpd

                  cyrus     7234  0.9  0.0  75456  4976 ?        S   
                  11:12   0:04 lmtpd -a

                  cyrus     8758  0.6  0.0  75452  4112 ?        S   
                  11:19   0:00 lmtpd -a

                  root      8898  0.0  0.0 103236   972 pts/1    S+  
                  11:19   0:00 grep --colour=auto lmtp

                  cyrus    46046  0.0  0.0  73376  4648 ?        S   
                  10:28   0:01 lmtpd

                  cyrus    48107  0.0  0.0  73372  4740 ?        S   
                  10:37   0:01 lmtpd

                  cyrus    48108  0.0  0.0  73376  4700 ?        S   
                  10:37   0:01 lmtpd

                  
                  lmtpd :-- 7 process

                  lmtp -t unix -u :-- 7 process

                  lmtpd -a :-- 2 process

                  
                  Cyrus config is as follows:

                  -----------------------------------

                  
                  SERVICES {

                    imap        cmd="imapd" listen="imap" prefork=5

                    imaps        cmd="imapd -s" listen="imaps" prefork=1

                    pop3        cmd="pop3d" listen="pop3" prefork=3

                    pop3s        cmd="pop3d -s" listen="pop3s" prefork=1

                    sieve        cmd="timsieved" listen="sieve"
                  prefork=0

                  
                    lmtp  cmd="lmtpd -a" listen="lmtp" prefork=0

                    lmtpunix    cmd="lmtpd"
                  listen="/var/lib/imap/socket/lmtp" prefork=1

                  }

                  
                  [root@Mailbox ~]# cat /etc/imapd.conf  

                  configdirectory: /var/lib/imap

                  partition-default: /var/spool/imap

                  admins: cyrus

                  sievedir: /var/lib/imap/sieve

                  sendmail: /usr/sbin/sendmail

                  hashimapspool: true

                  sasl_pwcheck_method: saslauthd

                  sasl_mech_list: PLAIN

                  allowplaintext: yes

                  unixhierarchysep: 1

                  virtdomains: yes

                  altnamespace: on

                  defaultdomain: mydomain.com

                  expunge_mode: immediate

                  delete_mode: immediate

                  tls_cert_file: /usr/share/ssl/certs/server.pem

                  tls_key_file: /usr/share/ssl/certs/server.key

                  tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt

                  lmtp_over_quota_perm_failure: 1

                  duplicatesuppression: 1

                  duplicate_db_path:
                  /var/lib/imap/cyrus-ram-tmpfs/deliver.db

                  statuscache_db_path:
                  /var/lib/imap/cyrus-ram-tmpfs/statuscache.db

                  tlscache_db_path:
                  /var/lib/imap/cyrus-ram-tmpfs/tls_sessions.db 

                  
                  Regards

                  Jayesh Shinde

                 
              ----

              Cyrus Home Page: http://www.cyrusimap.org/

              List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

              To Unsubscribe:

              https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
          
        
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus