We brought up a test cluster to investigate GlusterFS. Using the Quick Start instructions, we brought up a 2 server 1 brick replicating setup and mounted to it from a third box with the fuse mount (all ver 3.4.1) # gluster volume info Volume Name: mailtest Type: Replicate Volume ID: 9e412774-b8c9-4135-b7fb-bc0dd298d06a Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster2:/gluster1/mailtest/brick1 Brick2: gluster1:/gluster1/mailtest/brick1 Options Reconfigured: server.root-squash: no We then began loading the cluster with data (from a nearby Mailserver) by rsyncing Maildirs onto the mount. We chose maildirs because small files are supposed to be a worst case scenario in Gluster. During our testing, everything worked great. Speed was acceptable (we are familiar with other Distributed File Systems, so we don't have unrealistic expectations). We yanked cords, turned off machines and generally tortured the setup, all to good effect. Everything performs as advertised, though you have to do a LOT of googling to get some of the answers when 'recovering'. There appears to be a lot of 'secret' recipes to get things done and I think the doc site should link to JoeJulians blog, among others <grin>. We then decided to see what email Maildir performance was like using a pop3 tester program. So we quickly installed qmail which has the qmail-pop3d daemon. The result was that the pop daemon cant see the email # telnet localhost 110 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. +OK <20481.1385248027 at mailtest> USER glusttest +OK PASS somepass +OK LIST +OK . If we copy that SAME directory over to /users (not on the gluster mount), then the LIST command shows that email. So we know that the qmail-pop3d setup is working fine. LIST +OK 1 3120 2 11157 3 3267 etc. So since we normally use POP/IMAP over NFS, we decided to try the gluster NFS mount. That the NFS mount gave us an even stranger result. It doesn't even see the Maildir. # telnet localhost 110 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. +OK <20488.1385248223 at mailtest> USER glusttest +OK PASS somepass -ERR this user has no $HOME/Maildir Connection closed by foreign host. We have verified that the NTP daemon and thus the time settings are correct. There are NO errors corresponding to this activity in any of the log files on either the data servers or the mount (logs only refer to the mount/unmount ops). Using the command line, we can manually manipulate files on the mount to our hearts content (as root or as an appropriate user) with no errors. mkdir, rm, touch, all work fine. However, using the VmailMGR command line tools, we are unable to add new users when mounted under NFS # su - glusttest $ vadduser test2 Enter the user's new password: Please type it again for verification: vadduser: error adding the virtual user: Can't create the mail directory './users/test2' then when its manually created qmail refused to deliver to that Maildir because it cant see the install (in this case .qmail file) 1385249260.729221 delivery 146770: failure: Sorry,_no_mailbox_here_by_that_name._(#5.1.1)/ Under the FUSE mount, VmailMGR tools work fine and email is able to be delivered to the Maildir. $ vadduser test3 Enter the user's new password: Please type it again for verification: vadduser: user 'test3' successfully added. but of course the pop3 daemon doesn't see the email that qmail process just was just delivered. Finally, you can see from the info provided, that we deliberately disabled root_squash (since thats a new 3.4 feature) but that made no difference to any of the above results. SELINUX is disabled on all 3 machines. So.... What is going on here? Its not a mission critical, as GlusterFS is probably inappropriate for a mailserver, but I really need to understand what is going on so I can recommend GlusterFS for other situations. Is this a MailDir thing or is something else going on? Sincerely, -bill
