Just create one 3 replica volume with 1 brick on each of 3 storage servers. Raid5 for servers will be more then enough - it is already replica 3.
Use ovirt to mount glusterfs to VM from hosts (as it uses libgfapi) rather then fuse mount from VM itself.
libgfapi is supposedly faster. Might depend on which mail storage type you use - if maildir, libgfapi to VM on he host should be better.
Also mid that you might need compression for mail storage and deduplication (of attachments)
On Wed, Jul 25, 2018 at 5:11 AM, Γιώργος Βασιλόπουλος <g.vasilopoulos@xxxxxx> wrote:
Hello
I am trying to lay down my options regarding storage with glusterfs, vm storage and email storage
Hardware in my disposal is specific : I have 9 servers for running vm's under ovirt 4.2 and 3 servers for storage
The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB RAM and 2x10G ethernet
each storage server has inside 2x300gb 10k drives which I intent to use as os install and maybe a litle volume for isos on nfs
also present are 6x200GB SSD drives which I think of using as tiering volume(s)
And the main storage is on an external JBOD box with 12x4TB drives connected via SAS to the server with RAID controller capable of varius raid levels.
So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes (high availability is the no 1 requirement) in a cyclic fashion, 2 data bricks and one arbiter on each storage server
I will implement raid6 with one spare drive on each server (we are in an island and getting a disk replacement can take days occasionaly) which will give me about 36T of usable storage per server.
So regarding the vm storage I am thinking that 2 volumes with 17TB each and an arbiter of 1TB.
What messes things up is that I was required to put the email storage in this installation. Our email is pretty big and busy with about 50000 users curently at 13T of storage.
Currently it runs on a few vm's and uses storage from nfs given from another vm. It is runs postfix/dovecot and right now a single big vm does mailbox delivery but this reaches it's limits. Mail storage now is on a EMC VNX5500
But it will be moved to glusterfs for various reasons.
I would like some advise regarding the email storage. I think my options are
1a. use a VM as NFS give it a huge disk (raw image on gluster vm optimized) and be done with it
1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw images on gluster vm optimized) and maybe take some advantage of using 2-3 io-threads in o virt to write to 2-3 disks simultaneously. Will this give extra performance ?
2.Give gluster as nfs straight to dovecot but I wonder if this will have performance drawback since it will be fuse mounted. I am also worried about the arbiter volume since there will be thousands of small files, practically arbiter
will probably have to be as large as the data bricks or half that size
3. Give gluster as glusterfs mount point which I think will have about the same issues as 2.
I have read about problems with dovecot indexes and glusterfs. Is this still an issue? or is it a problem that only shows when there is no dovecot director.
Personaly I am inclined on using solution 1 because I think that arbiter volumes will be smaller (Am I right?) though it may have some overhead regarding nfs on the vm. On the other hand this solution will use libgfapi
which might balance things a bit.
Will it help if in such a case use small (16mb) shard size and tiering ?
I'm afraid I have it a bit mixed up in my mind and I could really use some help.
--
Βασιλόπουλος Γιώργος
Ηλεκτρολόγος Μηχανικός Τ.Ε.
Διαχειριστής Υπολ. Συστημάτων
Πανεπιστήμιο Κρήτης
Κ.Υ.Υ.Τ.Π.Ε.
Τμήμα Επικοινωνιών και Δικτύων
Βούτες Ηρακλείου 70013
Τηλ : 2810393310
email : g.vasilopoulos@xxxxxx
http://www.ucnet.uoc.gr
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
