I have 3 servers with replica 3 volumes, 4 bricks per server on lvm partitions that are placed on each of 4 hard drives, 15 volumes resulting in 60 bricks per server. One of my servers is also a kvm host running (only) 24 vms. Each vm image is only 6 gig, enough for the operating system and applications and is hosted on one volume. The data for each application is hosted on its own GlusterFS volume. For mysql, I set up my innodb store to use 4 files (I don't do 1 file per table), each file distributes to each of the 4 replica subvolumes. This balances the load pretty nicely. I don't really do anything special for anything else, other than the php app recommendations I make on my blog (http://joejulian.name) which all have nothing to do with the actual filesystem. The thing that I think some people (even John Mark) miss apply is that this is just a tool. You have to engineer a solution using the tools you have available. If you feel the positives that GlusterFS provides outweigh the negatives, then you will simply have to engineer a solution that suits your end goal using this tool. It's not a question of whether it works, it's whether you can make it work for your use case. On 12/27/2012 03:00 PM, Miles Fidelman wrote: > Ok... now that's diametrically the opposite response from Dan Cyr's of > a few minutes ago. > > Can you say just a bit more about your configuration - how many nodes, > do you have storage and processing combined or separated, how do you > have your drives partitioned, and so forth? > > Thanks, > > Miles > > > Joe Julian wrote: >> Trying to return to the actual question, the way I handle those is to >> mount gluster volumes that host the data for those tasks from within >> the vm. I've done that successfully since 2.0 with all of those >> services. >> >> The limitations that others are expressing have as much to do with >> limitations placed on their own designs as with their hardware. Sure, >> there are other less stable and/or scalable systems that are faster, >> but with proper engineering you should be able to build a system that >> meets those design requirements. >> >> The one piece that wasn't there before but now is in 3.3 is the >> "locking and performance problems during disk rebuilds" which is now >> done at a much more granular level and I have successfully >> self-healed several vm images simultaneously while doing it on all of >> them without any measurable delays. >> >> Miles Fidelman <mfidelman at meetinghouse.net> wrote: >> >> Joe Julian wrote: >> >> It would probably be better to ask this with end-goal >> questions instead of with a unspecified "critical feature" >> list and "performance problems". >> >> >> Ok... I'm running a 2-node cluster that's essentially a mini >> cloud stack >> - with storage and processing combined on the same boxes. I'm >> running a >> production VM that hosts a mail server, list server, web server, and >> database; another production VM providing a backup server for the >> cluster and for a bunch of desktop machines; and several VMs used >> for a >> variety of development and testing purposes. It's all backed by a >> storage stack consisting of linux raid10 -> lvm -> drbd, and uses >> pacemaker for high-availability failover of the >> production VMs. It all >> performs reasonably well under moderate load (mail flows, web >> servers >> respond, database transactions complete, without notable user-level >> delays; queues don't back up; cpu and io loads stay within >> reasonable >> bounds). >> >> The goals are to: >> - add storage and processing capacity by adding two more nodes - >> each >> consisting of several CPU cores and 4 disks each >> - maintain the flexibility to create/delete/migrate/failover virtual >> machines - across 4 nodes instead of 2 >> - avoid having to play games with pairwise DRBD configurations by >> moving >> to a clustered filesystem >> - in essence, I'm looking to do what Sheepdog purports to do, >> except in >> a Xen environment >> >> Earlier versions of gluster had reported problems with: >> - supporting databases >> - supporting VMs >> - locking and performance problems during disk rebuilds >> - and... most of the gluster documentation implies that it's >> preferable >> to separate storage nodes from processing nodes >> >> It looks like Gluster 3.2 and 3.3 have addressed some of these >> issues, >> and I'm trying to get a general read on whether it's worth >> putting in >> the effort of moving forward with some experimentation, or >> whether this >> is a non-starter. Is there anyone out there who's tried to run this >> kind of mini-cloud with gluster? What kind of results have you had? >> >> >> >> On 12/26/2012 08:24 PM, Miles Fidelman wrote: >> >> Hi Folks, I find myself trying to expand a 2-node >> high-availability cluster from to a 4-node cluster. I'm >> running Xen virtualization, and currently using DRBD to >> mirror data, and pacemaker to failover cleanly. The thing >> is, I'm trying to add 2 nodes to the cluster, and DRBD >> doesn't scale. Also, as a function of rackspace limits, >> and the hardware at hand, I can't separate storage nodes >> from compute nodes - instead, I have to live with 4 nodes, >> each with 4 large drives (but also w/ 4 gigE ports per >> server). The obvious thought is to use Gluster to assemble >> all the drives into one large storage pool, with >> replication. But.. last time I looked at this (6 months or >> so back), it looked like some of the critical features >> were brand new, and performance seemed to be a problem in >> the configuration I'm thinking of. Which leads me to my >> question: Has the situation improved to the point that I >> can use Gluster this way? Thanks very much, Miles Fidelman >> >> ------------------------------------------------------------------------ >> Gluster-users mailing list Gluster-users at gluster.org >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >> > >
