Joe, Thanks for the details, but I'm having just a bit of a problem parsing and picturing the description. Any possibility of walking this through from the bottom up - hardware (server + n drives) -> drive partitioning -> lvm setup -> gluster config -> VM setup? Thanks again, Miles Joe Julian wrote: > I have 3 servers with replica 3 volumes, 4 bricks per server on lvm > partitions that are placed on each of 4 hard drives, 15 volumes > resulting in 60 bricks per server. One of my servers is also a kvm > host running (only) 24 vms. > > Each vm image is only 6 gig, enough for the operating system and > applications and is hosted on one volume. The data for each > application is hosted on its own GlusterFS volume. > > For mysql, I set up my innodb store to use 4 files (I don't do 1 file > per table), each file distributes to each of the 4 replica subvolumes. > This balances the load pretty nicely. > > I don't really do anything special for anything else, other than the > php app recommendations I make on my blog (http://joejulian.name) > which all have nothing to do with the actual filesystem. > > The thing that I think some people (even John Mark) miss apply is that > this is just a tool. You have to engineer a solution using the tools > you have available. If you feel the positives that GlusterFS provides > outweigh the negatives, then you will simply have to engineer a > solution that suits your end goal using this tool. It's not a question > of whether it works, it's whether you can make it work for your use case. > > On 12/27/2012 03:00 PM, Miles Fidelman wrote: >> Ok... now that's diametrically the opposite response from Dan Cyr's >> of a few minutes ago. >> >> Can you say just a bit more about your configuration - how many >> nodes, do you have storage and processing combined or separated, how >> do you have your drives partitioned, and so forth? >> >> Thanks, >> >> Miles >> >> >> Joe Julian wrote: >>> Trying to return to the actual question, the way I handle those is >>> to mount gluster volumes that host the data for those tasks from >>> within the vm. I've done that successfully since 2.0 with all of >>> those services. >>> >>> The limitations that others are expressing have as much to do with >>> limitations placed on their own designs as with their hardware. >>> Sure, there are other less stable and/or scalable systems that are >>> faster, but with proper engineering you should be able to build a >>> system that meets those design requirements. >>> >>> The one piece that wasn't there before but now is in 3.3 is the >>> "locking and performance problems during disk rebuilds" which is now >>> done at a much more granular level and I have successfully >>> self-healed several vm images simultaneously while doing it on all >>> of them without any measurable delays. >>> >>> Miles Fidelman <mfidelman at meetinghouse.net> wrote: >>> >>> Joe Julian wrote: >>> >>> It would probably be better to ask this with end-goal >>> questions instead of with a unspecified "critical feature" >>> list and "performance problems". >>> >>> >>> Ok... I'm running a 2-node cluster that's essentially a mini >>> cloud stack >>> - with storage and processing combined on the same boxes. I'm >>> running a >>> production VM that hosts a mail server, list server, web server, >>> and >>> database; another production VM providing a backup server for the >>> cluster and for a bunch of desktop machines; and several VMs >>> used for a >>> variety of development and testing purposes. It's all backed by a >>> storage stack consisting of linux raid10 -> lvm -> drbd, and uses >>> pacemaker for high-availability failover of the >>> production VMs. It all >>> performs reasonably well under moderate load (mail flows, web >>> servers >>> respond, database transactions complete, without notable user-level >>> delays; queues don't back up; cpu and io loads stay within >>> reasonable >>> bounds). >>> >>> The goals are to: >>> - add storage and processing capacity by adding two more nodes - >>> each >>> consisting of several CPU cores and 4 disks each >>> - maintain the flexibility to create/delete/migrate/failover >>> virtual >>> machines - across 4 nodes instead of 2 >>> - avoid having to play games with pairwise DRBD configurations >>> by moving >>> to a clustered filesystem >>> - in essence, I'm looking to do what Sheepdog purports to do, >>> except in >>> a Xen environment >>> >>> Earlier versions of gluster had reported problems with: >>> - supporting databases >>> - supporting VMs >>> - locking and performance problems during disk rebuilds >>> - and... most of the gluster documentation implies that it's >>> preferable >>> to separate storage nodes from processing nodes >>> >>> It looks like Gluster 3.2 and 3.3 have addressed some of these >>> issues, >>> and I'm trying to get a general read on whether it's worth >>> putting in >>> the effort of moving forward with some experimentation, or >>> whether this >>> is a non-starter. Is there anyone out there who's tried to run >>> this >>> kind of mini-cloud with gluster? What kind of results have you >>> had? >>> >>> >>> >>> On 12/26/2012 08:24 PM, Miles Fidelman wrote: >>> >>> Hi Folks, I find myself trying to expand a 2-node >>> high-availability cluster from to a 4-node cluster. I'm >>> running Xen virtualization, and currently using DRBD to >>> mirror data, and pacemaker to failover cleanly. The thing >>> is, I'm trying to add 2 nodes to the cluster, and DRBD >>> doesn't scale. Also, as a function of rackspace limits, >>> and the hardware at hand, I can't separate storage nodes >>> from compute nodes - instead, I have to live with 4 nodes, >>> each with 4 large drives (but also w/ 4 gigE ports per >>> server). The obvious thought is to use Gluster to assemble >>> all the drives into one large storage pool, with >>> replication. But.. last time I looked at this (6 months or >>> so back), it looked like some of the critical features >>> were brand new, and performance seemed to be a problem in >>> the configuration I'm thinking of. Which leads me to my >>> question: Has the situation improved to the point that I >>> can use Gluster this way? Thanks very much, Miles Fidelman >>> >>> ------------------------------------------------------------------------ >>> >>> Gluster-users mailing list Gluster-users at gluster.org >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >>> >>> >> >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
