On 08/02/2013 06:22 AM, Xavier Trilla wrote: > > Hi, > > We have been playing for a while with GlusterFS (Now with ver 3.4). We > are running tests and playing with it to check if GlusterFS can be > really used as the distributed storage for OpenStack block storage > (Cinder) as new features in KVM, GlusterFS and OpenStack are pointing > to GlusterFS as the future of OpenStack open source block and object > storage. > > But we've found a problem just when we started playing with > GlusterFS... The way distribute translator (DHT) balances the load. I > mean, we understand and see the benefits of metadata less setup. Using > hashes based on filenames and assigning a hash range to each brick is > clever, reliable and fast, but from our understanding there is a big > problem when it comes to storing VM images of a OpenStack deployment. > > I mean, OpenStack Block Storage (Cinder) assigns a name to each volume > it creates (GUID), so GlusterFS does a hash of the filename and > decides in which brick it should be stored. But as in this scenario we > don't have many files (I mean, we would just have one big file per VM) > we may end with a really unbalanced storage. > > Let's say we have a 4 bricks setup with DHT distribute, and we want to > store 100 VMs there, so the ideal scenario would be: > > Brick1: 25 VMs > > Brick2: 25 VMs > > Brick3: 25 VMs > > Brick4: 25 VMs > > As VMs are IO intensive it's really important to correctly balance the > load, as each brick has a limited amount of IOPS, but as DHT is just > based on a filename HASH, we could end with something like the > following scenario (Or even worse): > > Brick1: 50 VMs > > Brick2: 10 VMs > > Brick3: 35 VMs > > Brick4: 5 VMs > > And if we scale this out, things may get even worse. I mean, we may > end with almost all VM file in one or two bricks and all the other > bricks almost empty. And if we use growing VM disk image files like > qcow2 the option "min-free-disk" will not prevent all VMs disk image > files being stored in the same brick. So, I understand DHT works well > for large amount of small files, but for few big IO intensive files > doesn't seem to be a really good solution... (I mean, we are looking > for a solution able to handle around 32 bricks and around 1500 VM for > the initial deployment and able to scale up to 256 bricks and 12000 > VMs :/ ) > > So, anybody has a suggestion about how to handle this? I mean so far > we only see two options: Either using legacy unify translator with ALU > scheduler or either use cluster/stripe translator with a big > block-size so at least load gets balanced across all bricks in some > way. But obviously we don't like unify as it needs a namespace brick, > and using stripping seems to have an impact on performance and really > complicates backup/restore/recovery strategies. > > Another suggestion that you may want to try is, have your GlusterFS node also serve as OpenStack Cinder and use NUFA[1] ~shanks [1] http://gluster.org/community/documentation/index.php/Translators/cluster/nufa -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130805/98ae88e0/attachment.html>
