On Fri, 2 May 2008, Brandon Lamb wrote:
On Fri, May 2, 2008 at 9:30 AM, Brandon Lamb <brandonlamb@xxxxxxxxx> wrote:
On Fri, May 2, 2008 at 9:21 AM, Shaofeng Yang <syang@xxxxxxxxxxx> wrote:
Can anybody share some thoughts about those cluster file systems? We are
trying to compare the pros and cons for each solution.
Thanks,
Shaofeng
Tought question as it depends on what you are needing. Myself I have
messed around with 3 of those for the last 2 years, so far I am still
just using an 2 NFS servers, one for mail and one for web for my 14 or
so client machines until I figure out how to use glusterfs.
I tried gfs (redhat) and I dont remember if I even ever got it to
actually run, I was trying it out on fedora distros. It seemed very
over complicated and not very user friendly (just my experience).
The key to your problems is in Fedora. It _REALLY_ isn't fit for anything
more than a hobbyist home setup. It is the alphaware versin of RHEL. For
example, FC{7,8} ships only with GFS2 which is not yet stable, and nobody
claims it to be. RHEL5 comes with GFS1 and GFS2, GSF2 being there just as
a tech preview but not for use in production systems.
RHCS has a somewhat steep learning curve, but it's not one that can't be
overcome in half a day with assistance from the mailing list. Once you
figure out what you're doing it's pretty straightforward, and I've
deployed quite a few clusters based on it for various clients.
OCFS2 seemed very clean and I was able to use with with ISCSI but man
the load on my server was running at 7 and it was on the slow side.
What I was trying to do with it was create a single drive to put my
maildir data onto (millions of small mail files). The way it worked
was you actually mounted the file system like it was a local file
system on all machines that needed it and the cluster part would
handle the locking or whatnot. Cool concept but overkill for what I
needed.
ANY shared storage FS will suffer major performance penalties for this.
File write requires a directory lock. If you start getting contention
(e.g. shared imap folders), the performance will go through the floor
because you're dealing with distributed lock management overhead and
network latencies on top of normal disk latencies. Having said that, most
POSIX locking supporting cluster FS-es will suffer from this issue, some
more than others.
Also I believe both GFS and OCFS2 are these "specialized" file
systems. What happens if it breaks or goes down? How do you access
your data? Well if gfs or ocfs2 is broken you cant.
That's a bit like saying that if ext3 breaks, you can't access your data.
The only specilist thing about them is that they are designed for shared
storage, i.e. SAN or DRBD replicated volume.
With glusterfs,
you have direct access to your underlying data. So you can have your
big raid mounted on a server and use XFS file system, glusterfs just
sits on top of this so if for some reason you break your glusterfs
setup you *could* revert back to some other form of serving files
(such as NFS). Obviously this totally depends on your situation and
how you are using it.
Indeed, but it is fundamentally a distributed rather than centralized
storage approach. This isn't a bad thing, but it is an important
distinction. GlusterFS is a essentially for a cluster oriented NAS. GFS
and OCFS2 are SAN oriented. That is a major difference.
Hence the reason that *so far* I am still using NFS. It comes on every
linux installation, its fairly easy to setup by editing what, 4 lines
or so. GlusterFS takes the same simple approach and if you do break
it, you still have access to your data.
The learning curve for glusterfs is much better than the others from
my experience so far. The biggest thing is just learning all of the
different ways you can configure spec files.
IME, RHCS/GFS didn't take me any more head scratching than GlusterFS did
when I first got into it. They are designed for different purposes, and
chosing one over the other should be based on project requirements, not on
simplicity.
I just wanted to add the stressing of simplicity.
When the *#($ hits the fan, I would much rather be fixing something
that is on the simple side from the start, rather wondering what the
### is going on with a specialized filesystem and all the extra pieces
it adds and not having access to my data.
Calling GFS or OCFS2 specialized for this reason is bogus, as I explained
earlier. You might as well call ext3 specialized then, along with every
other FS.
That is what my company
finally decided on. I was looking into buying iscsi hbas and seeing
about upgrading our network, using DRBD and OCFS2 to sync our two RAID
servers and after two weeks we just looked at each other and said, you
know what. NFS may not be the most kickass thing or lightning fast, or
have builtin replication, but it WORKS. And if a server failed well it
would suck but we could copy from a backup onto the other nfs server
and be running again.
*shrug*
I got it all working and without any nightmarish effort. Sure, it's more
than 4 lines of config than NFS requires, but the benefits are worth it.
You only have to put in the setup time once, and any O(1) effort is
preferable to dealing with downtimes in the future.
This is the reason I am down to only investing time into glusterfs.
Its simple but powerful! It does all kinds of cool stuff, and if the
worst happens, Im not really all THAT worried because I know I can
still get my files and have a SIMPLE backup plan.
If GlusterFS works for you - great. I use both GlusterFS and GFS, and use
them for fundamentally different tasks. They simply aren't
interchangeable in a reasonably thought out systems architecture.
Gordan
