I've mentioned a few times previously my frustrations with MacOSX, and in particular Mac's Finder application which seems to do odd things with network file systems. The short background is that NFS on MacOSX since 10.5 is broken when using GUI (Cocoa/Carbon) applications. Files and directories mysteriously vanish from view at random intervals, renaming files from other protocols (FUSE+GlusterFS or SMB) can sometimes trigger this. and changing the case of filenames causes havoc. It's all very random and frustrating as there's zero consistency to try and map a root cause. SMB works, however is painfully slow with clustered file systems due to the requirement for MacOSX to have resource forks stored in separate files (a "._filename" for each "filename", along with .DS_Store files) causing negative lookups for every single file (which can take a very long time on directories with thousands of images inside them, which is common for us - up to 3 minutes to open a single directory in Finder!). There is a negative lookup cache for Gluster, but it hasn't been touched for years as far as I can tell, and is covered in "don't use this in production, it's in testing" warnings that scare me. It's also merely removing the symptoms of this particular problem, and not addressing the root cause. AFP works (via Netatalk3) by storing resource fork information in it's own private CNID database, but comes with it's own problems (top level directories can't be marked read-only as it causes the whole share to be read-only, which is a security problem for us). AFP is also up to 25% slower than SMB for our workloads. However I think I've finally stumbled upon the solution! (There's a lot of noise out there when it comes to technical solutions for OSX, which makes finding this stuff incredibly frustrating). Samba offers a "streams_xattr" VFS object which allows simulation of NTFS file streams, which allows the storing of "alternate data streams" (i.e.: things like resource forks and other "non-file" information) into xattr, which Gluster supports on top of any file system that also offers xattr (we use XFS with 512 byte inodes, as per the RHEL best practice guides, and that supports xattr natively). What tipped me off was this post: http://www.libertypages.com/clarktech/?p=6284 Which lead me to the appropriate man page: http://www.samba.org/samba/docs/man/manpages/vfs_streams_xattr.8.html While folks are looking for solutions to SMB under Mavericks, the "no ugly .DS_Store" comment was what captured my attention (as .DS_Store is another resource fork filetype Macs enforce over SMB). We've tested a handful of machines today on our setup (6 GlusterFS nodes running CentOS6 and Samba 3.6 straight from the default repos). I added the "vfs objects = streams_xattr" line to our config, and instantly speed returned to the Macs mounting SMB from Gluster (even without libgfapi, which I'm yet to get working due to time constraints). Clients are directed to the Gluster nodes via a DNS round robin, and distribute pretty well over the course of the day. Throughput is higher and latency lower from a GUI file browsing point of view than AFP, and the breakages of NFS aren't there. The glaring downside of course is that we're not in NFS land, so system-wide network mounts aren't happening. That sucks for us due to the way we work, and a working NFS stack in Mac is far more desirable. However this is a better place than having to fall back to AFP for us with respect to performance, ease of distributing load, and fewer protocols/services to manage. I'll test a bit more, and contribute our complete smb.conf files a little later on if it all works nicely. For now, anyone wanting to use Samba with Gluster and connect MacOSX machines, I strongly recommend you also test "streams_xattr" as part of your Samba config. I've tested Samba 3.6 as well as Samba 4.1, and both work fine with this VFS object enabled. Samba 3.6 only offers SMB1, but that's fine for OSX 10.6 through 10.8 inclusive. Samba 4.1 offers SMB2 and SMB3, however you need Mavericks to use the later revisions of the protocols. A quick test suggests that Samba 4.1 is slightly faster for OSX 10.9 clients, and makes no difference to OSX 10.8 and below. I've not seen any change in Windows client behaviour with this VFS object enabled All of this should speed up *any* Samba storage for Mac clients theoretically, but is especially useful to clustered filesystems like GlusterFS. Hope that helps anyone also suffering through Mac OSX deployments! -Dan ---------------- Dan Mons R&D SysAdmin Unbreaker of broken things Cutting Edge http://cuttingedge.com.au _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
