This sounds like a perfect use case for https://github.com/jdarcy/negative-lookup Jeff Darcy made this as an example of building your own translators. But it basically caches nagative-lookups so repeated asking of the same questions go away. Hell, might be beneficial to make a translator that just immediately replies "does not exist" for all filenames starting ._* On Wed, Dec 19, 2012 at 8:00 PM, Adam Tygart <mozes at k-state.edu> wrote: > Dan, > > Does something like this help for SAMBA? > http://www.e-rave.nl/disable-creation-of-ds_store-files-on-samba-shares > > Negative lookups are a particularly expensive operation on distributed > filesystems, GlusterFS in particular. It has to check through each > brick to see that the file *really* doesn't exist, increasing the > lookup overhead immensely. If samba doesn't pass these queries to > GlusterFS, it might help a lot. > > -- > Adam Tygart > Beocat Sysadmin > > On Wed, Dec 19, 2012 at 8:29 PM, Daniel Mons <daemons at kanuka.com.au> > wrote: > > I'm rolling out 4 lots of 6-node GlusterFS setups for my employer. Each > > node is ~33TB of RAID6 backed storage (16x 3TB SATA disks in RAID6 with a > > hot spare hanging off an LSI controller, with 2x SSDs configured for > > caching), and Gluster is configured in distribute-replicate. Each > cluster > > is 200TB of raw space, 100TB usable after replication. When complete, > there > > will be 4 of these clusters. > > > > Nodes are configured as XFS with 512byte inodes, running a fully patched > > CentOS6 and Gluster 3.3.1. Each node has a 6 core Xeon processor (with > HT > > for 12 threads) with 32GB of RAM. Each node runs 2x 10Gbps Ethernet over > > fiber in a bonded configuration (single IP address per node) for a full > > 20Gbits per node. > > > > GlusterFS FUSE performance under Linux is great (clients run a mix of > Ubuntu > > 12.04 LTS for workstations and CentOS6 for servers). Samba performance > back > > to Windows 7 clients is great. NFS performance via both Gluster's > userspace > > setup as well as CentOS6's native NFS4 kernel server are great to most > other > > systems where we can't get the Gluster FUSE client loaded (large > > industry-specific Linux boxes that are provided by vendors as a "black > box" > > solution, and only allow limited access via NFS or SMB/CIFS). All > testing > > so far under those conditions proves orders of magnitude faster > throughput > > than our existing single NAS solutions. > > > > MacOSX Finder performance is a problem, however. There's a huge bug in > > MacOSX itself that prevents using NFS at all (discussions on other > mailing > > lists suggest it occurred somewhere around 10.6, and continues through > into > > 10.7 and 10.8). > > > > Mounting via SMB under OSX is more stable than NFS, however in folders > with > > a large amount of files, Finder goes looking for a corresponding Apple > > Resource Fork file (for every "filename.ext", it looks for a > > "._filename.ext"). Running tcpdump and wireshark on the Gluster nodes > shows > > that the resulting "FILE_NOT_FOUND" error back to the client takes a very > > long time. Configuring a single node as a pure NAS with the same > software > > (but no Gluster implementation) is lightening fast. As soon as > GlusterFS > > comes in to play, reporting of each "FILE_NOT_FOUND" slows down the > process > > dramatically, causing a directory with ~1000 images in it to take well > over > > 5 minutes to display the contents in MacOSX finder. > > > > This problem is resolved somewhat by switching to AFP (via Netatalk > loaded > > on the GlusterFS nodes), but it has it's own problems unique to that > > protocol, and I'd rather stick to GlusterFS-FUSE, NFS or SMB in that > order > > of preference. > > > > It's worth noting that through the terminal, these problems don't exist. > > Mounting via SMB, browsing to the volume in terminal and running "ls" or > > "find" style commands retrieve file listings at a similar speed to Linux > and > > Windows. The problem is limited to clients using Finder to browse > > directories, and again particularly ones with a large number of files > that > > don't have matching Apple Resource Fork files. (Of note, creating empty > > files of the matching "._filename.ext" format solves the performance > > problem, but litters our filestores with millions of empty files, which > we > > don't want). > > > > I understand the problem is not strictly Gluster's issue. Finder is > looking > > for a heck of a lot of files that don't exist (which is a pretty silly > > design), and it tends to occur only with Samba re-exporting GlusterFS > > volumes that we can see. And likewise Apple's NFS bug that has now been > in > > existence across three releases of their OS is pretty horrible. But > > hopefully I can at least describe the problem and prompt some testing by > > others. > > > > I haven't had a chance to test a MacOSX FUSE client due to time > constraints, > > but that would at least answer the question if the problem is Gluster's > lag > > in reporting of files not found, or Samba's. > > > > -Dan > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121219/6565cac0/attachment.html>
