On Fri, 2008-04-11 at 10:28 -0500, Wendy Cheng wrote: > christopher barry wrote: > > On Tue, 2008-04-08 at 09:37 -0500, Wendy Cheng wrote: > > > >> gordan@xxxxxxxxxx wrote: > >> > >>> > >>>> my setup: > >>>> 6 rh4.5 nodes, gfs1 v6.1, behind redundant LVS directors. I know it's > >>>> not new stuff, but corporate standards dictated the rev of rhat. > >>>> > >>> [...] > >>> > >>>> I'm noticing huge differences in compile times - or any home file access > >>>> really - when doing stuff in the same home directory on the gfs on > >>>> different nodes. For instance, the same compile on one node is ~12 > >>>> minutes - on another it's 18 minutes or more (not running concurrently). > >>>> I'm also seeing weird random pauses in writes, like saving a file in vi, > >>>> what would normally take less than a second, may take up to 10 seconds. > >>>> > > > > Anyway, thought I would re-connect to you all and let you know how this > > worked out. We ended up scrapping gfs. Not because it's not a great fs, > > but because I was using it in a way that was playing to it's weak > > points. I had a lot of time and energy invested in it, and it was hard > > to let it go. Turns out that connecting to the NetApp filer via nfs is > > faster for this workload. I couldn't believe it either, as my bonnie and > > dd type tests showed gfs to be faster. But for the use case of large > > sets of very small files, and lots of stats going on, gfs simply cannot > > compete with NetApp's nfs implementation. GFS is an excellent fs, and it > > has it's place in the landscape - but for a development build system, > > the NetApp is simply phenomenal. > > > > Assuming you run both configurations (nfs-wafl vs. gfs-san) on the very > same netapp box (?) ... yes. > > Both configurations have their pros and cons. The wafl-nfs runs on > native mode that certainly has its advantages - you've made a good > choice but the latter (gfs-on-netapp san) can work well in other > situations. The biggest problem with your original configuration is the > load-balancer. The round-robin (and its variants) scheduling will not > work well if you have a write intensive workload that needs to fight for > locks between multiple GFS nodes. IIRC, there are gfs customers running > on build-compile development environment. They normally assign groups of > users on different GFS nodes, say user id starting with a-e on node 1, > f-j on node2, etc. exactly. I was about to implement the sh (source hash) scheduler in LVS, which I believe would have accomplished the same thing, only automatically, and in a statistically balanced way. Actually still might. I've had some developers test out the nfs solution and for some gfs is still better. I know that if users are pinned to a node - but can still failover in the event of node failure - this would yield the best possible performance. The main reason the IT group wants to use nfs, is for all of the other benefits, such as file-level snapshots, better backup performance, etc. Now that they see a chink in the gfs performance armor, mainly because I implemented the wrong load balancing algorithm, they're circling for the kill. I'm interested how well the nfs will scale with users vs. the gfs-san approach. > > One encouraging news from this email is gfs-netapp-san runs well on > bonnie. GFS1 has been struggling with bonnie (large amount of smaller > files within one single node) for a very long time. One of the reasons > is its block allocation tends to get spread across the disk whenever > there are resource group contentions. It is very difficult for linux IO > scheduler to merge these blocks within one single server. When the > workload becomes IO-bound, the locks are subsequently stalled and > everything start to snow-ball after that. Netapp SAN has one more layer > of block allocation indirection within its firmware and its write speed > is "phenomenal" (I'm borrowing your words ;) ), mostly to do with the > NVRAM where it can aggressively cache write data - this helps GFS to > relieve its small file issue quite well. Thanks for all of your input Wendy. -C > > -- Wendy > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
