I am _pretty_sure_ its only 3.7+ based, IIRC on upstream anything 3.6 should work around it. Lemme see if I can dig up the patch that fixed the issue that lead to this regression so we can be sure, but I would try the latest 3.6. For Red Hat based distros go with 3.0.4(which has the MT epoll patch but no lookup optimize, both of which REALLY help smallfile perf). -b ----- Original Message ----- > From: "Thibault Godouet" <tibo92@xxxxxxxxxxx> > To: "Ben Turner" <bturner@xxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx > Sent: Friday, September 25, 2015 11:43:55 AM > Subject: Re: Very slow roaming profiles on top of glusterfs > > Hi Ben, > > Regarding https://bugzilla.redhat.com/show_bug.cgi?id=1250241which does > look like a serious regression for small file performance, do you know > which versions are affected, or is there a way to find out? > > Also the patch didn't make it: do you have visibility on whether another > patch is likely to land soon? > > If not I may try the version before the regression was introduced... > > Thanks, > Thibault. > On 14 Sep 2015 4:22 pm, "Ben Turner" <bturner@xxxxxxxxxx> wrote: > > > ----- Original Message ----- > > > From: "Diego Remolina" <dijuremo@xxxxxxxxx> > > > To: "Alex Crow" <acrow@xxxxxxxxxxxxxxxx> > > > Cc: gluster-users@xxxxxxxxxxx > > > Sent: Monday, September 14, 2015 9:26:17 AM > > > Subject: Re: Very slow roaming profiles on top of > > glusterfs > > > > > > Hi Alex, > > > > > > Thanks for the reply, I was aware of the performance issues with small > > > files, but never expected an order of magnitude slower. I understand > > > some improvements were made to 3.7.x to help with low small file > > > performance, however I did not see any big changes after upgrading > > > from 3.6.x to 3.7.x. > > > > > > > > http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf > > > > > > And the ssd metadata support feature seems to have not had any changes > > > since September 2014: > > > > > > https://forge.gluster.org/gluster-meta-data-on-ssd > > > > > > Am I just totally out of luck with gluster for now? > > > > Are you using glusterFS mounts or SMB mounts? As for SMB mounts we are > > working VERY hard to improve metadata / smallfile performance but as it > > sits right now we are limited by the number of lookup / stat calls that are > > issued. When we can reduce the number of lookups and prefetch the xattrs > > that SMB / windows needs(I am working on the stat prefetch but don't have a > > testable solution yet) I expect to see a vast perf improvement but I don't > > have an ETA for you. > > > > On the glusterFS side I see ~300% improvement in smallfile create > > performance between 3.6 and 3.7. Try setting: > > > > gluster volume set testvol server.event-threads 4 > > gluster volume set testvol client.event-threads 4 > > gluster volume set testvol cluster.lookup-optimize on > > > > Unfortunately WRT to metadata operations a fix went in that has negatively > > affected performance: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1250241 > > > > I used to see about 25k metatdata operations per second, now I am only > > seeking 6k. It looks like there is a patch but I don't know if the fix > > will get us back to the 25k OPs per second, maybe Pranith can comment on > > expectations for: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1250803 > > > > To summarize: > > > > SMB - no ETA for improvement > > GlusterFS smallfile create - 300% increase in my env between 3.6 and 3.7 > > GlusterFS metadata - BZ is in POST(patch is submitted) but I am not sure > > on the ETA of the fix and if the fix will get back to what I was seeing in > > 3.6 > > > > Hope this helps. > > > > -b > > > > > > > > Diego > > > > > > On Mon, Sep 14, 2015 at 8:37 AM, Alex Crow <acrow@xxxxxxxxxxxxxxxx> > > wrote: > > > > Hi Diego, > > > > > > > > I think it's the overhead of fstat() calls. Gluster keeps its metadata > > on > > > > the bricks themselves, and this has to be looked up for every file > > access. > > > > For big files this is not an issue as it only happens once, but when > > > > accessing lots of small files this overhead rapidly builds up, the > > smaller > > > > the file the worse the issue. Profiles do have hundreds of very small > > > > files! > > > > > > > > I was looking to use GlusterFS for generic file sharing as well, but I > > > > noticed the same issue while testing backups from a GlusterFS volume. > > On > > > > one > > > > vol (scanned 4-bit greyscale images and small PDFs) backups were taking > > > > over > > > > 16 hours whereas with a traditional FS they were completing in just > > over 1 > > > > hour. > > > > > > > > It may be worth trying out one of the distributed filesystems that use > > a > > > > separate in-memory metadata server. I've tried LizardFS and MooseFS and > > > > they > > > > are both much faster than GlusterFS for small files, although > > large-file > > > > sequential performance is not as good (but still plenty for a Samba > > > > server). > > > > > > > > Alex > > > > > > > > > > > > On 14/09/15 13:21, Diego Remolina wrote: > > > >> > > > >> Bump... > > > >> > > > >> Anybody has any clues as to how I can try and identify the cause of > > > >> the slowness? > > > >> > > > >> Diego > > > >> > > > >> On Wed, Sep 9, 2015 at 7:42 PM, Diego Remolina <dijuremo@xxxxxxxxx> > > wrote: > > > >>> > > > >>> Hi, > > > >>> > > > >>> I am running two glusterfs servers as replicas. I have a 3rd server > > > >>> which provides quorum. Since gluster was introduced, we have had an > > > >>> issue where windows roaming profiles are extremely slow. The initial > > > >>> setup was done on 3.6.x and since 3.7.x has small file performance > > > >>> improvements, I upgraded to 3.7.3, but that has not helped. > > > >>> > > > >>> It seems that for some reason gluster is very slow when dealing with > > > >>> lots of small files. I am not sure how to really troubleshoot this > > via > > > >>> samba, but I have come up with other tests that produce rather > > > >>> disconcerting results as shown below. > > > >>> > > > >>> If I run directly on the brick: > > > >>> [root@ysmha01 /]# time ( find > > > >>> /bricks/hdds/brick/home/jgibbs/.winprofile.V2 -type f > /dev/null ) > > > >>> real 0m3.683s > > > >>> user 0m0.042s > > > >>> sys 0m0.154s > > > >>> > > > >>> Now running on the gluster volume mounted via fuse: > > > >>> [root@ysmha01 /]# mount | grep export > > > >>> 10.0.1.6:/export on /export type fuse.glusterfs > > > >>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072) > > > >>> > > > >>> [root@ysmha01 /]# time ( find /export/home/jgibbs/.winprofile.V2 > > -type > > > >>> f > /dev/null ) > > > >>> real 0m57.812s > > > >>> user 0m0.118s > > > >>> sys 0m0.374s > > > >>> > > > >>> In general, the time to run the command on this particular user can > > be > > > >>> up to 2 minutes. If I run the command on the brick first, then it > > > >>> seems the time to run on the mounted gluster volume is lower like in > > > >>> the example above. I assume some caching in preserved. > > > >>> > > > >>> This particular user has 13,216 files in his roaming profile, which > > > >>> adds up to about 452MB of data. > > > >>> > > > >>> The server performance over samba for copying big files (both read > > and > > > >>> write) is great, I can almost max out the gigabit connections on the > > > >>> desktops. > > > >>> > > > >>> Reading from samba share on the server and writing to local drive: > > > >>> 111MB/s (Copying a 650MB iso file) > > > >>> Reading from local drive and writing to server samba share: 94MB/s > > > >>> (Copying a 3.2GB ISO file) > > > >>> > > > >>> The servers are connected to the network with 10Gbit adapters and > > also > > > >>> use separate adapters; one 10 Gbit adapter is used for services, and > > > >>> other for the backend storage communication. > > > >>> > > > >>> The servers have hardware raid controllers and the samba shares are > > on > > > >>> top of an Areca ARC-1882 controller, with a volume made out of 12 2TB > > > >>> drives in raid 6. > > > >>> > > > >>> If you can provide any steps to better troubleshoot this problem and > > > >>> fix the issue, I will really appreciate it. > > > >>> > > > >>> Diego > > > >>> > > > >>> Further details about the machines below: > > > >>> > > > >>> [root@ysmha01 /]# cat /etc/redhat-release > > > >>> CentOS Linux release 7.1.1503 (Core) > > > >>> > > > >>> [root@ysmha01 /]# gluster volume info export > > > >>> Volume Name: export > > > >>> Type: Replicate > > > >>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff > > > >>> Status: Started > > > >>> Number of Bricks: 1 x 2 = 2 > > > >>> Transport-type: tcp > > > >>> Bricks: > > > >>> Brick1: 10.0.1.7:/bricks/hdds/brick > > > >>> Brick2: 10.0.1.6:/bricks/hdds/brick > > > >>> Options Reconfigured: > > > >>> performance.io-cache: on > > > >>> performance.io-thread-count: 64 > > > >>> nfs.disable: on > > > >>> cluster.server-quorum-type: server > > > >>> performance.cache-size: 1024MB > > > >>> server.allow-insecure: on > > > >>> cluster.server-quorum-ratio: 51% > > > >>> > > > >>> Each server has dual Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz with > > > >>> 32GB of memory. > > > >> > > > >> _______________________________________________ > > > >> Gluster-users mailing list > > > >> Gluster-users@xxxxxxxxxxx > > > >> http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > -- > > > > This message is intended only for the addressee and may contain > > > > confidential information. Unless you are that person, you may not > > > > disclose its contents or use it in any way and are requested to delete > > > > the message along with any attachments and notify us immediately. > > > > "Transact" is operated by Integrated Financial Arrangements plc. 29 > > > > Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 > > > > 5300. (Registered office: as above; Registered in England and Wales > > > > under number: 3727592). Authorised and regulated by the Financial > > > > Conduct Authority (entered on the Financial Services Register; no. > > 190856). > > > > > > > > . > > > > > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users@xxxxxxxxxxx > > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users@xxxxxxxxxxx > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users@xxxxxxxxxxx > > http://www.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
