I get well over 2k IOPs in my OLD 12 disk RAID 6 HW in the lab(4 nodes 2x2 volume): https://access.redhat.com/sites/default/files/attachments/rhgs3_1_perfbrief_portalv1.pdf That data is from 3.1, things have improved much since then(I think closer to 3.2k IOPs on the same HW?). I have a total of 48 disks though(20 data, 8 parity, 20 redundancy) I'm not sure what you have. I can extract a kernel in between 4 minutes and 1 min 30 secs depending on tunibles and if I use multi threaded TAR tools developed by Ben England. If you don't have access to the RH paywall you will just have to trust me since the perf brief requires a sub. The key to getting smallfile perf out of gluster is to use multiple threads and multiple clients. What is your back end like? -b ----- Original Message ----- > From: "Gandalf Corvotempesta" <gandalf.corvotempesta@xxxxxxxxx> > To: "Szymon Miotk" <szymon.miotk@xxxxxxxxx>, "gluster-users" <Gluster-users@xxxxxxxxxxx> > Sent: Friday, October 13, 2017 3:56:14 AM > Subject: Re: small files performance > > Where did you read 2k IOPS? > > Each disk is able to do about 75iops as I'm using SATA disk, getting even > closer to 2000 it's impossible > > Il 13 ott 2017 9:42 AM, "Szymon Miotk" < szymon.miotk@xxxxxxxxx > ha scritto: > > > Depends what you need. > 2K iops for small file writes is not a bad result. > In my case I had a system that was just poorly written and it was > using 300-1000 iops for constant operations and was choking on > cleanup. > > > On Thu, Oct 12, 2017 at 6:23 PM, Gandalf Corvotempesta > < gandalf.corvotempesta@xxxxxxxxx > wrote: > > So, even with latest version, gluster is still unusable with small files ? > > > > 2017-10-12 10:51 GMT+02:00 Szymon Miotk < szymon.miotk@xxxxxxxxx >: > >> I've analyzed small files performance few months ago, because I had > >> huge performance problems with small files writes on Gluster. > >> The read performance has been improved in many ways in recent releases > >> (md-cache, parallel-readdir, hot-tier). > >> But write performance is more or less the same and you cannot go above > >> 10K smallfiles create - even with SSD or Optane drives. > >> Even ramdisk is not helping much here, because the bottleneck is not > >> in the storage performance. > >> Key problems I've noticed: > >> - LOOKUPs are expensive, because there is separate query for every > >> depth level of destination directory (md-cache helps here a bit, > >> unless you are creating lot of directories). So the deeper the > >> directory structure, the worse. > >> - for every file created, Gluster creates another file in .glusterfs > >> directory, doubling the required IO and network latency. What's worse, > >> XFS, the recommended filesystem, doesn't like flat directory sturcture > >> with thousands files in each directory. But that's exactly how Gluster > >> stores its metadata in .glusterfs, so the performance decreases by > >> 40-50% after 10M files. > >> - complete directory structure is created on each of the bricks. So > >> every mkdir results in io on every brick you have in the volume. > >> - hot-tier may be great for improving reads, but for small files > >> writes it actually kills performance even more. > >> - FUSE driver requires context switch between userspace and kernel > >> each time you create a file, so with small files the context switches > >> are also taking their toll > >> > >> The best results I got were: > >> - create big file on Gluster, mount it as XFS over loopback interface > >> - 13.5K smallfile writes. Drawback - you can use it only on one > >> server, as XFS will crash when two servers will write to it. > >> - use libgfapi - 20K smallfile writes performance. Drawback - no nice > >> POSIX filesystem, huge CPU usage on Gluster server. > >> > >> I was testing with 1KB files, so really small. > >> > >> Best regards, > >> Szymon Miotk > >> > >> On Fri, Oct 6, 2017 at 4:43 PM, Gandalf Corvotempesta > >> < gandalf.corvotempesta@xxxxxxxxx > wrote: > >>> Any update about this? > >>> I've seen some works about optimizing performance for small files, is > >>> now gluster "usable" for storing, in example, Maildirs or git sources > >>> ? > >>> > >>> at least in 3.7 (or 3.8, I don't remember exactly), extracting kernel > >>> sources took about 4-5 minutes. > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users@xxxxxxxxxxx > >>> http://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
