Where did you read 2k IOPS?
Each disk is able to do about 75iops as I'm using SATA disk, getting even closer to 2000 it's impossible
Il 13 ott 2017 9:42 AM, "Szymon Miotk" <szymon.miotk@xxxxxxxxx> ha scritto:
Depends what you need.
2K iops for small file writes is not a bad result.
In my case I had a system that was just poorly written and it was
using 300-1000 iops for constant operations and was choking on
cleanup.
On Thu, Oct 12, 2017 at 6:23 PM, Gandalf Corvotempesta
<gandalf.corvotempesta@gmail.com > wrote:
> So, even with latest version, gluster is still unusable with small files ?
>
> 2017-10-12 10:51 GMT+02:00 Szymon Miotk <szymon.miotk@xxxxxxxxx>:
>> I've analyzed small files performance few months ago, because I had
>> huge performance problems with small files writes on Gluster.
>> The read performance has been improved in many ways in recent releases
>> (md-cache, parallel-readdir, hot-tier).
>> But write performance is more or less the same and you cannot go above
>> 10K smallfiles create - even with SSD or Optane drives.
>> Even ramdisk is not helping much here, because the bottleneck is not
>> in the storage performance.
>> Key problems I've noticed:
>> - LOOKUPs are expensive, because there is separate query for every
>> depth level of destination directory (md-cache helps here a bit,
>> unless you are creating lot of directories). So the deeper the
>> directory structure, the worse.
>> - for every file created, Gluster creates another file in .glusterfs
>> directory, doubling the required IO and network latency. What's worse,
>> XFS, the recommended filesystem, doesn't like flat directory sturcture
>> with thousands files in each directory. But that's exactly how Gluster
>> stores its metadata in .glusterfs, so the performance decreases by
>> 40-50% after 10M files.
>> - complete directory structure is created on each of the bricks. So
>> every mkdir results in io on every brick you have in the volume.
>> - hot-tier may be great for improving reads, but for small files
>> writes it actually kills performance even more.
>> - FUSE driver requires context switch between userspace and kernel
>> each time you create a file, so with small files the context switches
>> are also taking their toll
>>
>> The best results I got were:
>> - create big file on Gluster, mount it as XFS over loopback interface
>> - 13.5K smallfile writes. Drawback - you can use it only on one
>> server, as XFS will crash when two servers will write to it.
>> - use libgfapi - 20K smallfile writes performance. Drawback - no nice
>> POSIX filesystem, huge CPU usage on Gluster server.
>>
>> I was testing with 1KB files, so really small.
>>
>> Best regards,
>> Szymon Miotk
>>
>> On Fri, Oct 6, 2017 at 4:43 PM, Gandalf Corvotempesta
>> <gandalf.corvotempesta@gmail.com > wrote:
>>> Any update about this?
>>> I've seen some works about optimizing performance for small files, is
>>> now gluster "usable" for storing, in example, Maildirs or git sources
>>> ?
>>>
>>> at least in 3.7 (or 3.8, I don't remember exactly), extracting kernel
>>> sources took about 4-5 minutes.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
