On Fri, Apr 28, 2017 at 3:44 PM, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote:
Dear Gluster community,
I have problems with tuning Gluster for small files performance on SSD.
My usage scenario is, as I've learned, worst possible scenario, but
it's not up to me to change it:
- small 1KB files
- at least 20M of those
- approx. 10 files/directory
- mostly writes
- average speed 1000 files/sec with peaks up to 10K files/sec.
I'm doing something wrong, because I cannot get past performance metrics
4K files/sec for distributed volume (2 bricks)
2K files/sec for replicated volume (2 bricks).
I've been experimenting with various XFS formatting and mounting
options and with Gluster tuning (md-cache, lookup optimize, thread,
writeback, tiering), but no luck.
I've learned that it's not disk IO that is the bottleneck (direct
tests on mounted XFS partition show waaay better results, like 100K
files/sec).
As I've learned from
http://blog.gluster.org/2014/03/experiments-using-ssds- with-gluster/
it's possible to get 24K files/sec performance (and that was three years ago).
How many clients are you running? Considering Gluster is a distributed solution, the performance should be measured as aggregate of all the clients.
Test I'm using, run on one server (2 x Xeons, 256 GB RAM, 10GbE network):
smallfile_cli.py --operation create --threads 32 --file-size 1 --files
15625 --top /mnt/testdir/test
Setup:
2 servers with 2 Xeons each, 256GB RAM, 8 x 800GB SSD drives in RAID6,
10GbE network
Ubuntu 14.04
Gluster 3.7.3
Do you have any hints where I should start investigating for the bottleneck?
Currently just the fuse mount may be the bottleneck, but I recommend running multiple clients (from different machines) doing these operations in parallel to get the best results out of Gluster.
-Amar
Best regards,
Szymon
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
--
Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
