Re: Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 3/19/2018 10:52 AM, Rik Theys wrote:
Hi,

On 03/19/2018 03:42 PM, TomK wrote:
On 3/19/2018 5:42 AM, Ondrej Valousek wrote:
Removing NFS or NFS Ganesha from the equation, not very impressed on my
own setup either.  For the writes it's doing, that's alot of CPU usage
in top. Seems bottle-necked via a single execution core somewhere trying
to facilitate read / writes to the other bricks.

Writes to the gluster FS from within one of the gluster participating
bricks:

[root@nfs01 n]# dd if=/dev/zero of=./some-file.bin

393505+0 records in
393505+0 records out
201474560 bytes (201 MB) copied, 50.034 s, 4.0 MB/s

That's not really a fare comparison as you don't specify a blocksize.
What does

dd if=/dev/zero of=./some-file.bin bs=1M count=1000 oflag=direct

give?


Rik

Correct. Higher block sizes gave me better numbers earlier. Curious about improving the small file size performance though, preferrably via gluster tunables, if possible.

Though it could be said I guess that compressing a set of large files and transferring them over that way is one solution. However needed the small block size on dd to perhaps quickly simulate alot of small requests in a somewhat ok-ish way.

Here's the numbers from the VM:

[ Via Gluster ]
[root@nfs01 n]# dd if=/dev/zero of=./some-file.bin bs=1M count=10000 oflag=direct
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 96.3228 s, 109 MB/s
[root@nfs01 n]# rm some-file.bin
rm: remove regular file âsome-file.binâ? y

[ Via XFS ]
[root@nfs01 n]# cd /bricks/0/gv01/
[root@nfs01 gv01]# dd if=/dev/zero of=./some-file.bin bs=1M count=10000 oflag=direct
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 44.79 s, 234 MB/s
[root@nfs01 gv01]#



top - 12:49:48 up 1 day,  9:39,  2 users,  load average: 0.66, 1.15, 1.82
Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu0 : 10.3 us, 9.6 sy, 0.0 ni, 28.0 id, 50.4 wa, 0.0 hi, 1.8 si, 0.0 st %Cpu1 : 13.8 us, 13.8 sy, 0.0 ni, 38.6 id, 30.0 wa, 0.0 hi, 3.8 si, 0.0 st %Cpu2 : 8.7 us, 6.9 sy, 0.0 ni, 48.7 id, 34.9 wa, 0.0 hi, 0.7 si, 0.0 st %Cpu3 : 10.6 us, 7.8 sy, 0.0 ni, 57.1 id, 24.1 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem :  3881708 total,  3543280 free,   224008 used,   114420 buff/cache
KiB Swap:  4063228 total,  3836612 free,   226616 used.  3457708 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
14115 root 20 0 2504832 27640 2612 S 43.5 0.7 432:10.35 glusterfsd 1319 root 20 0 1269620 23780 2636 S 38.9 0.6 752:44.78 glusterfs 1334 root 20 0 2694264 56988 1672 S 16.3 1.5 311:20.90 ganesha.nfsd
27458 root      20   0  108984   1404    540 D   3.0  0.0   0:00.24 dd
14127 root 20 0 1164720 4860 1960 S 0.7 0.1 1:47.59 glusterfs
  750 root      20   0  389864   5528   3988 S   0.3  0.1   0:08.77 sssd_be

--
Cheers,
Tom K.
-------------------------------------------------------------------------------------

Living on earth is expensive, but it includes a free trip around the sun.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux