Re: Problem during reproducing smallfile experiment on Gluster 10

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



I think there could be multiple reasons for not getting similar improvement as we were getting.

1) You are executing a test case in a VM environment. The resources are limited as compared to a physical machine.
2) The improvement was showing after enable tcmalloc and disable glusterfs based mempool, I am not
   sure how the memory managed in case of vm environment. On physical server there is no doubt we were
   getting significant improvement.
3) Can you share what gluster options you have enabled to run the test ?
     gluster v info
4) What is smallfile parameters you have used to run the test case (no_of_file and no_of_threads)

Mohit Agrawal

On Thu, Jan 20, 2022 at 12:24 PM 박현승 <hspark0582@xxxxxxxxxxx> wrote:

Dear Gluster developers,


This is Hyunseung Park at Gluesys, South Korea.


We are trying to replicate the test in but to no avail.

In our experiments, Gluster version 10 unfortunately did not perform noticably better than version 9.



v9 2x2       average
create 3399.99 3484.79 2702.57 3195.783333
ls -l 65605.2 64930.6 72018.7 67518.16667
chmod 4858.95 4965.29 5597.73 5140.656667
stat 7334.88 7755.89 8335.11 7808.626667
read 7015.64 8255.48 7007.01 7426.043333
append 2554.93 2777.65 2572.57 2635.05
mkdir 1800.29 1865.07 1805.48 1823.613333
rmdir 1854.09 1722.89 1876.81 1817.93
cleanup 2402.02 2447.36 2438.71 2429.363333
v10 2x2       average
create 3741.39 3174.82 3234.42 3383.543333
ls -l 71543.7 67275.9 72975.1 70598.23333
chmod 5441.11 5109.22 5004.08 5184.803333
stat 7746.37 7677.99 7885.72 7770.026667
read 7061.12 7165.21 7121.07 7115.8
append 3458.93 2641.84 2887.46 2996.076667
mkdir 2685.22 1879.35 1970.91 2178.493333
rmdir 2240.11 1648.37 1602.16 1830.213333
cleanup 3739.68 2407.57 2403.48 2850.243333


The result above is from the test that deployed 32 threads on each of the 4 clients.

Some results were better than others, but it is not good enough when compared to the result in the aforementioned link.

We are wondering what we can do to get the full potential of the new version.


We have been running tests with varying file sizes, number of threads, different volume topology, etc. but we were not able to see data conclusive enough.

We were also not able to find meaningful output from running tests using other benchmark tools such as bonnie++ and FIO.


To find the potential cause we tried to look into the program by calling malloc_stats() and using perf.

However, we also could not find something noteworthy from the result.

Here is the data recorded during one set of smallfile test (from create to cleanup):


Below is our test environment:


Basic HW info: VM (vSphere), 2 core CPU, 4G RAM. 4 servers and 4 clients.

OS: Centos 7

kernel version: 3.10.0-1160

Gluster version: built from git, checked out from branch 'release-9' and 'release-10' respectively (commits 7094da and a804f8)

build option: default setting except "./configure --disable-linux-io_uring" (kernel does not support io_uring)

Configuration result (case of version 10):

GlusterFS configure summary
FUSE client          : yes
epoll IO multiplex   : yes
fusermount           : yes
readline             : no
georeplication       : yes
Linux-AIO            : yes
Linux io_uring       : no
Use liburing         : no
Enable Debug         : no
Run with Valgrind    : no
Sanitizer enabled    : none
XML output           : yes
Unit Tests           : no
Track priv ports     : yes
POSIX ACLs           : yes
SELinux features     : yes
firewalld-config     : no
Events               : yes
EC dynamic support   : x64 sse avx
Use memory pools     : no
Nanosecond m/atimes  : yes
Server components    : yes
Legacy gNFS server   : no
IPV6 default         : no
Use TIRPC            : yes
With Python          : 3.6
Cloudsync            : yes
Metadata dispersal   : no
Link with TCMALLOC   : yes
Enable Brick Mux     : no
Building with LTO    : no





Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

Gluster-devel mailing list


Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

Gluster-devel mailing list

[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux