Here are my results:
Summary: I am not able to reproduce the problem, IOW I get relatively
equivalent numbers for sequential IO when going against 3.10.5 or 3.12.0
Next steps:
- Could you pass along your volfile (both for a brick and also the
client vol file (from
/var/lib/glusterd/vols/<yourvolname>/patchy.tcp-fuse.vol and a brick vol
file from the same place)
- I want to check what options are in use in your setup as compared
to mine and see if that makes a difference
- Is it possible for you to run the IOZone test, as below (it needs more
clarification in case you have not used IOZone before, so reach out in
that case) in your setup and report the results?
- Details:
Test: IOZone iozone -+m /<clientlistfile> -+h <hostaddr> -C -w -c -e -i
0 -+n -r 256k -s 2g -t 4/8/16/32
- Sequential IOZone write tests, with -t number of threads (files)
per client, across 4 clients, and using a 256k record size
- Essentially 8/16/32 threads per client, which are 4 in total
Volume: 6x(4+2) disperse, on 36 disks, each disk is a SAS 10k JBOD,
configured with the defaults, when creating the volume using 3.10.5 as a
start point.
Server network saturation expectation: The brick nodes will get 1.5
times the data that the client generated (4+2). As a result, aggregate
IOZone results should be seen as, 1.5XAggregate/4 per server network
utilization.
Results:
Threads/client 3.10.5 3.12.0
(bytes/sec aggregate)
8 1938564 1922249
16 2044677 2082817
32 2465990 2435669
The brick nodes (which are separate from the client nodes) have a
(greater than) 10G interface.
At best (32 threads/client case), I see the server link getting utilized as,
3.10.5: (2465990*1.5)/(4*1024) = 903MB/sec
3.12.0: (2465990*1.5)/(4*1024) = 892MB/sec
Shyam
On 09/07/2017 12:07 AM, Serkan Çoban wrote:
It is sequential write with file size 2GB. Same behavior observed with
3.11.3 too.
On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana@xxxxxxxxxx> wrote:
On 09/06/2017 05:48 AM, Serkan Çoban wrote:
Hi,
Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
100 clients are writing each has 5 threads total 500 threads.
With 3.10.5 each server has 800MB/s network traffic, cluster total is
32GB/s
With 3.12.0 each server has 200MB/s network traffic, cluster total is
8GB/s
I did not change any volume options in both configs.
I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume
and compared this against 3.10.5 and 3.12.0. The tests are no where near
your capacity, but I do not see anything alarming in the results. (4 server,
4 clients, 4 worker thread per client)
I do notice a 6% drop in Sequential and random write performance, and gains
in the sequential and random reads.
I need to improve the test to do larger files and for a longer duration,
hence not reporting any numbers as yet.
Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
remounting on the clients (after the versions were upgraded there as well).
I guess your test can be characterized as a sequential write workload
(ingestion of data). What is the average file size being ingested? I can
mimic something equivalent to that to look at this further.
I would like to ensure there are no evident performance regressions as you
report.
Shyam
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
