Hi,
So I used Sanjay's setup to get these numbers. So I'm guessing it's a 10G network. I will check again and let you know if that isn't the case.On Tue, Jun 6, 2017 at 9:38 PM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote:
Nice work!What is the network interconnect bandwidth? How much of the network bandwidth is in use while the test is being run? Wondering if there is saturation in the network layer.-VijayOn Tue, Jun 6, 2017 at 7:35 AM, Krutika Dhananjay <kdhananj@xxxxxxxxxx> wrote:______________________________Based on a 60-second run of randrd test and subsequent analysis of the stats dumped by the individual io-stats instances, the following is what I found:On the brick stack: Below protocol/server, above and below io-threads and just above storage/posix.On the client stack: Above client-io-threads and above protocol/client-0 (the first child of AFR).3 node cluster; 1x3 plain replicate volume with group virt settings, direct-io.Before I get to the results, a little bit about the configuration ...Hi,As part of identifying performance bottlenecks within gluster stack for VM image store use-case, I loaded io-stats at multiple points on the client and brick stack and ran randrd test using fio from within the hosted vms in parallel.3 FUSE clients, one per node in the cluster (which implies reads are served from the replica that is local to the client).io-stats was loaded at the following places:Translator Position Avg Latency of READ fop as seen by this translator1. parent of client-io-threads1666us ∆ (1,2) = 50us2. parent of protocol/client-0 1616us∆ (2,3) = 1453us----------------- end of client stack -------------------------------------- beginning of brick stack -----------3. child of protocol/server163us ∆ (3,4) = 7us4. parent of io-threads156us ∆ (4,5) = 20us5. child-of-io-threads136us ∆ (5,6) = 11us6. parent of storage/posix125us
...---------------- end of brick stack ------------------------So it seems like the biggest bottleneck here is a combination of the network + epoll, rpc layer?I must admit I am no expert with networks, but I'm assuming if the client is reading from the local brick, theneven latency contribution from the actual network won't be much, in which case bulk of the latency is coming from epoll, rpc layer, etc at both client and brick end? Please correct me if I'm wrong.I will, of course, do some more runs and confirm if the pattern is consistent.-Krutika_________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-devel
