On 04/20/2011 03:12 PM, Mohit Anchlia wrote: > I guess I am not clear enough :) I am running multiple concurrent > threads (in parallel) writing actual files and OVERALL combined > throughput (from all the threads) never go above 20MB/s. It's not just > running one single dd at a time. > > gluster volume info all > > Volume Name: stress-volume > Type: Distributed-Replicate > Status: Started > Number of Bricks: 3 x 2 = 6 > Transport-type: tcp > Bricks: > Brick1: dsdb1:/data/gluster > Brick2: dsdb2:/data/gluster > Brick3: dsdb3:/data/gluster > Brick4: dsdb4:/data/gluster > Brick5: dsdb5:/data/gluster > Brick6: dsdb6:/data/gluster Ok ... your write bandwidth is an example of something called a zero sum game. No matter how many threads you throw at it, you will never exceed the maximum bandwidth (sum of the bandwidths per thread). In the simplest case, you have 1 write thread which will get 20 MB/s. With two simultaneous writers, each may get ~10 MB/s. 4 simultaneous would be about 5 MB/s. Now add in the mirroring operation while doing the write. You have 1 write and one mirror operation going. Thats 10MB/s per thread. 10 for the write, and 10 for the mirror. What I am getting at is that mirroring is not cheap on the bandwidth level of the system. Given the low performance of this unit to begin with, mirroring is going to be problematic in terms of the amount of bandwidth consumed and thus available for other processes. Take the fio input decks I indicated in the previous post, and vary the number of jobs from 1 to say 24. Please report what happens to the aggregate write bandwidth (it will report it). You only need a few points, say 1, 4, 8, 12, 16, 20, 24 to see the effect. Assuming my guess is correct, the bandwidth won't be constant, but will drop rapidly with the oversubscription of write threads. You will get 2 writes for every fs write. So at some point the bandwidth will drop rapidly. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman at scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
