Do both read and write throughput peak at 1Gbit/s? What is the block size used for performing I/O? Can you get the output of - 1. dd if=/dev/zero of=/mnt/stripe/file bs=1M count=1K 2. dd if=/mnt/stripe/file of=/dev/null bs=1M count=1K Just one instance of dd is enough as the client network interface (10Gbit/s) has enough juice to saturate 4x1Gbit servers. Avati On Fri, Dec 3, 2010 at 6:06 PM, Gotwalt, P. <peter.gotwalt at uci.ru.nl> wrote: > Craig, > > Using multiple parallel bonnie++ benchmarks (4,8,16) does use several > files. These file are 1GB each, and we take care there will be at least > 32 of them. As we have multiple processes (4,8,16 bonnie++s) and each > uses several files, we spread the io over different storage bricks. I > can see this when monitoring network and disk activity on the bricks. > For example: when bonnie++ does block read/writes on a striped (4 > bricks) volume I notice that the load of the client (network throughput) > is evenly spread over the 4 nodes. These nodes have enough cpu, memory, > network and disk resources left! The accoumulated throughput doesn't get > over the 1 Gb. > The 10Gb nic at the client is set to fixed 10Gb, full duplex, All the > nics on the storage bricks are 1Gb, fixed, full duplex. The 10Gb client > (dual quadcore, 16GB) has plenty of resources to run 16 bonnie++s > parallel. We should be able to get more than this 1Gb throughput, > especially with a striped volume. > > What kind of benchmarks do you run? And with what kind of setup? > > Peter > > > > > Peter - > > Using Gluster the performance of any single file is going to be > > limited to the performance of the server on which it exists, or in the > > > case of a striped volume of the server on which the segment of the > file > > you are accessing exists. If you were able to start 4 processes, > > accessing different parts of the striped file, or lots of different > > files in a distribute cluster you would see your performance increase > > significantly. > > > Thanks, > > > Craig > > > --> > > Craig Carl > > Senior Systems Engineer > > Gluster > > > > > > On 11/26/2010 07:57 AM, Gotwalt, P. wrote: > > > Hi All, > > > > > > I am doing some tests with gluster (3.1) and have a problem of not > > > getting higher throughput than 1 Gb (yes bit!) with 4 storage > bricks. > > > My setup: > > > > > > 4 storage bricks (dualcore, 4GB mem) each with 3 sata 1Tb disks, > > > connected to a switch with 1 Gb nics. In my tests I only use 1 SATA > > > disk as a volume, per brick. > > > 1 client (2xquad core, 16 GB mem) with a 10Gb nic to the same switch > as > > > the bricks. > > > > > > When using striped of distributed configurations, with all 4 bricks > > > configured to act as a server, the performance will never be higher > than > > > just below 1 Gb! I tested with 4, 8 and 16 parallel bonnie++ runs. > > > > > > The idea is that parallel bonnie's create enough files to get > > > distributed over the storage bricks. And all this bonnie's wil > deliver > > > enough throughput to fill up this 10Gb line. I expect the throughput > to > > > be maximum 4Gb because that's the maximum the 4 storage bricks > together > > > can produce. > > > > > > I also tested the throughput of the network with iperf3 and got: > > > - 5Gb to a second temporary client on another switch 200 Km from my > > > site, connected with a 5Gb fiber > > > - 908-920 Mb to the interfaces of the bricks. > > > So the network seems ok. > > > > > > Can someone advise me on why I don't get 4Gb? Or can someone advise > me > > > on a better setup with the equipment I have? > > > > > > > > > Peter Gotwalt > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
