I agree it is a matter of tuning. So what do we tune? I am not stuck
on 2.0.9, there just was no performance benefit for my data set.
(Millions of 8-50 megabyte files) My test system has 12 Gigabytes of
RAM which is why I used a size of 16 Gigabytes so that cache is not a
factor in the test. But away what did you learn from these results?
Read performance is the most important to me. (That is because the
write performance is pretty good already)
Thanks again.
Todd
iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Fri Jun 11 06:48:48 2010
Auto Mode
File size set to 8388608 KB
Record Size 64 KB
Command line used: iozone -a -i0 -i1 -s 8192m -r 64k iozone.1059.tmp
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
8388608 64 394906 728411 266523 255494
iozone test complete.
iozone -a -i0 -i1 -s 8192m -r 128k iozone.$$.tmp
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Fri Jun 11 06:53:03 2010
Auto Mode
File size set to 8388608 KB
Record Size 128 KB
Command line used: iozone -a -i0 -i1 -s 8192m -r 128k iozone.1059.tmp
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
8388608 128 406930 746811 233122 225809
iozone test complete.
iozone -a -i0 -i1 -s 8192m -r 1m iozone.$$.tmp
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Fri Jun 11 06:55:56 2010
Auto Mode
File size set to 8388608 KB
Record Size 1024 KB
Command line used: iozone -a -i0 -i1 -s 8192m -r 1m iozone.1059.tmp
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
8388608 1024 459811 717837 231110 219715
iozone test complete.
glusterfs with /dev/ram0 (cluster)
iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Fri Jun 11 06:47:11 2010
Auto Mode
File size set to 8388608 KB
Record Size 64 KB
Command line used: iozone -a -i0 -i1 -s 8192m -r 64k iozone.1059.tmp
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
8388608 64 1038493 1171185 511793 508550
iozone test complete.
On Fri, Jun 11, 2010 at 6:25 AM, Harshavardhana <harsha at gluster.com> wrote:
> On 06/10/2010 08:52 PM, Todd Daugherty wrote:
>>
>> I am not sure what you are saying? The default record size of
>> ib_write_bw is 64K. But here is 128K.
>>
>> Do you think that the QDR Infiniband is not doing 3.3 gigabytes per
>> seconds? And that is why I am only getting 500 megabytes per seconds
>> via GlusterFS? You think that it is something wrong with the QDR
>> setup? What are the actual commands that you want me run on the
>> machines? I will run them.
>>
>> Do you have a single stream GlusterFS that can go OVER 3.3 gigabytes
>> per second? What setup is it?
>>
>
> Excuse me for not being clear
>
> you need to run "iozone" with higher record sizes. Your earlier command
> suggested you used "16k" for creating a 8GB file.
>
> 16k chunk writes over 2.0.9 version has lot of performance penalty, i think
> that is the reason why you are seeing such a slowness.
>
> Please run the below commands
>
> "iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp"
> "iozone -a -i0 -i1 -s 8192m -r 128k iozone.$$.tmp"
> "iozone -a -i0 -i1 -s 8192m -r 1m iozone.$$.tmp"
>
> Since your infiniband fabric looks quite alright and also your backend disk
> performing at 1GB/sec. ?We should be near to 700-850MB/sec over glusterfs
> for writes as well as reads, if not then its a tuning or configuration
> issue.
>
> Thanks
>
> --
> Harshavardhana
> Gluster Inc - http://www.gluster.com
> +1(408)-770-1887, Ext-113
> +1(408)-480-1730
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
