Hi,
we are planning a small cluster (3 nodes in the SAN and 13 nodes with gnbd)
and I have tried gfs 6.1 from 29th march on RHEL testkernel 2.6.9-6.36.ELsmp x64 (elevator=deadline) Opteron x64 Hardware
(we want to use RHEL4 based systems, because our cluster application performes much better with
RHEL4 gcc on the opteron Hardware compared to RHEL3 gcc)
Installation was fine :-)
We have done some iozone runs on a local disk (SAN hardware is not avaliable yet) with
- gfs and lock_dlm
- gfs mounted with localcache
- ext3
- gnbd (test on remote host)
- nfs (test on remote host)
The avaliable memory of the computers was reduced to 1G to speed up the test.
There are some interesting points:
- gfs seems to perform nearly as good as ext3 only with reclen 1024 and during write
- gfs read performance seems to be not very good (are there any flags to improve it?)
- there seems to be no big difference in mount-option with localcache and using lock_dlm
- gnbd's write performance seems to be better as nfs
- nfs read performance seems to be better as gnbd's
- running two gnbd tests in parallel reduces read performance dramatically (may be an hardware issue,
because it seems to be the same with nfs)
We want to use he cluster filesystem mostly to read and proceed 2 GB datasets.
So we will test it with an application soon, which ist not a synthetic benchmark.
Is there any prefered elevator, one should use with gfs? (I will try some tests this evening)
It would be nice, if anyone with gfs experience could comment on the results.
We will have the hardware avaliable for testing until the mid of next week,
though if someone wants me to try some other configurations (including current CVS)
give me a note.
Thank you very much
Hansjörg
-- Ext3
Run began: Tue Apr 12 20:55:54 2005
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 56657 55876 38324 37653 11437 20542 2097152 128 59898 56835 38341 38075 19130 33323 2097152 256 61116 57787 35269 38075 15809 43095 2097152 512 61501 58274 37937 38266 23262 39051 2097152 1024 58998 53493 37869 35665 37163 38920 2097152 2048 58333 58310 35291 38273 47074 38147 2097152 4096 60143 60270 37873 38445 54552 37152 2097152 8192 57725 50131 37767 38112 54441 37164 2097152 16384 53178 57389 38317 38283 58545 34425
iozone test complete.
-- GFS lock_dlm
Run began: Tue Apr 12 19:16:51 2005
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 37142 37320 24185 25787 7972 22459 2097152 128 34475 34459 25381 25408 13556 21645 2097152 256 33592 33405 25220 25222 11011 21268 2097152 512 32269 31527 25081 25308 14364 23356 2097152 1024 57865 35774 24190 24919 23027 28430 2097152 2048 41923 34880 25364 24216 29515 31278 2097152 4096 46328 35647 24187 25127 32752 30084 2097152 8192 32296 34830 25172 24000 35714 31385 2097152 16384 40136 38008 24386 25317 39407 37255
iozone test complete.
-- GFS mounted with localcache
Run began: Tue Apr 12 19:17:53 2005
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 35995 35988 24151 26080 8119 21277 2097152 128 31582 35475 25624 25818 13604 22245 2097152 256 30484 34576 25344 25073 11117 21686 2097152 512 29355 35323 25542 25107 14942 24594 2097152 1024 33213 32260 24896 25344 22427 26454 2097152 2048 34852 36949 24417 25377 30143 31454 2097152 4096 42722 33431 24978 24774 32416 31650 2097152 8192 43942 32786 25752 24461 36606 33237 2097152 16384 32568 33575 25072 25506 38057 32971
- NFS mounted ext3
Run began: Tue Apr 12 22:08:04 2005
Include close in write timing
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -c -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 29933 24934 34428 34329 11396 5095 2097152 128 29848 24387 33693 33815 18280 5822 2097152 256 30033 25856 33915 33520 24666 7303 2097152 512 29868 26192 33932 32201 16207 7797 2097152 1024 30857 25485 31165 32378 27212 9730 2097152 2048 28617 24478 33258 35049 41341 10497 2097152 4096 29514 25804 33221 31459 49188 9050 2097152 8192 30777 25721 32443 32264 46874 8502 2097152 16384 28470 25419 34607 34286 57369 8056
iozone test complete.
- GNBD mounted GFS
Run began: Tue Apr 12 19:16:51 2005
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 37142 37320 24185 25787 7972 22459 2097152 128 34475 34459 25381 25408 13556 21645 2097152 256 33592 33405 25220 25222 11011 21268 2097152 512 32269 31527 25081 25308 14364 23356 2097152 1024 57865 35774 24190 24919 23027 28430 2097152 2048 41923 34880 25364 24216 29515 31278 2097152 4096 46328 35647 24187 25127 32752 30084 2097152 8192 32296 34830 25172 24000 35714 31385 2097152 16384 40136 38008 24386 25317 39407 37255
iozone test complete.
--GNBD mounted GFS (2 simultanous runs und 2 gnbd clients)
Run began: Tue Apr 12 22:33:15 2005
Using minimum file size of 2097152 kilobytes.
Using maximum file size of 2097152 kilobytes.
Auto Mode
Command line used: /opt/iozone/bin/iozone -n 2G -g 2G -a -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
2097152 64 21310 20580 9275 10662 3472 10022 2097152 128 11953 17971 8171 10598 4633 17555 2097152 256 15856 22844 5131 11313 4157 25773 2097152 512 15887 28725 4182 12117 7046 27678 2097152 1024 28444 26170 3780 10233 9771 30745 2097152 2048 38449 27667 4042 10059 12872 30426 2097152 4096 27096 35431 4699 11047 15139 31802 2097152 8192 29305 37275 4696 9728 14856 39259 2097152 16384 26824 49085 8836 5279 16865 57405
iozone test complete.
-- _________________________________________________________________
Dr. Hansjoerg Maurer | LAN- & System-Manager
|
Deutsches Zentrum | DLR Oberpfaffenhofen
f. Luft- und Raumfahrt e.V. |
Institut f. Robotik |
Postfach 1116 | Muenchner Strasse 20
82230 Wessling | 82234 Wessling
Germany |
|
Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer@xxxxxx
Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/
__________________________________________________________________There are 10 types of people in this world, those who understand binary and those who don't.
-- Linux-cluster@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/linux-cluster
