Hi
I have done some futher testing on this issue, and I noticed an interesting behavior.
We have an installation with RHEL4 on i386 hardware (old SAN) and one under
new x86_64 (dual CPU each hyptherthreading).
On i386 read performance is better than write performance (under ext3 and gfs)
On x86_64 read performance is better the write performance under ext3
but two times bader under gfs (Filsystem created with lock_nolock)
Here a short summary: i386 ext3 write: 1:16 read: 0:55 read: 0:43 (blockdev --setra 8192)
i386 ext3 write: 1:30 read: 1:08 read: 1:08 (blockdev --setra 8192)
x86_64 ext3 write: 0:35 read: 0:27 read: 0:18 (blockdev --setra 8192)
x86_64 gfs write: 0:27 read: 1:03 read: 1:03 (blockdev --setra 8192)
The gnbd tests I did last week were under x86_64 to, so that might not be an gnbd issue, but an gfs issue under
x86_64.
The tests where done under
2.6.9-5.0.3.ELsmp and 2.6.9-6.38.ELsmp with current GFS from CVS (RHEL4 TAG) with no difference.
Can anyone reproduce this behavior?
Greetings
Hansjörg
Here the detailed tests
[root@chianti sda1]# uname -a
Linux chianti.itsd.de 2.6.9-6.38.EL #1 Wed Apr 13 01:36:09 EDT 2005 i686 athloni386 GNU/Linux
[root@chianti ~]# mkfs.ext3 /dev/sda1
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
Dateisystem-Label=
OS-Typ: Linux
BlockgröÃe=4096 (log=2)
FragmentgröÃe=4096 (log=2)
5341184 Inodes, 10673176 Blöcke
533658 Blöcke (5.00%) reserviert für den Superuser
erster Datenblock=0
Maximum filesystem blocks=12582912
326 Blockgruppen
32768 Blöcke pro Gruppe, 32768 Fragmente pro Gruppe
16384 Inodes pro Gruppe
Superblock-Sicherungskopien gespeichert in den Blöcken:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Schreibe Inode-Tabellen: erledigt inode.i_blocks = 98312, i_size = 4243456 Erstelle Journal (8192 Blöcke): erledigt Schreibe Superblöcke und Dateisystem-Accountinginformationen: erledigt
Das Dateisystem wird automatisch alle 28 Mounts bzw. alle 180 Tage überprüft,
je nachdem, was zuerst eintritt. Veränderbar mit tune2fs -c oder -t .
[root@chianti ~]# mount /dev/sda1 /mnt/sda1/
[root@chianti ~]# cd /mnt/sda1
[root@chianti sda1]# free
total used free shared buffers cached
Mem: 515736 164504 351232 0 43544 25104
-/+ buffers/cache: 95856 419880
Swap: 786424 152 786272
[root@chianti sda1]# time mkfile 1000M a
real 1m16.242s user 0m0.009s sys 0m16.273s [root@chianti sda1]# time mkfile 1000M b
real 1m17.738s user 0m0.009s sys 0m15.511s [root@chianti sda1]# time cat a > /dev/null
real 0m55.025s user 0m0.193s sys 0m6.529s
[root@chianti sda1]# time cat b > /dev/null
real 0m54.241s user 0m0.220s sys 0m6.331s
[root@chianti sda1]# blockdev --setra 8192 /dev/sda1 [root@chianti sda1]# time cat a > /dev/null
real 0m43.565s user 0m0.189s sys 0m6.014s
[root@chianti sda1]# time cat b > /dev/null
real 0m43.214s user 0m0.196s sys 0m6.727s
[root@chianti ~]# gfs_mkfs -p lock_nolock -j 3 /dev/sda1 This will destroy any data on /dev/sda1. It appears to contain a EXT2/3 filesystem.
Are you sure you want to proceed? [y/n] y Device: /dev/sda1 Blocksize: 4096 Filesystem Size: 10573560 Journals: 3 Resource Groups: 162 Locking Protocol: lock_nolock Lock Table:
Syncing... All Done
[root@chianti sda1]# time mkfile 1000M a
real 1m27.421s user 0m0.010s sys 0m12.202s [root@chianti sda1]# time mkfile 1000M b
real 1m35.009s user 0m0.006s sys 0m12.513s [root@chianti sda1]# time cat a > /dev/null
real 1m12.609s user 0m0.153s sys 0m9.980s [root@chianti sda1]# time cat b > /dev/null
real 1m7.989s user 0m0.154s sys 0m10.427s
[root@chianti sda1]# blockdev --setra 256 /dev/sda1 [root@chianti sda1]# time cat a > /dev/null
real 1m8.402s user 0m0.082s sys 0m8.841s [root@chianti sda1]# time cat b > /dev/null
real 1m8.647s user 0m0.124s sys 0m9.565s [root@chianti sda1]# blockdev --setra 8192 /dev/sda1 [root@chianti sda1]# time cat a > /dev/null
real 1m8.419s user 0m0.115s sys 0m9.262s
[root@rmvbs02 ~]# uname -a
Linux rmvbs02.cluster.robotic.dlr.de 2.6.9-5.0.3.ELsmp #1 SMP Sat Feb 19 15:45:14 CST 2005 x86_64 x86_64 x86_64 GNU/Linux
[root@rmvbs02 ~]# mkfs.ext3 /dev/sdb1
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 131072, rsv_gdb = 977
Dateisystem-Label=
OS-Typ: Linux
BlockgröÃe=4096 (log=2)
FragmentgröÃe=4096 (log=2)
97615872 Inodes, 195221872 Blöcke
9761093 Blöcke (5.00%) reserviert für den Superuser
erster Datenblock=0
Maximum filesystem blocks=4294967296
5958 Blockgruppen
32768 Blöcke pro Gruppe, 32768 Fragmente pro Gruppe
16384 Inodes pro Gruppe
Superblock-Sicherungskopien gespeichert in den Blöcken:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Schreibe Inode-Tabellen: erledigt inode.i_blocks = 140696, i_size = 4243456 Erstelle Journal (8192 Blöcke): erledigt Schreibe Superblöcke und Dateisystem-Accountinginformationen: erledigt
Das Dateisystem wird automatisch alle 38 Mounts bzw. alle 180 Tage überprüft,
je nachdem, was zuerst eintritt. Veränderbar mit tune2fs -c oder -t .
[root@rmvbs02 tmp]# free total used free shared buffers cached Mem: 1026364 65960 960404 0 684 15544 -/+ buffers/cache: 49732 976632 Swap: 2739040 1360 2737680 [root@rmvbs02 tmp]# time mkfile 2000M a
real 0m35.560s user 0m0.005s sys 0m7.596s [root@rmvbs02 tmp]# time mkfile 2000M b
real 0m30.917s user 0m0.005s sys 0m7.707s [root@rmvbs02 tmp]# time mkfile 2000M c
real 0m40.663s user 0m0.010s sys 0m7.806s [root@rmvbs02 tmp]# time cat a > /dev/null
real 0m28.787s user 0m0.122s sys 0m2.818s [root@rmvbs02 tmp]# time cat b > /dev/null
real 0m27.468s user 0m0.124s sys 0m2.678s [root@rmvbs02 tmp]# blockdev --getra /dev/sdb1 128 [root@rmvbs02 tmp]# blockdev --setra 8192 /dev/sdb1 [root@rmvbs02 tmp]# time cat c > /dev/null
real 0m18.541s user 0m0.105s sys 0m2.064s [root@rmvbs02 tmp]# time cat a > /dev/null
real 0m18.464s user 0m0.117s sys 0m2.035s
[root@rmvbs02 ~]# gfs_mkfs -p lock_nolock -j 3 /dev/sdb1 This will destroy any data on /dev/sdb1. It appears to contain a EXT2/3 filesystem.
Are you sure you want to proceed? [y/n] y
Device: /dev/sdb1 Blocksize: 4096 Filesystem Size: 195108656 Journals: 3 Resource Groups: 2978 Locking Protocol: lock_nolock Lock Table:
Syncing... All Done [root@rmvbs02 ~]# mount -t gfs /dev/sdb1 /mnt/tmp/ [root@rmvbs02 ~]# cd /mnt/tmp/ [root@rmvbs02 tmp]# time mkfile 2000M a
real 0m24.616s user 0m0.004s sys 0m4.892s [root@rmvbs02 tmp]# time mkfile 2000M b
real 0m27.023s user 0m0.005s sys 0m5.027s [root@rmvbs02 tmp]# time mkfile 2000M c
real 0m29.205s user 0m0.005s sys 0m5.163s
[root@rmvbs02 tmp]# time cat a > /dev/null
real 1m4.698s user 0m0.120s sys 0m6.138s [root@rmvbs02 tmp]# time cat b > /dev/null
real 1m2.958s user 0m0.132s sys 0m6.175s [root@rmvbs02 tmp]# time cat c > /dev/null
real 1m2.867s user 0m0.109s sys 0m6.079s [root@rmvbs02 tmp]# blockdev --getra /dev/sdb1 8192 [root@rmvbs02 tmp]# blockdev --setra 256 /dev/sdb1 [root@rmvbs02 tmp]# time cat a > /dev/null
real 1m2.931s user 0m0.101s sys 0m6.073s
Benjamin Marzinski wrote:
On Thu, Apr 21, 2005 at 07:58:49AM +0200, Hansjörg Maurer wrote:
Hi
thank you very much,
Do you have any idea what causes the speed difference between vanilla and RHEL4 kernel?
Should I officially file a bug in bugzilla, or do you take care of the problem?
Go ahead and file a bug on gnbd if you are interested in tracking the status of this. I'll put it on my list of things to do.
-Ben
Greetings
Hansjörg
I got those numbers on a vanilla 2.6.11 kernel.
Running on a 2.6.9-6.24.EL kernel, I get
45.6608 MB/sec writes 35.8578 MB/sec reads
Not as pronouced as your numbers, but a noticeable speed difference.
-Ben
Thank you very much
Hansjörg
-- _________________________________________________________________
Dr. Hansjoerg Maurer | LAN- & System-Manager | Deutsches Zentrum | DLR Oberpfaffenhofen f. Luft- und Raumfahrt e.V. | Institut f. Robotik | Postfach 1116 | Muenchner Strasse 20 82230 Wessling | 82234 Wessling Germany | | Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer@xxxxxx Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/ __________________________________________________________________
There are 10 types of people in this world, those who understand binary and those who don't.
--
Linux-cluster@xxxxxxxxxx
http://www.redhat.com/mailman/listinfo/linux-cluster
-- Linux-cluster@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/linux-cluster
-- _________________________________________________________________
Dr. Hansjoerg Maurer | LAN- & System-Manager | Deutsches Zentrum | DLR Oberpfaffenhofen f. Luft- und Raumfahrt e.V. | Institut f. Robotik | Postfach 1116 | Muenchner Strasse 20 82230 Wessling | 82234 Wessling Germany | | Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer@xxxxxx Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/ __________________________________________________________________
There are 10 types of people in this world, those who understand binary and those who don't.
--
Linux-cluster@xxxxxxxxxx
http://www.redhat.com/mailman/listinfo/linux-cluster
-- Linux-cluster@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/linux-cluster
-- Linux-cluster@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/linux-cluster
