I am new to cluster and gfs so my apologies if I missed something obvious.
I am having problems using gfs2 filesystems where writing processes hang forever waiting for i/o. It does not seem to be locking (or even cluster) related since its reproducible on a local gfs2 filesystem with 'lock_nolock' as locking protocol.
I can create and mount the gfs2 file system without any problems. Creating and writing to files works as well, but when I write larger amounts of data the process gets stuck waiting for i/o. It doesn't matter whether I amd using lvm or not thre results are the same. The hardware is working and I have no problem writing to an ext3 filesystem on the same device.
The problem can be reproduced by doing the following:
-------------------------------------------------------------------------------------------------
[root@indigo ~]# mkfs.gfs2 -p lock_nolock /dev/cciss/c0d0p3
This will destroy any data on /dev/cciss/c0d0p3.
It appears to contain a gfs2 filesystem.
Are you sure you want to proceed? [y/n] y
Device: /dev/cciss/c0d0p3
Blocksize: 4096
Device Size 644.44 GB (168935523 blocks)
Filesystem Size: 644.44 GB (168935523 blocks)
Journals: 1
Resource Groups: 2578
Locking Protocol: "lock_nolock"
Lock Table: ""
[root@indigo ~]# mount -o locktable=xyz /dev/cciss/c0d0p3 /export
[root@indigo ~]# mount|grep /export
/dev/cciss/c0d0p3 on /export type gfs2 (rw,locktable=xyz,localflocks,localcaching)
[root@indigo ~]# dd if=/dev/zero of=/export/8GB bs=1024k count=8000
## A partial file is created, but the process never returns, it is forever waiting for i/o:
[root@indigo ~]# ls -l /export/;ps auxww|grep 8GB
total 1995152
-rw-r--r-- 1 root root 2039009280 Jun 1 07:12 8GB
root 3857 42.8 0.0 4880 1584 pts/0 D 07:12 0:07 dd if /dev/zero of /export/8GB bs 1024k count 8000
----------------------------------------------------------------------------------------------------
It's still possible to create new files to the file system but these processes are also waiting for i/o after a while.
After the mounting there is nothing else appearing in dmesg or messages log. Since the processes are uninterruptible we are forced to reboot the machine in order to resolve the lockup.
Basic info on the system setup:
HP dl380 G5 2 xeon cpu with dual cores each, 12 GB RAM
local raid controller is p400 but we also reproduced the behaviour on the san disk (Emulex card)
OS is RH 5 (32 bit), output of uname is:
Linux indigo 2.6.18-8.1.4.el5xen #1 SMP Fri May 4 22:42:50 EDT 2007 i686 i686 i386 GNU/Linux
The same problem could also be reproduced on RH 4 and Ubuntu 7.04
Thanks in advance for your help!
/jmc
This will destroy any data on /dev/cciss/c0d0p3.
It appears to contain a gfs2 filesystem.
Are you sure you want to proceed? [y/n] y
Device: /dev/cciss/c0d0p3
Blocksize: 4096
Device Size 644.44 GB (168935523 blocks)
Filesystem Size: 644.44 GB (168935523 blocks)
Journals: 1
Resource Groups: 2578
Locking Protocol: "lock_nolock"
Lock Table: ""
[root@indigo ~]# mount -o locktable=xyz /dev/cciss/c0d0p3 /export
[root@indigo ~]# mount|grep /export
/dev/cciss/c0d0p3 on /export type gfs2 (rw,locktable=xyz,localflocks,localcaching)
[root@indigo ~]# dd if=/dev/zero of=/export/8GB bs=1024k count=8000
## A partial file is created, but the process never returns, it is forever waiting for i/o:
[root@indigo ~]# ls -l /export/;ps auxww|grep 8GB
total 1995152
-rw-r--r-- 1 root root 2039009280 Jun 1 07:12 8GB
root 3857 42.8 0.0 4880 1584 pts/0 D 07:12 0:07 dd if /dev/zero of /export/8GB bs 1024k count 8000
----------------------------------------------------------------------------------------------------
It's still possible to create new files to the file system but these processes are also waiting for i/o after a while.
After the mounting there is nothing else appearing in dmesg or messages log. Since the processes are uninterruptible we are forced to reboot the machine in order to resolve the lockup.
Basic info on the system setup:
HP dl380 G5 2 xeon cpu with dual cores each, 12 GB RAM
local raid controller is p400 but we also reproduced the behaviour on the san disk (Emulex card)
OS is RH 5 (32 bit), output of uname is:
Linux indigo 2.6.18-8.1.4.el5xen #1 SMP Fri May 4 22:42:50 EDT 2007 i686 i686 i386 GNU/Linux
The same problem could also be reproduced on RH 4 and Ubuntu 7.04
Thanks in advance for your help!
/jmc
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
