Good day all,
I?ve built a new glusterfs volume using 20 nodes of one of my clusters, each
with a 2TB SATA disk, formatted with ext3 (system is centos 5.2, x86_64).
The volume is such:
Volume Name: gfsvol1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 10 x 2 = 20
Transport-type: rdma
Bricks:
Brick1: node002:/gfs
Brick2: node003:/gfs
Brick3: node004:/gfs
Brick4: node005:/gfs
Brick5: node006:/gfs
Brick6: node007:/gfs
Brick7: node008:/gfs
Brick8: node009:/gfs
Brick9: node010:/gfs
Brick10: node011:/gfs
Brick11: node012:/gfs
Brick12: node013:/gfs
Brick13: node014:/gfs
Brick14: node015:/gfs
Brick15: node016:/gfs
Brick16: node017:/gfs
Brick17: node019:/gfs
Brick18: node020:/gfs
Brick19: node021:/gfs
Brick20: node022:/gfs
The volume mounts on a client:
[root at moldyn ~]# mount -t glusterfs -o transport=rdma node002:/gfsvol1
/gfsvol1
[root at moldyn ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
glusterfs#node002:/gfsvol1
19228583424 2001664 18249825792 1% /gfsvol1
I get this error on a copy into the gluster volume:
[mgx at moldyn ~]$ cp -R pmemd/ /gfsvol1/mgx/pmemd
cp: writing `/gfsvol1/mgx/pmemd/fmdrun.out': Transport endpoint is not
connected
cp: closing `/gfsvol1/mgx/pmemd/fmdrun.out': Resource temporarily
unavailable
it did copy files, just failed on that one:
/gfsvol1/mgx/pmemd/:
total 4357376
-rw-rw-r-- 1 root root 514711552 Oct 27 13:02 fmdrun.out
-rw-rw-r-- 1 mgx mgx 4754 Oct 27 13:01 fmdrun.out.new
-rw-rw-r-- 1 mgx mgx 851832631 Oct 27 13:03 fmdrun.out_run1
-rw-rw-r-- 1 mgx mgx 81 Oct 27 13:01 mdinfo
-rw------- 1 mgx mgx 803 Oct 27 13:02 md.out
-rw-rw-r-- 1 mgx mgx 342 Oct 27 13:03 md.sub
-rw-rw-r-- 1 mgx mgx 1567835776 Oct 27 13:02 new.mdcrd
-rw-rw-r-- 1 mgx mgx 1522326100 Oct 27 13:01 new.mdcrd_run1
-rw-rw-r-- 1 mgx mgx 155957 Oct 27 13:02 new.rst
-rw-rw-r-- 1 mgx mgx 155957 Oct 27 13:01 old.rst
drwxrwxr-x 3 mgx mgx 40960 Oct 27 13:01 rbenew
-rw-rw-r-- 1 mgx mgx 1008 Oct 27 13:03 vp_mdrun.in
-rw-rw-r-- 1 mgx mgx 26190 Oct 27 13:01 vp.prmtop
-rw-rw-r-- 1 mgx mgx 348092 Oct 27 13:01 vp_wat.prmtop
pmemd/:
total 4711216
-rw-rw-r-- 1 mgx mgx 876818259 Apr 2 2010 fmdrun.out
-rw-rw-r-- 1 mgx mgx 4754 Mar 19 2010 fmdrun.out.new
-rw-rw-r-- 1 mgx mgx 851832631 Mar 6 2010 fmdrun.out_run1
-rw-rw-r-- 1 mgx mgx 81 Apr 2 2010 mdinfo
-rw------- 1 mgx mgx 803 Apr 2 2010 md.out
-rw-rw-r-- 1 mgx mgx 342 Mar 31 2010 md.sub
-rw-rw-r-- 1 mgx mgx 1567835776 Apr 2 2010 new.mdcrd
-rw-rw-r-- 1 mgx mgx 1522326100 Mar 6 2010 new.mdcrd_run1
-rw-rw-r-- 1 mgx mgx 155957 Apr 2 2010 new.rst
-rw-rw-r-- 1 mgx mgx 155957 Mar 9 2010 old.rst
drwxrwxr-x 3 mgx mgx 4096 Mar 31 2010 rbenew
-rw-rw-r-- 1 mgx mgx 1008 Mar 2 2010 vp_mdrun.in
-rw-rw-r-- 1 mgx mgx 26190 Mar 2 2010 vp.prmtop
-rw-rw-r-- 1 mgx mgx 348092 Mar 2 2010 vp_wat.prmtop
The fmdrun.out file is truncated and incorrect ownership.
The volume was created following the 3.1 docu.
Where is the problem at? Gluster? IB? my ib is ofed 1.3.1 and I have SDR
mellenox HCA?s.
--- michael
