Re: Re:NFS re-export works for me

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Note that you can also use the UNFS3 user-mode NFS server rather than the nfs-kernel-server. With it, you don't need a custom FUSE, and you don't need to disable direct-IO. Probably the big downside, though, is that it doesn't support NFS file locking. 

Thanks,

Brent

On Mon, 9 Mar 2009, Raghavendra G wrote:


Hi,

starting glusterfs with direct-io disabled (glusterfs --disable-direct-io)
ensures working of NFS re-export of glusterfs.

regards,

On Fri, Apr 27, 2007 at 12:47 AM, gpvcs@xxxxxxxxxx <gpvcs@xxxxxxxxxx> wrote:


Hi Avati,

I first thought that I found the error when suddenly it was
working on my machine but I never managed to reproduce it.
Then I realized that in cluster mode, one of my machine was
running using the default fuse but it did it again on my
standalone machine.
One think for sure is that with default fuse (API 7.6)
NFS mount fail always. You have to use a latter fuse.

After many try I started to trace NFS and Fuse in kernel mode:

cd /proc/sys/sunrpc
echo 32767 > nfs_debug
echo 32767 > nfsd_debug

and did my test again:

exportfs:
localhost:/gfs/fs1 on /gfs/d610.nfsV3 type nfs
(rw,nfsvers=3,noac,hard,intr,tcp,lock,posix,addr=127.0.0.1)

cd /gfs/nfsV3
ls > 11


The trace between a successfull writing other NFS and error
change here:
Apr 26 16:21:06 fturi kernel: NFS call  write 6 @ 0
Apr 26 16:21:06 fturi kernel: nfsd_dispatch: vers 3 proc 7
Apr 26 16:21:06 fturi kernel: nfsd: WRITE(3)    24: 02010001
00000000 00000003 00000000 00000001 00000000 6 bytes at 0 stable
Apr 26 16:21:06 fturi kernel: nfsd: fh_verify(24: 02010001
00000000 00000003 00000000 00000001 00000000)




Apr 26 16:21:06 fturi kernel: # NO PERM 531 nfsd: write
complete err=-1
Apr 26 16:21:06 fturi kernel: NFS reply write: -1
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

The NO PERM message comes from this function in fuse
2.6.3/kernel/file.c

static int fuse_get_user_pages(struct fuse_req *req, const
char __user *buf,
                              unsigned nbytes, int write)
{
       unsigned long user_addr = (unsigned long) buf;
       unsigned offset = user_addr & ~PAGE_MASK;
       int npages;

       /* This doesn't work with nfsd */
       if (!current->mm)
               {printk(KERN_CRIT "# NO PERM
%d",__LINE__);return -EPERM;};

       nbytes = min(nbytes, (unsigned)
FUSE_MAX_PAGES_PER_REQ << PAGE_SHIFT);
       npages = (nbytes + offset + PAGE_SIZE - 1) >>
PAGE_SHIFT;
       npages = max(npages, 1);
       npages = min(npages, FUSE_MAX_PAGES_PER_REQ);
       down_read(&current->mm->mmap_sem);
       npages = get_user_pages(current, current->mm,
user_addr, npages, write,
                               0, req->pages, NULL);
       up_read(&current->mm->mmap_sem);
       if (npages < 0)
               return npages;

       req->num_pages = npages;
       req->page_offset = offset;
       return 0;
}


I Just added the dprintk to make sure it was there. There is
a comment about: /* doesnt work with nfsd */

So, its look like its fuse refusing (error=-1 = NOPERM) the
writting other NFS.
I'm puzzled by the fact it works under another linux. maybe
the fuse_get_user_pages  is not called in your case of the
structure "current->mm" is filled up in your case.

This is the trace collected by the debug mode of server.vol
and client.vol ( debug on top of the brick):

glusterfsd is empty
glusterfs contains:
[Apr 26 16:21:02] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:02] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:967/trace_open()]
trace:trace_open (*this=0x8395650, path=/11, flags=32769,
mode=0)
[Apr 26 16:21:04] [DEBUG/trace.c:88/trace_open_cbk()]
trace:trace_open_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *ctx=0x8396f58), *buf=0x8396c60 {st_dev=64768,
st_ino=
64356, st_mode=33206, st_nlink=1, st_uid=99, st_gid=99,
st_rdev=0, st_size=0, st_blksize=4096, st_blocks=8,
st_atime=[Apr 26 15:48:19], st_mtime=[Apr 26 15:54:32], st_cti
me=[Apr 26 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:1089/trace_release()]
trace:trace_release (*this=0x8395650, *ctx=0x8396f58)
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:386/trace_release_cbk()]
trace:trace_release_cbk (*this=0x8395650, op_ret=0, op_errno=95)
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8397098 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:1155/trace_getxattr()]
trace:trace_getxattr (*this=0x8395650, path=/,
name=system.posix_acl_access, size=0)
[Apr 26 16:28:31] [DEBUG/trace.c:513/trace_getxattr_cbk()]
trace:trace_getxattr_cbk (*this=0x8395650, op_ret=-1,
op_errno=95)


Cdt FT





---------- Initial Header -----------

From      : Anand Avati <avati@xxxxxxxxxxxxx>
To          : gpvcs@xxxxxxxxxx
Cc          : gluster-devel@xxxxxxxxxx
Date      : Wed, 18 Apr 2007 08:55:26 -0700
Subject : Re: Re:NFS re-export works for me

I was able to successfully mount a glusterfs export via NFS
too. I am
using slackware 10.2 and did nothing special apart from using
fuse-2.6.0. Maybe your problem could be because of the AIX
client.
please try from a GNU/Linux client and see if it helps.
Attaching the
log file by loading debug/trace will help get a better picture.

regards,
avati

On Wed, Apr 18, 2007 at 11:25:49AM -0400, Brent A Nelson wrote:

rw,fsid=10 are the export options.

2.6.15 is the kernel (it's a Ubuntu Dapper with its

kernel, probably

closer to 2.6.16).  Fuse 2.6.3 was added from an extra

repository.


I used no options at all on the Ubuntu Breezy NFS client

mount.  Looking

at the mtab, no special options are listed there, either,

so the options

are whatever is default for this platform.

Thanks,

Brent

On Wed, 18 Apr 2007, gpvcs@xxxxxxxxxx wrote:


Hi Brent,

That's a very good news for me. Could you please tell me
which linux kernel version are you using (uname -a) and
/etc/exports line and mount command you used ?
I'm not lucky with NFS and will very appreciate thoses

details.


Have a nice day, Francois
---------- Initial Header -----------

From      : gluster-devel-bounces+gpvcs=tiscali.fr@xxxxxxxxxx
To          : gluster-devel@xxxxxxxxxx
Cc          :
Date      : Wed, 18 Apr 2007 00:51:51 -0400 (EDT)
Subject : NFS re-export works for me

I tried a quick NFS re-export to another Linux box (with the
fsid= option
and fuse 2.6.3 on the NFS server, of course).  I didn't test
it much, but
copies to and from the NFS filesystem worked just fine.  I
think writes
were very slow, but they didn't give me any trouble.

No special mount options (such as NFS version or transport)
were needed.

Thanks,

Brent


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel



------------------------ ALICE C'EST ENCORE MIEUX AVEC

CANAL+ LE BOUQUET !

---------------
Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE

BOUQUET, en cliquant

ici http://alicebox.fr
Soumis à conditions.






_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel



--
ultimate_answer_t
deep_thought (void)
{
 sleep (years2secs (7500000));
 return 42;
}


------------------------ ALICE C'EST ENCORE MIEUX AVEC CANAL+ LE BOUQUET !
---------------
Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE BOUQUET, en cliquant
ici http://alicebox.fr
Soumis à conditions.




_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel





--
Raghavendra G
[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux