Re: Intermittent permission denied using kernel client with mds path cap

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I filled http://tracker.ceph.com/issues/17858 recently, I am seeing this problem on 10.2.3 ceph-fuse, but maybe kernel client is affected too.

It is easy to replicate, just do deep "mkdir -p", e.g. "mkdir -p 1/2/3/4/5/6/7/8/9/0/1/2/3/4/5/6/7/8/9"

On 16-11-11 10:46, Dan van der Ster wrote:
Hi Goncalo,

Thank you for those links. It appears that that fix was already in the
10.2.3 mds, which we are running. I've just upgraded the mds's to the
current jewel gitbuilder (10.2.3-358.g427f357.x86_64) and the problem
is still there.

(BTW, in testing this I've been toggling the mds caps between 'allow
rw' and 'allow r, allow rw path=/k8s'. During this I've noticed that
the new caps are only applied after I remount.)

Cheers, Dan


On Fri, Nov 11, 2016 at 1:29 AM, Goncalo Borges
<goncalo.borges@xxxxxxxxxxxxx> wrote:
Hi Dan,,,

I know there are path restriction issues in the kernel client. See the discussion here.

http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2016-June/010656.html

http://tracker.ceph.com/issues/16358

Cheers
Goncalo

________________________________________
From: ceph-users [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Dan van der Ster [dan@xxxxxxxxxxxxxx]
Sent: 11 November 2016 02:41
To: ceph-users; Yan, Zheng
Subject:  Intermittent permission denied using kernel client        with mds path cap

Hi all, Hi Zheng,

We're seeing a strange issue with the kernel cephfs clients, combined
with a path restricted mds cap. It seems that files/dirs are
intermittently not created due to permission denied.

For example, when I untar a kernel into cephfs, we see ~1/1000 files
failed to open/mkdir.
Client caps are in the PS [1].
We've tried kernels 3.10.0-493.el7, 4.8.6, and 4.9-rc4 -- all have the
same intermittent behaviour. We could *not* reproduce the issue with
ceph-fuse 10.2.3.

The cluster is running 10.2.3.

Now, if we remove the path restricted cap -- i.e. use mds 'allow rw'
-- then we have no more errors.

So it seems there is a race in the path restriction cap code.
We grabbed an mds log, and noticed that it seems that a file is opened
twice, then the second open fails with 'already xlocked'. A full log
for one such file is here: http://pastebin.com/raw/YyULfjND

Is anyone successfully using path caps with kernel clients? Maybe this
is a new bug?

Cheers, Dan


[1]
[client.k8s]
    key: xxx==
    caps: [mds] allow r, allow rw path=/k8s
    caps: [mon] allow r
    caps: [osd] allow rw pool=cephfs_data_k8s
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux