Re: cephfs : write error: Operation not permitted

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



There are two bugs that can cause these application tags to be
missing, one of them is fixed (but old pools aren't fixed
automatically), the other is https://tracker.ceph.com/issues/43061
which happens if you create the cephfs pools manually.

You can fix the pools like this:

ceph osd pool application set  <pool> cephfs <data/metadata> <cephfsname>

To work with "ceph fs authorize"

We automatically runs this on croit on startup on all cephfs pools to
make the permissions work properly for our users.


Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Fri, Jan 24, 2020 at 2:11 PM Yoann Moulin <yoann.moulin@xxxxxxx> wrote:
>
> Le 23.01.20 à 15:51, Ilya Dryomov a écrit :
> > On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@xxxxxxx> wrote:
> >>
> >> Hello,
> >>
> >> On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can
> >> create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool.
> >>
> >>> $ ceph -s
> >>>    cluster:
> >>>      id:     fded5bb5-62c5-4a88-b62c-0986d7c7ac09
> >>>      health: HEALTH_OK
> >>>
> >>>    services:
> >>>      mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
> >>>      mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042
> >>>      mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
> >>>      osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
> >>>      rgw: 1 daemon active (iccluster043.rgw0)
> >>>
> >>>    data:
> >>>      pools:   9 pools, 568 pgs
> >>>      objects: 800 objects, 225 KiB
> >>>      usage:   24 GiB used, 87 TiB / 87 TiB avail
> >>>      pgs:     568 active+clean
> >>
> >> The 2 cephfs pools:
> >>
> >>> $ ceph osd pool ls detail | grep cephfs
> >>> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs
> >>> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
> >>
> >> The status of the cephfs filesystem:
> >>
> >>> $ ceph fs status
> >>> cephfs - 1 clients
> >>> ======
> >>> +------+--------+--------------+---------------+-------+-------+
> >>> | Rank | State  |     MDS      |    Activity   |  dns  |  inos |
> >>> +------+--------+--------------+---------------+-------+-------+
> >>> |  0   | active | iccluster043 | Reqs:    0 /s |   34  |   18  |
> >>> |  1   | active | iccluster041 | Reqs:    0 /s |   12  |   16  |
> >>> |  2   | active | iccluster042 | Reqs:    0 /s |   10  |   13  |
> >>> +------+--------+--------------+---------------+-------+-------+
> >>> +-----------------+----------+-------+-------+
> >>> |       Pool      |   type   |  used | avail |
> >>> +-----------------+----------+-------+-------+
> >>> | cephfs_metadata | metadata | 4608k | 27.6T |
> >>> |   cephfs_data   |   data   |    0  | 27.6T |
> >>> +-----------------+----------+-------+-------+
> >>> +-------------+
> >>> | Standby MDS |
> >>> +-------------+
> >>> +-------------+
> >>> MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
> >>
> >>
> >>> # mkdir folder
> >>> # echo "foo" > bar
> >>> -bash: echo: write error: Operation not permitted
> >>> # ls -al
> >>> total 4
> >>> drwxrwxrwx  1 root root    2 Jan 22 07:30 .
> >>> drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
> >>> -rw-r--r--  1 root root    0 Jan 22 07:30 bar
> >>> drwxrwxrwx  1 root root    1 Jan 21 16:49 folder
> >>
> >>> # df -hT .
> >>> Filesystem                                     Type  Size  Used Avail Use% Mounted on
> >>> 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph   28T     0   28T   0% /cephfs
> >>
> >> I try 2 client config :
> >>
> >>> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> >>> [snip]
> >>> $ ceph auth  get client.fsadmin
> >>> exported keyring for client.fsadmin
> >>> [client.fsadmin]
> >>>        key = [snip]
> >>>        caps mds = "allow rw"
> >>>        caps mon = "allow r"
> >>>        caps osd = "allow rw tag cephfs data=cephfs"
> >
> > For "allow rw tag cephfs data=cephfs" to work, having your pools tagged
> > with cephfs is not enough.  There should be a key-value pair in the tag
> > metadata with the name of the filesystem as well.  What is the output
> > of:
> >
> >    ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name
> > | startswith("cephfs")) | .pool_name, .application_metadata'
>
> But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the
> latest release of the ceph-ansible playbook.
>
> dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name,
> .application_metadata'
> "cephfs_data"
> {
>    "cephfs": {}
> }
> "cephfs_metadata"
> {
>    "cephfs": {}
> }
>
> Best,
>
> --
> Yoann Moulin
> EPFL IC-IT
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux