There are two bugs that can cause these application tags to be missing, one of them is fixed (but old pools aren't fixed automatically), the other is https://tracker.ceph.com/issues/43061 which happens if you create the cephfs pools manually. You can fix the pools like this: ceph osd pool application set <pool> cephfs <data/metadata> <cephfsname> To work with "ceph fs authorize" We automatically runs this on croit on startup on all cephfs pools to make the permissions work properly for our users. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Jan 24, 2020 at 2:11 PM Yoann Moulin <yoann.moulin@xxxxxxx> wrote: > > Le 23.01.20 à 15:51, Ilya Dryomov a écrit : > > On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin <yoann.moulin@xxxxxxx> wrote: > >> > >> Hello, > >> > >> On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook stable-4.0, I have an issue with cephfs. I can create a folder, I can > >> create empty files, but cannot write data on like I'm not allowed to write to the cephfs_data pool. > >> > >>> $ ceph -s > >>> cluster: > >>> id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09 > >>> health: HEALTH_OK > >>> > >>> services: > >>> mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h) > >>> mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042 > >>> mds: cephfs:3 {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active} > >>> osd: 24 osds: 24 up (since 22h), 24 in (since 22h) > >>> rgw: 1 daemon active (iccluster043.rgw0) > >>> > >>> data: > >>> pools: 9 pools, 568 pgs > >>> objects: 800 objects, 225 KiB > >>> usage: 24 GiB used, 87 TiB / 87 TiB avail > >>> pgs: 568 active+clean > >> > >> The 2 cephfs pools: > >> > >>> $ ceph osd pool ls detail | grep cephfs > >>> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs > >>> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs > >> > >> The status of the cephfs filesystem: > >> > >>> $ ceph fs status > >>> cephfs - 1 clients > >>> ====== > >>> +------+--------+--------------+---------------+-------+-------+ > >>> | Rank | State | MDS | Activity | dns | inos | > >>> +------+--------+--------------+---------------+-------+-------+ > >>> | 0 | active | iccluster043 | Reqs: 0 /s | 34 | 18 | > >>> | 1 | active | iccluster041 | Reqs: 0 /s | 12 | 16 | > >>> | 2 | active | iccluster042 | Reqs: 0 /s | 10 | 13 | > >>> +------+--------+--------------+---------------+-------+-------+ > >>> +-----------------+----------+-------+-------+ > >>> | Pool | type | used | avail | > >>> +-----------------+----------+-------+-------+ > >>> | cephfs_metadata | metadata | 4608k | 27.6T | > >>> | cephfs_data | data | 0 | 27.6T | > >>> +-----------------+----------+-------+-------+ > >>> +-------------+ > >>> | Standby MDS | > >>> +-------------+ > >>> +-------------+ > >>> MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) > >> > >> > >>> # mkdir folder > >>> # echo "foo" > bar > >>> -bash: echo: write error: Operation not permitted > >>> # ls -al > >>> total 4 > >>> drwxrwxrwx 1 root root 2 Jan 22 07:30 . > >>> drwxr-xr-x 28 root root 4096 Jan 21 09:25 .. > >>> -rw-r--r-- 1 root root 0 Jan 22 07:30 bar > >>> drwxrwxrwx 1 root root 1 Jan 21 16:49 folder > >> > >>> # df -hT . > >>> Filesystem Type Size Used Avail Use% Mounted on > >>> 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph 28T 0 28T 0% /cephfs > >> > >> I try 2 client config : > >> > >>> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw > >>> [snip] > >>> $ ceph auth get client.fsadmin > >>> exported keyring for client.fsadmin > >>> [client.fsadmin] > >>> key = [snip] > >>> caps mds = "allow rw" > >>> caps mon = "allow r" > >>> caps osd = "allow rw tag cephfs data=cephfs" > > > > For "allow rw tag cephfs data=cephfs" to work, having your pools tagged > > with cephfs is not enough. There should be a key-value pair in the tag > > metadata with the name of the filesystem as well. What is the output > > of: > > > > ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name > > | startswith("cephfs")) | .pool_name, .application_metadata' > > But it's a fresh install, pools have been created with the ceph-ansible playbook and it used to work before, maybe something is missing in the > latest release of the ceph-ansible playbook. > > dslab2020@icitsrv5:~$ ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, > .application_metadata' > "cephfs_data" > { > "cephfs": {} > } > "cephfs_metadata" > { > "cephfs": {} > } > > Best, > > -- > Yoann Moulin > EPFL IC-IT > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |