I'm using CephFS, on CentOS 7. We're currently migrating away from using a catch-all cephx key to mount the filesystem (with the kernel module), to a much more restricted key. In my tests, I've come across an issue, extracting a tar archive with a mount using the restricted key routinely cannot create files or directories in recently created directories. I need to keep running a CentOS based kernel on the clients because of some restrictions from other software. Below looks like a race condition to me, although I am not versed well enough in Ceph or the inner workings of the kernel to know for sure. # tar xf gmp-6.1.2.tar.lz -C /homes/mozes/tmp/ tar: gmp-6.1.2/mpn/x86_64/mulx/adx/addmul_1.asm: Cannot open: Permission denied tar: Exiting with failure status due to previous errors This gets worse with tracing turned on in the kernel. (echo module ceph +p > /sys/kernel/debug/dynamic_debug/control) # tar xf gmp-6.1.2.tar.lz -C /homes/mozes/tmp/ tar: gmp-6.1.2/mpn/x86_64/mulx/adx: Cannot mkdir: Permission denied tar: gmp-6.1.2/mpn/x86_64/mulx/aorsmul_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/mulx/mul_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/mulx/adx: Cannot mkdir: Permission denied tar: gmp-6.1.2/mpn/x86_64/mulx/adx/addmul_1.asm: Cannot open: No such file or directory tar: gmp-6.1.2/mpn/x86_64/coreinhm/popcount.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/gmp-mparam.h: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/aorrlsh_n.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/aorsmul_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/sec_tabselect.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/redc_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreinhm/hamdist.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/copyd.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/copyi.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/popcount.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/gmp-mparam.h: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/gcd_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/nano/dive_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/redc_1.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/mullo_basecase.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/fat_entry.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/mod_1.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/gmp-mparam.h: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/redc_2.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/sqr_basecase.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/fat.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/fat/mul_basecase.c: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreihwl/mullo_basecase.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreihwl/mul_basecase.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreihwl/aorsmul_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreihwl/mul_1.asm: Cannot open: Permission denied tar: gmp-6.1.2/mpn/x86_64/coreihwl/redc_1.asm: Cannot open: Permission denied While extracting the same tar file using an unrestricted key works correctly. I've got some kernel traces to share if anyone is interested. https://people.beocat.ksu.edu/~mozes/ceph-20170425/ # uname -a Linux eunomia 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux We're currently running Ceph Jewel (10.2.5). We're looking to update soon, but we wanted a clean backup of everything in CephFS first. The new restricted key has these permissions: caps mds = "allow r, allow rw path=/homes, allow rw path=/bulk, allow rw path=/beocat" caps mon = "allow r" caps osd = "allow rw pool=scratch, allow rw pool=bulk, allow rw pool=homes" While the unrestricted key has these permissions: caps mds = "allow" caps mon = "allow *" caps osd = "allow *" I would appreciate any insights anyone might have. Thanks, Adam _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |