On Tue, Apr 19, 2022 at 08:51:50PM +0000, Ryan Taylor wrote: > Thanks for the pointers! It does look like https://tracker.ceph.com/issues/55090 > and I am not surprised Dan and I are hitting the same issue... Just a wild guess (already asked this on the tracker): Is it possible that you're using different credentials/keys so that the credentials used for mounting the subdir are not allowed to access the volume base directory? Would it be possible to get more details on the two mount commands being used? Cheers, -- Luís > > > I am using the latest available Almalinux 8, 4.18.0-348.20.1.el8_5.x86_64 > > Installing kernel-debuginfo-common-x86_64 > I see in /usr/src/debug/kernel-4.18.0-348.2.1.el8_5/linux-4.18.0-348.2.1.el8_5.x86_64/fs/ceph/quota.c > for example: > > static inline bool ceph_has_realms_with_quotas(struct inode *inode) > { > struct super_block *sb = inode->i_sb; > struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(sb); > struct inode *root = d_inode(sb->s_root); > > if (atomic64_read(&mdsc->quotarealms_count) > 0) > return true; > /* if root is the real CephFS root, we don't have quota realms */ > if (root && ceph_ino(root) == CEPH_INO_ROOT) > return false; > /* otherwise, we can't know for sure */ > return true; > } > > So this EL8.5 kernel already has at least some of the patches from https://lore.kernel.org/all/20190301175752.17808-1-lhenriques@xxxxxxxx/T/#u > for https://tracker.ceph.com/issues/38482 > That does not mention a specific commit, just says "Merged into 5.2-rc1." > > So it seems https://tracker.ceph.com/issues/55090 is either a new issue or a regression of the previous issue. > > Thanks, > -rt > > Ryan Taylor > Research Computing Specialist > Research Computing Services, University Systems > University of Victoria > > ________________________________________ > From: Hendrik Peyerl <hpeyerl@xxxxxxxxxxxx> > Sent: April 19, 2022 6:05 AM > To: Ramana Venkatesh Raja > Cc: Ryan Taylor; ceph-users@xxxxxxx > Subject: Re: df shows wrong size of cephfs share when a subdirectory is mounted > > Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information. > > > I did hit this issue aswell: https://tracker.ceph.com/issues/38482 > > you will need a kernel >= 5.2 that can handle the quotas on subdirectories. > > > > On 19. Apr 2022, at 14:47, Ramana Venkatesh Raja <rraja@xxxxxxxxxx> wrote: > > > > On Sat, Apr 16, 2022 at 10:15 PM Ramana Venkatesh Raja <rraja@xxxxxxxxxx> wrote: > >> > >> On Thu, Apr 14, 2022 at 8:07 PM Ryan Taylor <rptaylor@xxxxxxx> wrote: > >>> > >>> Hello, > >>> > >>> > >>> I am using cephfs via Openstack Manila (Ussuri I think). > >>> > >>> The cephfs cluster is v14.2.22 and my client has kernel 4.18.0-348.20.1.el8_5.x86_64 > >>> > >>> > >>> I have a Manila share > >>> > >>> /volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2 > >>> > >>> > >>> that is 5000 GB in size. When I mount it the size is reported correctly: > >>> > >>> > >>> # df -h /cephfs > >>> Filesystem Size Used Avail Use% Mounted on > >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2 4.9T 278G 4.7T 6% /cephfs > >>> > >>> > >>> However when I mount a subpath /test1 of my share, then both the size and usage are showing the size of the whole cephfs filesystem rather than my private share. > >>> > >>> > >>> # df -h /cephfs > >>> Filesystem Size Used Avail Use% Mounted on > >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1 4.0P 277T 3.7P 7% /cephfs > >>> > >> > >> What are the capabilities of the ceph client user ID that you used to > >> mount "/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1" ? > >> Maybe you're hitting this limitation in > >> https://docs.ceph.com/en/latest/cephfs/quota/#limitations , > >> "Quotas must be configured carefully when used with path-based mount > >> restrictions. The client needs to have access to the directory inode > >> on which quotas are configured in order to enforce them. If the client > >> has restricted access to a specific path (e.g., /home/user) based on > >> the MDS capability, and a quota is configured on an ancestor directory > >> they do not have access to (e.g., /home), the client will not enforce > >> it. When using path-based access restrictions be sure to configure the > >> quota on the directory the client is restricted too (e.g., /home/user) > >> or something nested beneath it. " > >> > > > > Hi Ryan, > > > > I think you maybe actually hitting this > > https://tracker.ceph.com/issues/55090 . Are you facing this issue with > > the FUSE client? > > > > -Ramana > > > >>> > >>> I tried setting the ceph.quota.max_bytes xattr on a subdirectory but it did not help. > >>> > >> > >> You can't set quota xattr if your ceph client user ID doesn't have 'p' > >> flag in its MDS capabilities, > >> https://docs.ceph.com/en/latest/cephfs/client-auth/#layout-and-quota-restriction-the-p-flag > >> . > >> > >> -Ramana > >> > >>> I'm not sure if the issue is in cephfs or Manila, but what would be required to get the right size and usage stats to be reported by df when a subpath of a share is mounted? > >>> > >>> > >>> Thanks! > >>> > >>> -rt > >>> > >>> > >>> Ryan Taylor > >>> Research Computing Specialist > >>> Research Computing Services, University Systems > >>> University of Victoria > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@xxxxxxx > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>> > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |