Thanks for the pointers! It does look like https://tracker.ceph.com/issues/55090 and I am not surprised Dan and I are hitting the same issue... I am using the latest available Almalinux 8, 4.18.0-348.20.1.el8_5.x86_64 Installing kernel-debuginfo-common-x86_64 I see in /usr/src/debug/kernel-4.18.0-348.2.1.el8_5/linux-4.18.0-348.2.1.el8_5.x86_64/fs/ceph/quota.c for example: static inline bool ceph_has_realms_with_quotas(struct inode *inode) { struct super_block *sb = inode->i_sb; struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(sb); struct inode *root = d_inode(sb->s_root); if (atomic64_read(&mdsc->quotarealms_count) > 0) return true; /* if root is the real CephFS root, we don't have quota realms */ if (root && ceph_ino(root) == CEPH_INO_ROOT) return false; /* otherwise, we can't know for sure */ return true; } So this EL8.5 kernel already has at least some of the patches from https://lore.kernel.org/all/20190301175752.17808-1-lhenriques@xxxxxxxx/T/#u for https://tracker.ceph.com/issues/38482 That does not mention a specific commit, just says "Merged into 5.2-rc1." So it seems https://tracker.ceph.com/issues/55090 is either a new issue or a regression of the previous issue. Thanks, -rt Ryan Taylor Research Computing Specialist Research Computing Services, University Systems University of Victoria ________________________________________ From: Hendrik Peyerl <hpeyerl@xxxxxxxxxxxx> Sent: April 19, 2022 6:05 AM To: Ramana Venkatesh Raja Cc: Ryan Taylor; ceph-users@xxxxxxx Subject: Re: df shows wrong size of cephfs share when a subdirectory is mounted Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information. I did hit this issue aswell: https://tracker.ceph.com/issues/38482 you will need a kernel >= 5.2 that can handle the quotas on subdirectories. > On 19. Apr 2022, at 14:47, Ramana Venkatesh Raja <rraja@xxxxxxxxxx> wrote: > > On Sat, Apr 16, 2022 at 10:15 PM Ramana Venkatesh Raja <rraja@xxxxxxxxxx> wrote: >> >> On Thu, Apr 14, 2022 at 8:07 PM Ryan Taylor <rptaylor@xxxxxxx> wrote: >>> >>> Hello, >>> >>> >>> I am using cephfs via Openstack Manila (Ussuri I think). >>> >>> The cephfs cluster is v14.2.22 and my client has kernel 4.18.0-348.20.1.el8_5.x86_64 >>> >>> >>> I have a Manila share >>> >>> /volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2 >>> >>> >>> that is 5000 GB in size. When I mount it the size is reported correctly: >>> >>> >>> # df -h /cephfs >>> Filesystem Size Used Avail Use% Mounted on >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2 4.9T 278G 4.7T 6% /cephfs >>> >>> >>> However when I mount a subpath /test1 of my share, then both the size and usage are showing the size of the whole cephfs filesystem rather than my private share. >>> >>> >>> # df -h /cephfs >>> Filesystem Size Used Avail Use% Mounted on >>> 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789:/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1 4.0P 277T 3.7P 7% /cephfs >>> >> >> What are the capabilities of the ceph client user ID that you used to >> mount "/volumes/_nogroup/55e46a89-31ff-4878-9e2a-81b4226c3cb2/test1" ? >> Maybe you're hitting this limitation in >> https://docs.ceph.com/en/latest/cephfs/quota/#limitations , >> "Quotas must be configured carefully when used with path-based mount >> restrictions. The client needs to have access to the directory inode >> on which quotas are configured in order to enforce them. If the client >> has restricted access to a specific path (e.g., /home/user) based on >> the MDS capability, and a quota is configured on an ancestor directory >> they do not have access to (e.g., /home), the client will not enforce >> it. When using path-based access restrictions be sure to configure the >> quota on the directory the client is restricted too (e.g., /home/user) >> or something nested beneath it. " >> > > Hi Ryan, > > I think you maybe actually hitting this > https://tracker.ceph.com/issues/55090 . Are you facing this issue with > the FUSE client? > > -Ramana > >>> >>> I tried setting the ceph.quota.max_bytes xattr on a subdirectory but it did not help. >>> >> >> You can't set quota xattr if your ceph client user ID doesn't have 'p' >> flag in its MDS capabilities, >> https://docs.ceph.com/en/latest/cephfs/client-auth/#layout-and-quota-restriction-the-p-flag >> . >> >> -Ramana >> >>> I'm not sure if the issue is in cephfs or Manila, but what would be required to get the right size and usage stats to be reported by df when a subpath of a share is mounted? >>> >>> >>> Thanks! >>> >>> -rt >>> >>> >>> Ryan Taylor >>> Research Computing Specialist >>> Research Computing Services, University Systems >>> University of Victoria >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |