On Tue, Nov 19, 2024 at 8:38 PM Rajmohan Ramamoorthy <ram.rajmohanr@xxxxxxxxx> wrote: > > CephFS subvolumes not inheriting ephemeral distributed pin configured on > the subvolumegroups. Subvolumes do not "inherit" the distributed ephemeral pin. What you should expect below is that the "csi" subvolumegroup will be fragmented and distributed across the ranks. Consequently, the subvolumes will also be distributed across ranks as part of the subtrees rooted at each fragment of the "csi" subvolumegroup (directory). > References: > > https://github.com/ceph/ceph/pull/35759 > https://github.com/ceph/ceph/pull/43896 > > # Environmet setup > > Ceph version: 18.2.4 and 19.2.0 Meaning you tested two versions? > CephFS with multiple MDS (2 active + 1 standby MDS) > RHEL 9.4 > Linux myserver 5.14.0-427.37.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep > 13 12:41:50 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux > > # Ceph FS configuration > ``` > ceph config set mds mds_export_ephemeral_distributed true > ceph fs set midline-a balance_automate false > ``` > # Subvolume group (csi) creation > ``` > ceph fs subvolumegroup create midline-a csi > ``` > # Subvolume group (csi) pinning (tried both methods) > ``` > ceph fs subvolumegroup pin midline-a csi distributed 1 > > setfattr -n ceph.dir.pin.distributed -v 1 setfattr -n > ceph.dir.pin.distributed -v 1 /mnt/cephfs/volumes/csi/ > ``` You ran setfattr and ` ceph fs subvolumegroup pin midline-a csi distributed 1` ? > # Subvolume creation > ``` > ceph fs subvolume create midline-a subvol1 csi > ceph fs subvolume create midline-a subvol2 csi > ``` > > # Get attribute (subvolume group) > ``` > getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi > > getfattr: Removing leading '/' from absolute path names > # file: mnt/cephfs/volumes/csi > ceph.dir.pin.distributed="1" > ``` > > # Get attribute (subvolume) - Subvolumes not inheriting the distributed pin > attribute > > ``` > getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol1 > > getfattr: Removing leading '/' from absolute path names > # file: mnt/cephfs/volumes/csi/subvol1 > ceph.dir.pin.distributed="0" > ``` > > ``` > getfattr -n ceph.dir.pin.distributed /mnt/cephfs/volumes/csi/subvol2 > > getfattr: Removing leading '/' from absolute path names > # file: mnt/cephfs/volumes/csi/subvol2 > ceph.dir.pin.distributed="0" > ``` these last two outputs are expected. > # MDS logs (subtree information) > ``` > ceph tell mds.abc123 get subtrees | jq '.[] | [.dir.path, .auth_first, > .export_pin, .export_pin_target]' > > [ > "~mds1", > 1, > -1, > -1 > ] > [ > "", > 0, > -1, > -1 > ] > [ > "/volumes/csi", > 1, > -1, > 1 > ] > ``` > > ``` > ceph tell mds.xyz123 get subtrees | jq '.[] | [.dir.path, .auth_first, > .export_pin, .export_pin_target]' > > [ > "/volumes/csi", > 1, > -1, > 1 > ] > [ > "", > 0, > -1, > -1 > ] > [ > "~mds0", > 0, > -1, > -1 > ] > [ > "/volumes/csi", > 0, > -1, > 0 > ] > ``` It looks like the "csi" directory was split but the output is incomplete. Please paste the full (anonymized) `get subtrees` output. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |