On Mon, Jan 24, 2022 at 9:26 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > On Fri, 2022-01-21 at 17:39 -0500, Ramana Venkatesh Raja wrote: > > On Fri, Jan 21, 2022 at 4:01 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > > > > > On Fri, 2022-01-21 at 15:44 -0500, Ramana Venkatesh Raja wrote: > > > > Hi, > > > > > > > > The mgr/nfs module manages nfs-ganesha services including their > > > > exports [1]. It has interfaces that allow creating ganesha exports for > > > > CephFS using `ceph nfs export create cephfs` or `ceph nfs export > > > > apply` command, and updating exports using `ceph nfs export apply` > > > > command. When creating an export for CephFS the mgr/nfs module > > > > creates a FSAL ceph user with path restricted MDS caps based on the > > > > requested export or clients' access type and the CephFS path to be > > > > exported [2]. The export's libcephfs client ends up with read-only or > > > > read-write MDS caps to the path being exported. > > > > > > > > If a ganesha export's or clients' access type is updated, then it is > > > > expected that the ganesha server enforces the access type change > > > > dynamically i.e., enforces the change after a SIGHUP or DBus signal > > > > and not require a ganesha server restart. > > > > > > > > > > That's a pretty big expectation. There are all sorts of settings that > > > are not generally changeable on the fly and that require some sort of > > > server restart. > > > > > > > NFS-ganesha already supports dynamic updates of its exports' access > > type and export clients' access types, > > https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-export-config.rst#configuration-reload > > . The issue is that the mgr/nfs module does not allow NFS-ganesha to > > do this for CephFS exports it currently creates. > > > > > > > > Why are restarts such a problem? Are you just concerned about having to > > > impose a grace period? > > > > > > > I don't think restarting is a problem. If the NFS-ganesha server > > already supports dynamic updates of the export's access type or > > clients' access type, then it would be great if the underlying Ceph > > FSAL can also be configured to support this feature. For example, > > OpenStack manila's CephFS driver creates ganesha exports for CephFS > > subdirectories. It always creates FSAL ceph user with path restricted > > 'read-write' MDS caps. > > > > Let me summarize this to make sure I understand: > > You're creating a cephx principal for each export with path restricted > caps as documented here: > > https://docs.ceph.com/en/latest/cephfs/client-auth/ > > ...based on whether the export is RO or RW. When you later go to change > the export from RO to RW, then you change the caps at the same time. The > result though is that the new caps don't "take" until you restart > ganesha. > > Is that correct? > Yes. > FSAL_CEPH doesn't care much about the RO or RW export modes. That's all > enforced in common NFS code in ganesha. So, the path restricted caps are > sort of a defense-in-depth approach to prevent a rogue ganesha from > accessing stuff it shouldn't. You'll lose that if you change to always > requesting r/w caps -- maybe that's ok. > > It really sounds like you need some way to force libcephfs to > renegotiate its MDS sessions when there is a cap change for the > principal. Maybe the MDSs should close sessions when their caps change? > Or, you could add some new function to libcephfs to do that? That really > sounds like the right thing to do. > Thanks a lot, Jeff. Yes, this does seem like the right approach. Maybe this feature request already exists in https://tracker.ceph.com/ ? If not, I will create one and investigate further. > > This allows the driver to dynamically change > > the export's client access types by modifying the export's client > > blocks, and sending a signal to ganesha to reload its exports. The > > ganesha server enforces the NFS client access type changes, read-write > > to read-only or vice-versa, even though the CephFS export's libcephfs > > client always has read-write permissions to the CephFS subdirectory. I > > was wondering if we could do the same for the 'mgr/nfs' module with > > option a) suggested below. As you mentioned in IRC #cephfs, options b) > > and options c) are better avoided, and option d) is ideal but > > non-trivial to implement. > > > > > > However in the case of > > > > exports managed by mgr/nfs module, changing the access type of a > > > > CephFS export (or its clients) requires changing the MDS caps of the > > > > FSAL ceph user. AFAIK this change cannot take effect without a ganesha > > > > server restart. The alternative is to remove the existing export, and > > > > create a new one. Both these solutions to update access of a CephFS > > > > export are not ideal. I could think of the following solutions to > > > > allow dynamic update of CephFS exports with the mgr/nfs module : > > > > > > > > a) Modify mgr/nfs module to always create a FSAL ceph user with > > > > read-write path restricted MDS caps for CephFS exports. > > > > > > > > b) Add a CLI option (e.g. --fsal_ceph_user_access) to `ceph nfs > > > > > > nit: please don't mix dashes and underscores in option names. ;) > > > > > > > export create cephfs` and `ceph nfs export apply` commands that > > > > allows setting FSAL ceph user's MDS caps access type during CephFS > > > > export creation. Add another option to `ceph nfs export apply` command > > > > that does not try to change the MDS caps access level of the fsal ceph > > > > user when updating a CephFS export's access type. > > > > > > > > c) Add a CLI option (e.g. --fsal_ceph_user_access) to `ceph nfs > > > > export create cephfs` and `ceph nfs export apply` commands that > > > > allows setting FSAL ceph user's MDS caps access type during CephFS > > > > export creation. Add a new CLI `ceph nfs export update cephfs` that > > > > does not try to change the > > > > to change the MDS caps access level of the fsal ceph user when > > > > updating a CephFS export's access type. > > > > > > > > Option a) is the simplest, but too much of a security compromise? > > > > Option b) and c) may confuse users and add complexity to the code. Any > > > > other suggestions? > > > > > > > > > > d) roll some special handling in libcephfs that allows you to transition > > > an existing client's access? Then you could teach ganesha to trigger > > > this at the right times. It's a non-trivial solution, but probably the > > > smoothest way to achieve what you want. > > > > > > e) Live with restarts? I'm not sure why they're such an issue and what's > > > driving this requirement. > > > > > > > Thanks, > > > > Ramana > > > > > > > > [1] https://docs.ceph.com/en/latest/mgr/nfs/#export-management > > > > [2] https://github.com/ceph/ceph/blob/v16.2.7/src/pybind/mgr/nfs/export.py#L188 > > > > > > > > _______________________________________________ > > > > Dev mailing list -- dev@xxxxxxx > > > > To unsubscribe send an email to dev-leave@xxxxxxx > > > > > > > > > > -- > > > Jeff Layton <jlayton@xxxxxxxxxx> > > > > > > > Thanks, > > Ramana > > > > -- > Jeff Layton <jlayton@xxxxxxxxxx> > -Ramana _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
