On Fri, 2022-01-21 at 17:39 -0500, Ramana Venkatesh Raja wrote: > On Fri, Jan 21, 2022 at 4:01 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > > > On Fri, 2022-01-21 at 15:44 -0500, Ramana Venkatesh Raja wrote: > > > Hi, > > > > > > The mgr/nfs module manages nfs-ganesha services including their > > > exports [1]. It has interfaces that allow creating ganesha exports for > > > CephFS using `ceph nfs export create cephfs` or `ceph nfs export > > > apply` command, and updating exports using `ceph nfs export apply` > > > command. When creating an export for CephFS the mgr/nfs module > > > creates a FSAL ceph user with path restricted MDS caps based on the > > > requested export or clients' access type and the CephFS path to be > > > exported [2]. The export's libcephfs client ends up with read-only or > > > read-write MDS caps to the path being exported. > > > > > > If a ganesha export's or clients' access type is updated, then it is > > > expected that the ganesha server enforces the access type change > > > dynamically i.e., enforces the change after a SIGHUP or DBus signal > > > and not require a ganesha server restart. > > > > > > > That's a pretty big expectation. There are all sorts of settings that > > are not generally changeable on the fly and that require some sort of > > server restart. > > > > NFS-ganesha already supports dynamic updates of its exports' access > type and export clients' access types, > https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-export-config.rst#configuration-reload > . The issue is that the mgr/nfs module does not allow NFS-ganesha to > do this for CephFS exports it currently creates. > > > > > Why are restarts such a problem? Are you just concerned about having to > > impose a grace period? > > > > I don't think restarting is a problem. If the NFS-ganesha server > already supports dynamic updates of the export's access type or > clients' access type, then it would be great if the underlying Ceph > FSAL can also be configured to support this feature. For example, > OpenStack manila's CephFS driver creates ganesha exports for CephFS > subdirectories. It always creates FSAL ceph user with path restricted > 'read-write' MDS caps. > Let me summarize this to make sure I understand: You're creating a cephx principal for each export with path restricted caps as documented here: https://docs.ceph.com/en/latest/cephfs/client-auth/ ...based on whether the export is RO or RW. When you later go to change the export from RO to RW, then you change the caps at the same time. The result though is that the new caps don't "take" until you restart ganesha. Is that correct? FSAL_CEPH doesn't care much about the RO or RW export modes. That's all enforced in common NFS code in ganesha. So, the path restricted caps are sort of a defense-in-depth approach to prevent a rogue ganesha from accessing stuff it shouldn't. You'll lose that if you change to always requesting r/w caps -- maybe that's ok. It really sounds like you need some way to force libcephfs to renegotiate its MDS sessions when there is a cap change for the principal. Maybe the MDSs should close sessions when their caps change? Or, you could add some new function to libcephfs to do that? That really sounds like the right thing to do. > This allows the driver to dynamically change > the export's client access types by modifying the export's client > blocks, and sending a signal to ganesha to reload its exports. The > ganesha server enforces the NFS client access type changes, read-write > to read-only or vice-versa, even though the CephFS export's libcephfs > client always has read-write permissions to the CephFS subdirectory. I > was wondering if we could do the same for the 'mgr/nfs' module with > option a) suggested below. As you mentioned in IRC #cephfs, options b) > and options c) are better avoided, and option d) is ideal but > non-trivial to implement. > > > > However in the case of > > > exports managed by mgr/nfs module, changing the access type of a > > > CephFS export (or its clients) requires changing the MDS caps of the > > > FSAL ceph user. AFAIK this change cannot take effect without a ganesha > > > server restart. The alternative is to remove the existing export, and > > > create a new one. Both these solutions to update access of a CephFS > > > export are not ideal. I could think of the following solutions to > > > allow dynamic update of CephFS exports with the mgr/nfs module : > > > > > > a) Modify mgr/nfs module to always create a FSAL ceph user with > > > read-write path restricted MDS caps for CephFS exports. > > > > > > b) Add a CLI option (e.g. --fsal_ceph_user_access) to `ceph nfs > > > > nit: please don't mix dashes and underscores in option names. ;) > > > > > export create cephfs` and `ceph nfs export apply` commands that > > > allows setting FSAL ceph user's MDS caps access type during CephFS > > > export creation. Add another option to `ceph nfs export apply` command > > > that does not try to change the MDS caps access level of the fsal ceph > > > user when updating a CephFS export's access type. > > > > > > c) Add a CLI option (e.g. --fsal_ceph_user_access) to `ceph nfs > > > export create cephfs` and `ceph nfs export apply` commands that > > > allows setting FSAL ceph user's MDS caps access type during CephFS > > > export creation. Add a new CLI `ceph nfs export update cephfs` that > > > does not try to change the > > > to change the MDS caps access level of the fsal ceph user when > > > updating a CephFS export's access type. > > > > > > Option a) is the simplest, but too much of a security compromise? > > > Option b) and c) may confuse users and add complexity to the code. Any > > > other suggestions? > > > > > > > d) roll some special handling in libcephfs that allows you to transition > > an existing client's access? Then you could teach ganesha to trigger > > this at the right times. It's a non-trivial solution, but probably the > > smoothest way to achieve what you want. > > > > e) Live with restarts? I'm not sure why they're such an issue and what's > > driving this requirement. > > > > > Thanks, > > > Ramana > > > > > > [1] https://docs.ceph.com/en/latest/mgr/nfs/#export-management > > > [2] https://github.com/ceph/ceph/blob/v16.2.7/src/pybind/mgr/nfs/export.py#L188 > > > > > > _______________________________________________ > > > Dev mailing list -- dev@xxxxxxx > > > To unsubscribe send an email to dev-leave@xxxxxxx > > > > > > > -- > > Jeff Layton <jlayton@xxxxxxxxxx> > > > > Thanks, > Ramana > -- Jeff Layton <jlayton@xxxxxxxxxx> _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
