Ricardo,
We chatted earlier about a new ceph mgr module that would spin up EXPORT
blocks for ganesha and stuff them into a RADOS object. Here's basically
what we're aiming for. I think it's pretty similar to what SuSE's
solution is doing so I think it'd be good to collaborate here.
Probably I should write this up in a doc somewhere, but here's what I'd
envision. First an overview:
The Rook.io ceph+ganesha CRD basically spins up nfs-ganesha pods under
k8s that don't export anything by default and have a fairly stock
config. Each ganesha daemon that is started has a boilerplate config
file that ends with a %url include like this:
%url rados://<pool>/<namespace>/conf-<nodeid>
The nodeid in this case is the unique nodeid within a cluster of ganesha
servers using the rados_cluster recovery backend in ganesha. Rook
enumerates these starting with 'a' and going through 'z' (and then 'aa',
'ab', etc.). So node 'a' would have a config object called "conf-a".
What we currently lack is the code to set up those conf-<nodeid>
objects. I know you have some code to do this sort of configuration via
the dashboard and a REST API. Would it make more sense to split this bit
out into a separate module, which would also allow it to be usable from
the command line?
My thinking was that we'd probably want to create a new mgr module for
that, and could wire it up to the command line with something like:
$ ceph nfsexport create --id=100 \
--pool=mypool \
--namespace=mynamespace \
--type=cephfs \
--volume=myfs \
--subvolume=/foo \
--pseudo=/foo \
--cephx_userid=admin \
--cephx_key=<base64 key> \
--client=10.0.0.0/8,ro,root \
--client=admhost,rw,none
...the "client" is a string that would be a tuple of client access
string, r/o or r/w, and the userid squashing mode, and could be
specified multiple times.
We'd also want to add a way to remove and enumerate exports. Maybe:
$ ceph nfsexport ls
$ ceph nfsexport rm --id=100
So the create command above would create an object called "export-100"
in the given rados_pool/rados_namespace.
>From there, we'd need to also be able to "link" and "unlink" these
export objects into the config files for each daemon. So if I have a
cluster of 2 servers with nodeids "a" and "b":
$ ceph nfsexport link --pool=mypool \
--namespace=mynamespace \
--id=100 \
--node=a \
--node=b
...with a corresponding "unlink" command. That would append objects
called "conf-a" and "conf-b" with this line:
%url rados://mypool/mynamespace/export-100
...and then call into the orchestrator to send a SIGHUP to the daemons
to make them pick up the new configs. We might also want to sanity check
whether any conf-* files are still linked to the export-* files before
removing those objects.
Thoughts?
--
Jeff Layton <jlayton@xxxxxxxxxx>
