On Mon, Apr 5, 2021 at 2:48 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> On Mon, 2021-04-05 at 13:55 -0500, Sage Weil wrote:
> > On Mon, Apr 5, 2021 at 1:33 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> > >
> > > On Thu, 2021-04-01 at 11:04 +0200, Dan van der Ster wrote:
> > > > If one kernel mounts the
> > > > same cephfs several times (with different prefixes), we observed that
> > > > this is a unique client session. But does the ceph module globally
> > > > share a single copy of cluster metadata, e.g. osdmaps, or is that all
> > > > duplicated per session?
> > > >
> > >
> > > One copy per-cluster client, which should generally be shared between
> > > mounts to the same cluster, provided that you're using similar-enough
> > > mount options for the kernel to do that.
> >
> > I suspect the problem is that if these are coming from mgr/volumes,
> > then each mount has a unique cephx user (and a client cap that locks
> > them into the exported directory), which means that the client
> > instances can't be shared.
> >
>
> Oof. You're probably right. In that case, you're sort of SoL since you
> really do have to have a different client if the creds are different.
We might consider:
1. An alternate mgr/volumes auth mode/model where a single user has
access to the whole volume (i.e., all subvolumes). This might not
require any change in mgr/volumes itself, actually--just use
volume-granularity creds for the client.
2. A hybrid kernel client mode where we can share a single
ceph_{mon,osd}_client for the data path, but have independent
ceph_mds_clients for each mount. (As a practical matter, the osd caps
are identical, so it's annoying that each mount has independent OSD
connections.)
3. A mechanism for the caps to be refreshed for a client after the
connection is established. That might allow a per-client auth
identity to be used, and the caps for that client to be adjusted as
volumes are added/removed from that host.
Not really wild about any of these except for the first once, since it
probably requires minimal changes to ceph-csi only... :)
> Still, it's hard to imagine that it's _that_ much overhead, even at 350
> mounts, but I guess it depends on the amount of memory in the host.
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
