On 06/09/2017 06:41 AM, Benjeman Meekhof wrote: > Hi Sage, > > We did at one time run multiple clusters on our OSD nodes and RGW > nodes (with Jewel). We accomplished this by putting code in our > puppet-ceph module that would create additional systemd units with > appropriate CLUSTER=name environment settings for clusters not named > ceph. IE, if the module were asked to configure OSD for a cluster > named 'test' it would copy/edit the ceph-osd service to create a > 'test-osd@.service' unit that would start instances with CLUSTER=test > so they would point to the right config file, etc Eventually on the > RGW side I started doing instance-specific overrides like > '/etc/systemd/system/ceph-radosgw@xxxxxxxxxxx.d/override.conf' so as > to avoid replicating the stock systemd unit. > > We gave up on multiple clusters on the OSD nodes because it wasn't > really that useful to maintain a separate 'test' cluster on the same > hardware. We continue to need ability to reference multiple clusters > for RGW nodes and other clients. For the other example, users of our > project might have their own Ceph clusters in addition to wanting to > use ours. > > If the daemon solution in the no-clustername future is to 'modify > systemd unit files to do something' we're already doing that so it's > not a big issue. However the current modification of over-riding > CLUSTER in the environment section of systemd files does seem cleaner > than over-riding an exec command to specify a different config file > and keyring path. Maybe systemd units could ship with those > arguments as variables for easily over-riding. systemd units can be templated/parameterized, but with only one parameter, the instance ID, which we're already using (ceph-mon@$(hostname), ceph-osd@$ID, etc.) > > thanks, > Ben > > On Thu, Jun 8, 2017 at 3:37 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: >> At CDM yesterday we talked about removing the ability to name your ceph >> clusters. There are a number of hurtles that make it difficult to fully >> get rid of this functionality, not the least of which is that some >> (many?) deployed clusters make use of it. We decided that the most we can >> do at this point is remove support for it in ceph-deploy and ceph-ansible >> so that no new clusters or deployed nodes use it. >> >> The first PR in this effort: >> >> https://github.com/ceph/ceph-deploy/pull/441 >> >> Background: >> >> The cluster name concept was added to allow multiple clusters to have >> daemons coexist on the same host. At the type it was a hypothetical >> requirement for a user that never actually made use of it, and the >> support is kludgey: >> >> - default cluster name is 'ceph' >> - default config is /etc/ceph/$cluster.conf, so that the normal >> 'ceph.conf' still works >> - daemon data paths include the cluster name, >> /var/lib/ceph/osd/$cluster-$id >> which is weird (but mostly people are used to it?) >> - any cli command you want to touch a non-ceph cluster name >> needs -C $name or --cluster $name passed to it. >> >> Also, as of jewel, >> >> - systemd only supports a single cluster per host, as defined by $CLUSTER >> in /etc/{sysconfig,default}/ceph >> >> which you'll notice removes support for the original "requirement". >> >> Also note that you can get the same effect by specifying the config path >> explicitly (-c /etc/ceph/foo.conf) along with the various options that >> substitute $cluster in (e.g., osd_data=/var/lib/ceph/osd/$cluster-$id). >> >> >> Crap preventing us from removing this entirely: >> >> - existing daemon directories for existing clusters >> - various scripts parse the cluster name out of paths >> >> >> Converting an existing cluster "foo" back to "ceph": >> >> - rename /etc/ceph/foo.conf -> ceph.conf >> - rename /var/lib/ceph/*/foo-* -> /var/lib/ceph/*/ceph-* >> - remove the CLUSTER=foo line in /etc/{default,sysconfig}/ceph >> - reboot >> >> >> Questions: >> >> - Does anybody on the list use a non-default cluster name? >> - If so, do you have a reason not to switch back to 'ceph'? >> >> Thanks! >> sage >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- Tim Serong Senior Clustering Engineer SUSE tserong@xxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
