On Tue, Jul 31, 2012 at 11:51 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: > It is also possible to feed initial keys to the monitors during the 'mkfs' > stage. If the keys can be agreed on somehow beforehand, then they will > already be in place when the initial quorum is reached. Not sure if that > helps in this situation or not... Yeah, we're going that way for the "mon." key in the chef cookbooks (to get the mons talking to each other at all, that *has* to be done that way), but putting more and more stuff there is not very nice. Your typical CM framework does not let the recipe run arbitrary code at that sort of an instantiation time, and pushing this work on the admin makes it laborous and brittle; what happens when we need a new type of a bootstrap-foo key? Get all admins to cram an extra entry into their environment json? http://ceph.com/docs/master/config-cluster/chef/#configure-your-ceph-environment That just does not seem like a good way. Juju seems to provide a real-time notification mechanism between peers, using it's <name>-relation-changed hook. Other CM frameworks may need to step up their game, or be subject to the "keep re-running chef-client until it works" limitation. If the CM makes it safe to trigger a run manually (e.g. "sudo chef-client" whenever you feel like it), we can trigger that locally when we finally create the keys. This still doesn't help the receiving side to notice any faster. If the CM makes it safe for us to change node attributes outside of the full CM run, we can do trigger that when we finally create the keys. Chef seems to have a "full overwrites only" semantic, so this is probably not safe with it. And as above, this does not help the receiving side to notice that it has information to fetch. What I want to do longer term, is make the Chef cookbook for Ceph very thin, push everything except the cross-node communication into Ceph proper, and then write a "mkcephfs v2.0" that uses SSH connections as appropriate, from a central "workstation" host that can SSH anywhere, to trigger these actions ASAP. Then that becomes the goal for CM frameworks: "provide me a communication mechanism between these nodes that can do *this*". -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
