On Mon, Dec 5, 2016 at 10:50 PM, Allen Samuels <Allen.Samuels@xxxxxxxxxxx> wrote: >> -----Original Message----- >> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- >> owner@xxxxxxxxxxxxxxx] On Behalf Of John Spray >> Sent: Monday, December 05, 2016 11:28 AM >> To: Lars Marowsky-Bree <lmb@xxxxxxxx> >> Cc: Ceph Development <ceph-devel@xxxxxxxxxxxxxxx> >> Subject: Re: disk enclosure LEDs >> >> On Mon, Dec 5, 2016 at 6:41 PM, Lars Marowsky-Bree <lmb@xxxxxxxx> >> wrote: >> > On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@xxxxxxxxxxx> >> wrote: >> > >> >> I'm indifferent to agent vs. agent-less. >> >> >> >> I *believe* that having a ceph-private distribution is easier/simpler/more >> reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I >> agree with John]. But this isn't a strongly held belief. >> >> >> >> I'm *metaphysically certain* that whatever distribution scheme is >> adopted that it not be optional. A large barrier to adoption of Ceph today is >> the lack of "middle-ware" that handles infrequent operational events (node >> addition/removal, media failure/recovery, migration, etc.). IMO, this middle- >> ware will have to be a standard part of Ceph, i.e., fully functional "out of the >> box" without site-specific twiddling (though having a mechanism to insert >> site-specific stuff is fine with me, it just can't be *required*). >> >> >> >> In my mind, the distribution scheme is the next step in the evolution >> >> of Ceph-mgr. It's what's missing :) >> > >> > I see the benefits of having a ceph-specific agent for hardware >> > interaction. However, that then shifts the problem for bootstrapping >> > said Ceph agent. >> >> Bootstrapping would be the same as we already have for installing OSDs and >> MDSs. So ceph-deploy/ceph-ansible/whatever needs to be able to do the >> same thing for the per-host agent that it currently does for OSDs, no overall >> increase in complexity. >> >> > And when you open the can of worms that is server addition/removal, >> > etc we start hitting the question of either spinning up a distribution >> > mechanism as well. >> > >> > When we want to look at container-izing Ceph in hyper-converged >> > environments, this gets even worse. >> >> I'm imagining that in a container-per-service model, where something >> external has configured the OSD containers to have access to the block >> device that they will run on, it doesn't seem unreasonable to have the same >> configuration process set up the ceph agent container with access to all the >> OSD block devices. What are your thoughts about how this would (or >> wouldn't) work? > > The current OSD design is per-drive and not-reliable. We need a piece of software, running on the target system, that's NOT per-drive and NOT not-reliable (i.e., reliable :)). We need the management system to be able to dig out of the OSD's system why it crashed -- i.e., read logs and other types of status, etc. It's possible to mutate the OSD there, but I don't think it's easy nor soon. I think I've lost you there -- what's the relation between what you've just said and the issue of containerisation? John >> >> > >> > e.g., the cephalopod turns into a cephaloblob. (Sorry. I'm terrible >> > with puns.) >> > >> > I need a mechanism for interacting with enclosures (to stick with the >> > example), but I don't need it to be part of Ceph, since I need it for >> > other parts of my infrastructure too anyway. >> > >> > >> > If it's part of Ceph, I end up writing a special case for Ceph. >> >> I think this would cease to be a problem for you if we just had a flag in Ceph >> to disable its own smartmontools type stuff? That way when someone was >> using an external tool there would be no conflict. >> >> There is some duplication of effort, but I don't think that's intrinsically >> problematic: I predict that we'll always have many users who do not take up >> any of the external tools and will benefit from the built-in Ceph bits. >> >> > And I need a way to handle it when Ceph itself isn't around yet; how >> > do I blink an enclosure that receives a new disk? Ah, I pre-register a >> > given enclosure with Ceph, before an OSD is even created. I know Ceph >> > has many tentacles, but ... ;-) >> >> While at runtime we shouldn't have two agents competing to manage the >> same device, I think it is reasonable to have a separate piece of software that >> does installation vs. does the ongoing monitoring. We shouldn't let the >> constraints over installation (especially the need to operate on cephless >> machines) restrict how we manage systems through their life cycles. Again, I >> don't think the built-in Ceph functionality is mutually exclusive with having a >> good external installation tool that touches some of the same functionality. >> >> John >> >> >> > >> > >> > Regards, >> > Lars >> > >> > -- >> > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, >> > HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to >> > their mistakes." -- Oscar Wilde >> > >> > -- >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> > in the body of a message to majordomo@xxxxxxxxxxxxxxx More >> majordomo >> > info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >> body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at >> http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
