On Thu, 28 Feb 2019, Brett Niver wrote:
> Why do we care about state? At some level the code has reasons to
> want the LED to be either on or off...
Mostly we don't need to care. I can think of a couple problem
scenarios, though:
- Someone out of band turns a light on. Then ceph turns on another light,
a human sees the first light, a pulls the wrong drive.
- What if the host is down, but you want the health warning to go away?
There needs to be some 'force' option that will proceed to forget the
light was ever on when we can't reach the host, but that relies on a human
operator promising that the host really is off and thus the light won't
come back on.
- We have some bug/race in our code that means we fail to turn off the
light before removing our notion that the light is on. Maybe an aborted
attempt to turn the light on has some slow request wandering through the
orchestrator queue of stuff to do and finally executes sometime after we
tell the system to turn the light back off?
sage
>
> On Thu, Feb 28, 2019 at 9:11 AM Sage Weil <sweil@xxxxxxxxxx> wrote:
> >
> > On Thu, 28 Feb 2019, Tim Serong wrote:
> > > On 02/28/2019 09:50 AM, Travis Nielsen wrote:
> > > > On Wed, Feb 27, 2019 at 3:42 PM Sage Weil <sweil@xxxxxxxxxx> wrote:
> > > >>
> > > >> On Wed, 27 Feb 2019, Travis Nielsen wrote:
> > > >>> Some questions and comments:
> > > >>> - What is the user interaction? Is he specifying an OSD ID for which
> > > >>> he wants to blink the light or what is $PATH? If $PATH is a device
> > > >>> name such as /dev/sdb we would need to translate the OSD ID to the
> > > >>> device.
> > > >>
> > > >> Right now the module implements
> > > >>
> > > >> ceph device {ident,fault}-light-{on,off} <devid>
> > > >>
> > > >> although once this is all working we can also add commands that operate on
> > > >> osd IDs.
> > >
> > > Presumably the OSD commands will just be implemented directly inside
> > > ceph-mgr (which can get OSD metadata to map IDs back to the relevant
> > > hostnames and device paths)? Or is there anything special an individual
> > > orchesetrator might need to do for this case?
> >
> > Right, it'll just be a slightly more complicated command in the blinky
> > module (or wherever we move this code to later).
> >
> > > >>> - This feels like a "desired state" way of doing things since you want
> > > >>> a light on until you decide to turn it off. In this case, we could
> > > >>> create a CRD for desired state of device lights. CRDs are the way the
> > > >>> rook module should interact with the rook operator.
> > > >>> - Whenever the CRD changes, rook would update the lights. When
> > > >>> rook starts, it would also ensure the lights are set appropriately.
> > > >>> - If a CRD is created it could mean the light should turn on for
> > > >>> that device. If the CRD is deleted, the light should turn off. If
> > > >>> there were different blinking modes, there could be a setting in the
> > > >>> CRD to indicate such.
> > > >>
> > > >> That works. I was just thinking that since the mgr is already maintaining
> > > >> this set of desired-on lights we could keep the rook side of it simple.
> > > >>
> > > >
> > > > Ah i missed that the mgr already stored this state. So if we can't
> > > > detect the actual state of the lights, this means the mgr is only
> > > > keeping track of the desire to turn the light on or off? And this
> > > > would translate to a health warning if a light should be on.
> > > >
> > > >>> - What does it take to detect the current state of the lights? Do we
> > > >>> run lsmcli on each node? If so, the discovery daemonset would make
> > > >>> sense to do this.
> > > >>
> > > >> If rook took the additional step of detecting lights that are on (due to
> > > >> external actors) that would make the whole thing a bit more robust, and be
> > > >> a good reason to bother with teh complexity of a CRD. I don't see
> > > >> anything to get current status from the version I have on fedora 29,
> > > >> though.
> > > >>
> > > >>> If we didn't use a CRD, the rook module could store the settings in a
> > > >>> configmap, then run a k8s job itself to turn the lights on or off.
> > > >>> However, I'd say the CRDs are the more natural approach.
> > > >>
> > > >> If we can't detect the current state with current tools, I wonder if just
> > > >> having the mgr module schedule a one-off command to run lsmcli is
> > > >> simpler... does having rook store the state in a configmap or crd buy us
> > > >> anything?
> > > >>
> > > >
> > > > Right, if we can't detect the current state of the lights, rook can't
> > > > really manage the desired state and may not make sense for rook to get
> > > > involved here. The mgr module could easily run a k8s job directly to
> > > > turn the light on or off and we wouldn't worry about managing desired
> > > > state.
> > > I'd suggest the same is true for other ochestrators
> > > (ansible/deepsea/ssh). If we can't detect the state, we shouldn't do
> > > anything at the individual orchestrator level. (If we could detect
> > > state, we'd just want to pass it up to ceph-mgr, rather than having each
> > > individual module implement its own record of LED state)
> >
> > Right.
> >
> > sage
>
>
>
