> On Feb 9, 2023, at 3:05 AM, Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx> wrote: > > On Wed, 8 Feb 2023 10:03:57 -0800 > "Viacheslav A.Dubeyko" <viacheslav.dubeyko@xxxxxxxxxxxxx> wrote: > >>> On Feb 8, 2023, at 8:38 AM, Adam Manzanares <a.manzanares@xxxxxxxxxxx> wrote: >>> >>> On Thu, Feb 02, 2023 at 09:54:02AM +0000, Jonathan Cameron wrote: >>>> On Wed, 1 Feb 2023 12:04:56 -0800 >>>> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@xxxxxxxxxxxxx> wrote: >>>> >>>>>> >> >> <skipped> >> >>>>> >>>>> Most probably, we will have multiple FM implementations in firmware. >>>>> Yes, FM on host could be important for debug and to verify correctness >>>>> firmware-based implementations. But FM daemon on host could be important >>>>> to receive notifications and react somehow on these events. Also, journalling >>>>> of events/messages/events could be important responsibility of FM daemon >>>>> on host. >>>> >>>> I agree with an FM daemon somewhere (potentially running on the BMC type chip >>>> that also has the lower level FM-API access). I think it is somewhat >>>> separate from the rest of this on basis it may well just be talking redfish >>>> to the FM and there are lots of tools for that sort of handling already. >>>> >>> >>> I would be interested in particpating in a BOF about this topic. I wonder what >>> happens when we have multiple switches with multiple FMs each on a separate BMC. >>> In this case, does it make more sense to have an owner of the global FM state >>> be a user space application. Is this the job of the orchestrator? > > This partly comes down to terminology. Ultimately there is an FM that is > responsible for the whole fabric (could be distributed software) and that > in turn will talk to a the various BMCs that then talk to the switches. > > Depending on the setup it may not be necessary for any entity to see the > whole fabric. > > Interesting point in general though. I think it boils down to getting > layering in any software correct and that is easier done from outset. > > I don't know whether the redfish stuff is flexible enough to cover this, but > if it is, I'd envision, the actual FM talking redfish to a bunch of sub-FMs > and in turn presenting redfish to the orchestrator. > > Any of these components might run on separate machines, or in firmware on > some device, or indeed all run on one server that is acting as the FM and > a node in the orchestrator layer. > >>> >>> The BMC based FM seems to have scalability issues, but will we hit them in >>> practice any time soon. > > Who knows ;) If anyone builds the large scale fabric stuff in CXL 3.0 then > we definitely will in the medium term. > >> >> I had discussion recently and it looks like there are interesting points: >> (1) If we have multiple CXL switches (especially with complex hierarchy), then it is >> very compute-intensive activity. So, potentially, FM on firmware side could be not >> capable to digest and executes all responsibilities without potential performance >> degradation. > > There is firmware and their is firmware ;) It's not uncommon for BMCs to be > significant devices in their own right and run Linux or other heavy weight OSes. > >> (2) However, if we have FM on host side, then there is security concerns because >> FM sees everything and all details of multiple hosts and subsystems. > > Agreed. Other than testing I wouldn't expect the FM to run on a 'host', but in > at lest some implementations it will be running on a capable Linux machine. > In large fabrics that may be very capable indeed (basically a server dedicated to > this role). > >> (3) Technically speaking, there is one potential capability that user-space FM daemon >> can run as on host side as on CXL switch side. I mean here that if we implement >> user-space FM daemon, then it could be used to execute FM functionality on CXL >> switch side (maybe????). :) > > Sure, anything could run anywhere. We should draw up some 'reference' architectures > though to guide discussion down the line. Mind you I think there are a lot of > steps along the way and starting point should be a simple PoC where all the FM > stuff is in linux userspace (other than comms). That's easy enough to do. > If I get a quiet week or so I'll hammer out what we need on emulation side to > start playing with this. > > Jonathan > > > >> >> <skipped> >> >>>>>>> - Manage surprise removal of devices >>>>>> >>>>>> Likewise, beyond reporting I wouldn't expect the FM daemon to have any idea >>>>>> what to do in the way of managing this. Scream loudly? >>>>>> >>>>> >>>>> Maybe, it could require application(s) notification. Let’s imagine that application >>>>> uses some resources from removed device. Maybe, FM can manage kernel-space >>>>> metadata correction and helping to manage application requests to not existing >>>>> entities. >>>> >>>> Notifications for the host are likely to come via inband means - so type3 driver >>>> handling rather than related to FM. As far as the host is concerned this is the >>>> same as case where there is no FM and someone ripped a device out. >>>> >>>> There might indeed be meta data to manage, but doubt it will have anything to >>>> do with kernel. >>>> >>> >>> I've also had similar thoughts, I think the OS responds to notifications that >>> are generated in-band after changes to the state of the FM are made through >>> OOB means. >>> >>> I envision the host sends REDFISH requests to a switch BMC that has an FM >>> implementation. Once the changes are implemented by the FM it would show up >>> as changes to the PCIe hierarchy on a host, which is capable of responding to >>> such changes. >>> >> >> I think I am not completely follow your point. :) First of all, I assume that if host >> sends REDFISH request, then it will be expected the confirmation of request execution. >> It means for me that host needs to receive some packet that informs that request >> executed successfully or failed. It means that some subsystem or application requested >> this change and only after receiving the confirmation requested capabilities can be used. >> And if FM is on CXL switch side, then how FM will show up the changes? It sounds for me >> that some FM subsystem should be on the host side to receive confirmation/notification >> and to execute the real changes in PCIe hierarchy. Am missing something here? > > Another terminology issue I think. FM from CXL side of things is an abstract thing > (potentially highly layered / distributed) that acts on instructions from an > orchestrator (also potentially highly distributed, one implementation is hosts > can be the orchestrator) and configures the fabric. > The downstream APIs to the switches and EPs are all in FM-API (CXL spec) > Upstream probably all Redfish. What happens in between is impdef (though > obviously mapping to Redfish or FM-API as applicable may make it more > reuseable and flexible). > > I think some diagrams of what is where will help. > I think we need (note I've always kept the controller hosts as normal hosts as well > as that includes the case where it never uses the Fabric - so BMC type cases as > a subset without needing to double the number of diagrams). > > 1) Diagram of single host with the FM as one 'thing' on that host - direct interfaces > to a single switch - interfaces options include switch CCI MB, mctp of PCI VDM, > mctp over say i2c. > > 2) Diagram of same as above, with a multiple head device all connected to one host. > > 3) Diagram of 1 (maybe with MHD below switches), but now with multiple hosts, > one of which is responsible for fabric management. FM in that manager host > and orchestrator) - agents on other hosts able to send requests for services to that host. > > 4) Diagram of 3, but now with multiple switches, each with separate controlling host. > Some other hosts that don't have any fabric control. > Distributed FM across the controlling hosts. > > 5) Diagram of 4 but with layered FM and separate Orchestrator. Hosts all talk to the > orchestrator, that then talks to the FM. > > 6) 4, but push some management entities down into switches (from architecture point of > view this is no different from layered case with a separate BMC per switch - there > is still either a distribute FM or a layered FM, which the orchestrator talks to.) > > Can mess with exactly distribution of who does what across the various layers. > > I can sketch this lot up (and that will probably make some gaps in these cases apparent) > but will take a little while, hence text descriptions in the meantime. > > I come back to my personal view though - which is don't worry too much at this early > stage, beyond making sure we have some layering in code so that we can distribute > it across a distributed or layered architecture later! > I had slightly more simplified image in my mind. :) We definitely need to have diagrams to clarify the vision. But which collaboration tool could we use to work publicly on diagrams? Any suggestion? Thanks, Slava.
