> -----Original Message----- > From: Jakub Kicinski <kuba@xxxxxxxxxx> > Sent: Wednesday, September 8, 2021 6:21 PM > Subject: Re: [PATCH net-next 1/2] rtnetlink: Add new RTM_GETEECSTATE > message to get SyncE status > > On Wed, 8 Sep 2021 08:03:35 +0000 Machnikowski, Maciej wrote: > > > > Yep! Yet let's go one step at a time. I believe once we have the basics > (EEC > > > > monitoring and recovered clock configuration) we'll be able to > implement > > > > a basic functionality - and add bells and whistles later on, as there are > more > > > > capabilities that we could support in SW. > > > > > > The set API may shape how the get API looks. We need a minimal viable > > > API where the whole control part of it is not "firmware or proprietary > > > tools take care of that". > > > > > > Do you have public docs on how the whole solution works? > > > > The best reference would be my netdev 0x15 tutorial: > > https://netdevconf.info/0x15/session.html?Introduction-to-time- > synchronization-over-Ethernet > > The SyncE API considerations starts ~54:00, but basically we need API for: > > - Controlling the lane to pin mapping for clock recovery > > - Check the EEC/DPLL state and see what's the source of reference > frequency > > (in more advanced deployments) > > - control additional input and output pins (GNSS input, external inputs, > recovered > > frequency reference) > > Thanks, good presentation! I haven't seen much in terms of system > design, at the level similar to the Broadcom whitepaper you shared. See the "drawing" below. > > > > I believe this is the state-of-art: here's the Broadcom public one > > > > https://docs.broadcom.com/doc/1211168567832, I believe Marvel > > > > has similar solution. But would also be happy to hear others. > > > > > > Interesting. That reveals the need for also marking the backup > > > (/secondary) clock. > > > > That's optional, but useful. And here's where we need a feedback > > on which port/lane is currently used, as the switch may be automatic > > What do you mean by optional? How does the user know if there is > fallback or not? Is it that Intel is intending to support only > devices with up to 2 ports and the second port is always the > backup (apologies for speculating). That will be a part of pin config API. It needs to give info about the number of supported pins that the PHY/MAC can use as recovered clock outputs. Once the pin is assigned to the lane the recovered clock (divided or not) will appear on the configured PHY/MAC pin and EEC will be able to use it. If there is more than 1 available - they will have some priority assigned to them that will be known to the EEC/board designer. And we plan to support devices that only comes with 1 recovered clock output as well. > > > Have you seen any docs on how systems with discreet PHY ASICs mux > > > the clocks? > > > > Yes - unfortunately they are not public :( > > Mumble, mumble. Ido, Florian - any devices within your purview which > would support SyncE? OK Found one that's public: https://www.marvell.com/content/dam/marvell/en/public-collateral/transceivers/marvell-phys-transceivers-alaska-c-88x5113-datasheet.pdf see Fig. 23 and chapter 3.11 for details, but conceptually it's similar. > > > > Ethernet IP/PHY usually outputs a divided clock signal (since it's > > > > easier to route) derived from the RX clock. > > > > The DPLL connectivity is vendor-specific, as you can use it to connect > > > > some external signals, but you can as well just care about relying > > > > the SyncE clock and only allow recovering it and passing along > > > > the QL info when your EEC is locked. That's why I backed up from > > > > a full DPLL implementation in favor of a more generic EEC clock. > > > > > > What is an ECC clock? To me the PLL state in the Ethernet port is the > > > state of the recovered clock. enum if_eec_state has values like > > > holdover which seem to be more applicable to the "system wide" PLL. > > > > EEC is Ethernet Equipment Clock. In most cases this will be a DPLL, but that's > > not mandatory and I believe it may be different is switches where > > you only need to drive all ports TX from a single frequency source. In this > > case the DPLL can be embedded in the multiport PHY, > > > > > Let me ask this - if one port is training the link and the other one has > > > the lock and is the source - what state will be reported for each port? > > > > In this case the port that has the lock source will report the lock and > > the EEC_SRC_PORT flag. The port that trains the link will show the > > lock without the flag and once it completes the training sequence it will > > use the EEC's frequency to transmit the data so that the next hop is able > > to synchronize its EEC to the incoming RX frequency > > Alright, I don't like that. It feels like you're attaching one object's > information (ECC) to other objects (ports), and repeating it. Prof > Goczyła and dr Landowska would not be proud. Hahaha - did not expect them to pop up here :) It's true, but the problem is that they depend on each other. The path is: Lane0 ------------- |\ Pin0 RefN ____ ------------- | |-----------------| | synced clk ... | |-----------------| EEC |------------------ ------------- |/ PinN RefM|____ | Lane N MUX To get the full info a port needs to know the EEC state and which lane is used as a source (or rather - my lane or any other). The lane -> Pin mapping is buried in the PHY/MAC, but the source of frequency is in the EEC. What's even more - the Pin->Ref mapping is board specific. The viable solutions are: - Limit to the proposed "I drive the clock" vs "Someone drives it" and assume the Driver returns all info - return the EEC Ref index, figure out which pin is connected to it and then check which MAC/PHY lane that drives it. I assume option one is easy to implement and keep in the future even if we finally move to option 2 once we define EEC/DPLL subsystem. In future #1 can take the lock information from the DPLL subsystem, but will also enable simple deployments that won't expose the whole DPLL, like a filter PLL embedded in a multiport PHY that will only work for SyncE in which case this API will only touch a single component. > > > > The Time IP is again relative and vendor-specific. If SyncE is deployed > > > > alongside PTP it will most likely be tightly coupled, but if you only > > > > care about having a frequency source - it's not mandatory and it can be > > > > as well in the PHY IP. > > > > > > I would not think having just the freq is very useful. > > > > This depends on the deployment. There are couple popular frequencies > > Most popular are 2,048 kHz, 10 MHz and 64 kHz. There are many > > deployments that only require frequency sync without the phase > > and/or time. I.e. if you deploy frequency division duplex you only need the > > frequency reference, and the higher frequency you have - the faster you > can > > lock to it.
