On Tue, Jul 11, 2023 at 01:34:20PM -0700, Jakub Kicinski wrote: > > Yep. At the high end open standards based ethernet has also notably > > "failed" as well. Every switch vendor now offers their own proprietary > > ecosystem on a whole bunch of different axis. They all present > > "ethernet" toward the host but the host often needs to work in a > > special way to really take full advantage of the proprietary fabric > > behaviors. > > I'm not familiar with "high end open standards based on ethernet", would > those be some RDMA / storage things? For TCP/IP networks pretty much > the only things that matter in a switch are bandwidth, size of buffers, > power... Implementation stuff. I would say when you are getting into ethernet deployments with 25 or 51 Tbps switches directly connected to hosts running at >100G you are getting into the high end side of things. These are very expensive networks. They run complex congestion provoking workloads. They have sophisticated multi-pathing. They often use use a non-blocking topology. Congestion management is important. Making this work with good utilization, and low tail latency is a really hard problem. Many of the solutions come with switch features supporting it. You'd proably say these are not TCP focused networks, even though they are based on ethernet and IP. So I think of them as high end "standards based" ethernet and IP looking networks that have proprietary elements mixed in throughout. Right now there is a bit of a press war between vendors on 'ethernet for AI'. Both Broadcom and NVIDIA are taking techonlogies that were originally built for TCP ethernet networks and remixing/speeding them up to run roce workloads effectively. There is alot more information available now without NDA that shows some detail on this space. AWS's SRD multipathing, Broadcom "AI Ethernet" and NVIDIA's Spectrum-X spring to mind as topical to what these sorts of ethernet networks are. > A lot of "standardization" efforts are just attempts to prove to > a buyers that an ecosystem exists. Heh, that was probably more true years ago. These days it seems like some standardization is also being done so the large hyperscalers can improve their Approved Vendors List. I suppose as long as the result is something we can implement openly in Linux the motivation for standardization is less important. Jason
