On Mon, 4 Mar 2024 18:01:49 -0800 Mina Almasry wrote: > +Intro > +===== > + > +Device memory TCP (devmem TCP) enables receiving data directly into device > +memory (dmabuf). The feature is currently implemented for TCP sockets. > + > + > +Opportunity > +----------- > + > +A large amount of data transfers have device memory as the source and/or s/amount/number/ > +destination. Accelerators drastically increased the volume of such transfers. s/volume/prevalence/ > +Some examples include: > + > +- Distributed training, where ML accelerators, such as GPUs on different hosts, > + exchange data among them. s/among them// > +- Distributed raw block storage applications transfer large amounts of data with > + remote SSDs, much of this data does not require host processing. > + > +Today, the majority of the Device-to-Device data transfers the network are "Today" won't age well. > +implemented as the following low level operations: Device-to-Host copy, > +Host-to-Host network transfer, and Host-to-Device copy. > + > +The implementation is suboptimal, especially for bulk data transfers, and can /The implementation/The flow involving host copies/ > +put significant strains on system resources such as host memory bandwidth and > +PCIe bandwidth. > + > +Devmem TCP optimizes this use case by implementing socket APIs that enable > +the user to receive incoming network packets directly into device memory. > +More Info > +--------- > + > + slides, video > + https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html > + > + patchset > + [RFC PATCH v3 00/12] Device Memory TCP > + https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@xxxxxxxxxx/T/ Won't age well? :) > +Interface > +========= > + > +Example > +------- > + > +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up > +the RX path of this API. > + > +NIC Setup > +--------- > + > +Header split, flow steering, & RSS are required features for devmem TCP. > + > +Header split is used to split incoming packets into a header buffer in host > +memory, and a payload buffer in device memory. > + > +Flow steering & RSS are used to ensure that only flows targeting devmem land on > +RX queue bound to devmem. > + > +Enable header split & flow steering: > + > +:: You can put the :: at the end of the text, IIRC, like this: Enable header split & flow steering:: > + > + # enable header split (assuming priv-flag) > + ethtool --set-priv-flags eth1 enable-header-split on Olek added the "set" in commit 50d73710715d ("ethtool: add SET for TCP_DATA_SPLIT ringparam"), no need for the priv flag any more.
