Who is merging patches that are submitted to the mailing list? AFAIK, I don't have write access to the repo. > ping? > > On Fri, Sep 23, 2016 at 05:34:39PM -0700, Christoph Hellwig wrote: > > And drop various bits that aren't relevant for our unified tree > > (Windows support, build instructions, etc). > > > > Signed-off-by: Christoph Hellwig <hch@xxxxxx> > > --- > > Documentation/ibacm.md | 109 ++++++++++++++++++++++++++++++++++++++ > > ibacm/acm_notes.txt | 141 --------------------------------------- > ---------- > > 2 files changed, 109 insertions(+), 141 deletions(-) > > create mode 100644 Documentation/ibacm.md > > delete mode 100644 ibacm/acm_notes.txt > > > > diff --git a/Documentation/ibacm.md b/Documentation/ibacm.md > > new file mode 100644 > > index 0000000..8ed293d > > --- /dev/null > > +++ b/Documentation/ibacm.md > > @@ -0,0 +1,109 @@ > > +# The Assistant for InfiniBand Communication Management (IB ACM) > > + > > +The IB ACM library implements and provides a framework for name, > address, and > > +route resolution services over InfiniBand. The IB ACM provides > information > > +needed to establish a connection, but does not implement the CM > protocol. > > + > > +IB ACM services are used by librdmacm to implement the > rdma_resolve_addr, > > +rdma_resolve_route, and rdma_getaddrinfo routines. > > + > > +The IB ACM is focused on being scalable and efficient. The current > > +implementation limits network traffic, SA interactions, and > centralized > > +services. ACM supports multiple resolution protocols in order to > handle > > +different fabric topologies. > > + > > +This release is limited in its handling of dynamic changes. > > + > > +The IB ACM package is comprised of two components: the ibacm service > > +and a test/configuration utility - ib_acme. > > + > > +# Details > > + > > +### ib_acme > > + > > +The ib_acme program serves a dual role. It acts as a utility to > test > > +ibacm operation and help verify if the ibacm service and selected > > +protocol is usable for a given cluster configuration. > Additionally, > > +it automatically generates ibacm configuration files to assist with > > +or eliminate manual setup. > > + > > + > > +### acm configuration files > > + > > +The ibacm service relies on two configuration files. > > + > > +The acm_addr.cfg file contains name and address mappings for each IB > > +<device, port, pkey> endpoint. Although the names in the > acm_addr.cfg > > +file can be anything, ib_acme maps the host name and IP addresses to > > +the IB endpoints. > > + > > +The acm_opts.cfg file provides a set of configurable options for the > > +ibacm service, such as timeout, number of retries, logging level, > etc. > > +ib_acme generates the acm_opts.cfg file using static information. A > > +future enhancement would adjust options based on the current system > > +and cluster size. > > + > > +### ibacm > > + > > +The ibacm service is responsible for resolving names and addresses > to > > +InfiniBand path information and caching such data. It is implemented > as a > > +daemon that execute with administrative privileges. > > + > > +The ibacm implements a client interface over TCP sockets, which is > > +abstracted by the librdmacm library. One or more back-end protocols > are > > +used by the ibacm service to satisfy user requests. Although the > > +ibacm supports standard SA path record queries on the back-end, it > > +provides an experimental multicast resolution protocol in hope of > > +achieving greater scalability. The latter is not usable on all > fabric > > +topologies, specifically ones that may not have reversible paths. > > +Users should use the ib_acme utility to verify that multicast > protocol > > +is usable before running other applications. > > + > > +Conceptually, the ibacm service implements an ARP like protocol and > either > > +uses IB multicast records to construct path record data or queries > the > > +SA directly, depending on the selected route protocol. By default, > the > > +ibacm services uses and caches SA path record queries. > > + > > +Specifically, all IB endpoints join a number of multicast groups. > > +Multicast groups differ based on rates, mtu, sl, etc., and are > prioritized. > > +All participating endpoints must be able to communicate on the > lowest > > +priority multicast group. The ibacm assigns one or more > names/addresses > > +to each IB endpoint using the acm_addr.cfg file. Clients provide > source > > +and destination names or addresses as input to the service, and > receive > > +as output path record data. > > + > > +The service maps a client's source name/address to a local IB > endpoint. > > +If a client does not provide a source address, then the ibacm > service > > +will select one based on the destination and local routing tables. > If the > > +destination name/address is not cached locally, it sends a multicast > > +request out on the lowest priority multicast group on the local > endpoint. > > +The request carries a list of multicast groups that the sender can > use. > > +The recipient of the request selects the highest priority multicast > group > > +that it can use as well and returns that information directly to the > sender. > > +The request data is cached by all endpoints that receive the > multicast > > +request message. The source endpoint also caches the response and > uses > > +the multicast group that was selected to construct or obtain path > record > > +data, which is returned to the client. > > + > > +The current implementation of the IB ACM has several additional > restrictions: > > +- The ibacm is limited in its handling of dynamic changes; > > + the ibacm should be stopped and restarted if a cluster is > reconfigured. > > +- Support for IPv6 has not been verified. > > +- The number of addresses that can be assigned to a single endpoint > is > > + limited to 4. > > +- The number of multicast groups that an endpoint can support is > limited to 2. > > + > > +The ibacm contains several internal caches. These include caches > for > > +GID and LID destination addresses. These caches can be > optionally > > +preloaded. ibacm supports the OpenSM dump_pr plugin "full" > PathRecord > > +format which is used to preload these caches. The file format is > specified > > +in the ibacm_opts.cfg file via the route_preload setting which > should > > +be set to opensm_full_v1 for this file format. Default format is > > +none which does not preload these caches. See dump_pr.notes.txt in > dump_pr > > +for more information on the opensm_full_v1 file format and how to > configure > > +OpenSM to generate this file. > > + > > +Additionally, the name, IPv4, and IPv6 caches can be be preloaded by > using > > +the addr_preload option. The default is none which does not preload > these > > +caches. To preload these caches, set this option to acm_hosts and > > +configure the addr_data_file appropriately. > > diff --git a/ibacm/acm_notes.txt b/ibacm/acm_notes.txt > > deleted file mode 100644 > > index c975555..0000000 > > --- a/ibacm/acm_notes.txt > > +++ /dev/null > > @@ -1,141 +0,0 @@ > > -Assistant for InfiniBand Communication Management (IB ACM) > > - > > -Note: The IB ACM should be considered experimental. > > - > > - > > -Overview > > --------- > > -The IB ACM package implements and provides a framework for > experimental name, > > -address, and route resolution services over InfiniBand. It is > intended to > > -address connection setup scalability issues running MPI applications > on > > -large clusters. The IB ACM provides information needed to establish > a > > -connection, but does not implement the CM protocol. > > - > > -The librdmacm can invoke IB ACM services when built using the -- > with-ib_acm > > -option. The IB ACM services tie in under the rdma_resolve_addr, > > -rdma_resolve_route, and rdma_getaddrinfo routines. For maximum > benefit, > > -the rdma_getaddrinfo routine should be used, however existing > applications > > -should still see significant connection scaling benefits using the > calls > > -available in librdmacm 1.0.11 and previous releases. > > - > > -The IB ACM is focused on being scalable and efficient. The current > > -implementation limits network traffic, SA interactions, and > centralized > > -services. ACM supports multiple resolution protocols in order to > handle > > -different fabric topologies. > > - > > -This release is limited in its handling of dynamic changes. > > - > > -The IB ACM package is comprised of two components: the ibacm service > > -and a test/configuration utility - ib_acme. Both are userspace > components > > -and are available for Linux and Windows. Additional details are > given below. > > - > > - > > -Quick Start Guide > > ------------------ > > -1. Prerequisites: libibverbs and libibumad must be installed. > > - The IB stack should be running with IPoIB configured. > > - These steps assume that the user has administrative privileges. > > -2. Install the IB ACM package > > - This installs ibacm, and ib_acme. > > -3. Run ib_acme -A -O > > - This will generate IB ACM address and options configuration > files. > > - (acm_addr.cfg and acm_opts.cfg) > > -4. Run ibacm -D > > - This will run ibacm as service/daemon. > > - Because ibacm uses the libibumad interfaces, it should be run > with > > - administrative privileges. > > -5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v > > - This will verify that the ibacm service is running. > > -5. Install librdmacm. > > - The librdmacm will automatically use the ibacm service. > > - On failures, the librdmacm will fall back to normal resolution. > > - > > - > > -Details > > -------- > > -ib_acme: > > -The ib_acme program serves a dual role. It acts as a utility to > test > > -ibacm operation and help verify if the ibacm service and selected > > -protocol is usable for a given cluster configuration. > Additionally, > > -it automatically generates ibacm configuration files to assist with > > -or eliminate manual setup. > > - > > - > > -acm configuration files: > > -The ibacm service relies on two configuration files. > > - > > -The acm_addr.cfg file contains name and address mappings for each IB > > -<device, port, pkey> endpoint. Although the names in the > acm_addr.cfg > > -file can be anything, ib_acme maps the host name and IP addresses to > > -the IB endpoints. > > - > > -The acm_opts.cfg file provides a set of configurable options for the > > -ibacm service, such as timeout, number of retries, logging level, > etc. > > -ib_acme generates the acm_opts.cfg file using static information. A > > -future enhancement would adjust options based on the current system > > -and cluster size. > > - > > - > > -ibacm: > > -The ibacm service is responsible for resolving names and addresses > to > > -InfiniBand path information and caching such data. It is implemented > as a > > -daemon that execute with administrative privileges. > > - > > -The ibacm implements a client interface over TCP sockets, which is > > -abstracted by the librdmacm library. One or more back-end protocols > are > > -used by the ibacm service to satisfy user requests. Although the > > -ibacm supports standard SA path record queries on the back-end, it > > -provides an experimental multicast resolution protocol in hope of > > -achieving greater scalability. The latter is not usable on all > fabric > > -topologies, specifically ones that may not have reversible paths. > > -Users should use the ib_acme utility to verify that multicast > protocol > > -is usable before running other applications. > > - > > -Conceptually, the ibacm service implements an ARP like protocol and > either > > -uses IB multicast records to construct path record data or queries > the > > -SA directly, depending on the selected route protocol. By default, > the > > -ibacm services uses and caches SA path record queries. > > - > > -Specifically, all IB endpoints join a number of multicast groups. > > -Multicast groups differ based on rates, mtu, sl, etc., and are > prioritized. > > -All participating endpoints must be able to communicate on the > lowest > > -priority multicast group. The ibacm assigns one or more > names/addresses > > -to each IB endpoint using the acm_addr.cfg file. Clients provide > source > > -and destination names or addresses as input to the service, and > receive > > -as output path record data. > > - > > -The service maps a client's source name/address to a local IB > endpoint. > > -If a client does not provide a source address, then the ibacm > service > > -will select one based on the destination and local routing tables. > If the > > -destination name/address is not cached locally, it sends a multicast > > -request out on the lowest priority multicast group on the local > endpoint. > > -The request carries a list of multicast groups that the sender can > use. > > -The recipient of the request selects the highest priority multicast > group > > -that it can use as well and returns that information directly to the > sender. > > -The request data is cached by all endpoints that receive the > multicast > > -request message. The source endpoint also caches the response and > uses > > -the multicast group that was selected to construct or obtain path > record > > -data, which is returned to the client. > > - > > -The current implementation of the IB ACM has several additional > restrictions: > > -- The ibacm is limited in its handling of dynamic changes; > > - the ibacm should be stopped and restarted if a cluster is > reconfigured. > > -- Support for IPv6 has not been verified. > > -- The number of addresses that can be assigned to a single endpoint > is > > - limited to 4. > > -- The number of multicast groups that an endpoint can support is > limited to 2. > > - > > -The ibacm contains several internal caches. These include caches > for > > -GID and LID destination addresses. These caches can be > optionally > > -preloaded. ibacm supports the OpenSM dump_pr plugin "full" > PathRecord > > -format which is used to preload these caches. The file format is > specified > > -in the ibacm_opts.cfg file via the route_preload setting which > should > > -be set to opensm_full_v1 for this file format. Default format is > > -none which does not preload these caches. See dump_pr.notes.txt in > dump_pr > > -for more information on the opensm_full_v1 file format and how to > configure > > -OpenSM to generate this file. > > - > > -Additionally, the name, IPv4, and IPv6 caches can be be preloaded by > using > > -the addr_preload option. The default is none which does not preload > these > > -caches. To preload these caches, set this option to acm_hosts and > > -configure the addr_data_file appropriately. > > -- > > 2.1.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" > in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > ---end quoted text--- > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" > in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
