And drop various bits that aren't relevant for our unified tree (Windows support, build instructions, etc). Signed-off-by: Christoph Hellwig <hch@xxxxxx> --- Documentation/ibacm.md | 109 ++++++++++++++++++++++++++++++++++++++ ibacm/acm_notes.txt | 141 ------------------------------------------------- 2 files changed, 109 insertions(+), 141 deletions(-) create mode 100644 Documentation/ibacm.md delete mode 100644 ibacm/acm_notes.txt diff --git a/Documentation/ibacm.md b/Documentation/ibacm.md new file mode 100644 index 0000000..8ed293d --- /dev/null +++ b/Documentation/ibacm.md @@ -0,0 +1,109 @@ +# The Assistant for InfiniBand Communication Management (IB ACM) + +The IB ACM library implements and provides a framework for name, address, and +route resolution services over InfiniBand. The IB ACM provides information +needed to establish a connection, but does not implement the CM protocol. + +IB ACM services are used by librdmacm to implement the rdma_resolve_addr, +rdma_resolve_route, and rdma_getaddrinfo routines. + +The IB ACM is focused on being scalable and efficient. The current +implementation limits network traffic, SA interactions, and centralized +services. ACM supports multiple resolution protocols in order to handle +different fabric topologies. + +This release is limited in its handling of dynamic changes. + +The IB ACM package is comprised of two components: the ibacm service +and a test/configuration utility - ib_acme. + +# Details + +### ib_acme + +The ib_acme program serves a dual role. It acts as a utility to test +ibacm operation and help verify if the ibacm service and selected +protocol is usable for a given cluster configuration. Additionally, +it automatically generates ibacm configuration files to assist with +or eliminate manual setup. + + +### acm configuration files + +The ibacm service relies on two configuration files. + +The acm_addr.cfg file contains name and address mappings for each IB +<device, port, pkey> endpoint. Although the names in the acm_addr.cfg +file can be anything, ib_acme maps the host name and IP addresses to +the IB endpoints. + +The acm_opts.cfg file provides a set of configurable options for the +ibacm service, such as timeout, number of retries, logging level, etc. +ib_acme generates the acm_opts.cfg file using static information. A +future enhancement would adjust options based on the current system +and cluster size. + +### ibacm + +The ibacm service is responsible for resolving names and addresses to +InfiniBand path information and caching such data. It is implemented as a +daemon that execute with administrative privileges. + +The ibacm implements a client interface over TCP sockets, which is +abstracted by the librdmacm library. One or more back-end protocols are +used by the ibacm service to satisfy user requests. Although the +ibacm supports standard SA path record queries on the back-end, it +provides an experimental multicast resolution protocol in hope of +achieving greater scalability. The latter is not usable on all fabric +topologies, specifically ones that may not have reversible paths. +Users should use the ib_acme utility to verify that multicast protocol +is usable before running other applications. + +Conceptually, the ibacm service implements an ARP like protocol and either +uses IB multicast records to construct path record data or queries the +SA directly, depending on the selected route protocol. By default, the +ibacm services uses and caches SA path record queries. + +Specifically, all IB endpoints join a number of multicast groups. +Multicast groups differ based on rates, mtu, sl, etc., and are prioritized. +All participating endpoints must be able to communicate on the lowest +priority multicast group. The ibacm assigns one or more names/addresses +to each IB endpoint using the acm_addr.cfg file. Clients provide source +and destination names or addresses as input to the service, and receive +as output path record data. + +The service maps a client's source name/address to a local IB endpoint. +If a client does not provide a source address, then the ibacm service +will select one based on the destination and local routing tables. If the +destination name/address is not cached locally, it sends a multicast +request out on the lowest priority multicast group on the local endpoint. +The request carries a list of multicast groups that the sender can use. +The recipient of the request selects the highest priority multicast group +that it can use as well and returns that information directly to the sender. +The request data is cached by all endpoints that receive the multicast +request message. The source endpoint also caches the response and uses +the multicast group that was selected to construct or obtain path record +data, which is returned to the client. + +The current implementation of the IB ACM has several additional restrictions: +- The ibacm is limited in its handling of dynamic changes; + the ibacm should be stopped and restarted if a cluster is reconfigured. +- Support for IPv6 has not been verified. +- The number of addresses that can be assigned to a single endpoint is + limited to 4. +- The number of multicast groups that an endpoint can support is limited to 2. + +The ibacm contains several internal caches. These include caches for +GID and LID destination addresses. These caches can be optionally +preloaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord +format which is used to preload these caches. The file format is specified +in the ibacm_opts.cfg file via the route_preload setting which should +be set to opensm_full_v1 for this file format. Default format is +none which does not preload these caches. See dump_pr.notes.txt in dump_pr +for more information on the opensm_full_v1 file format and how to configure +OpenSM to generate this file. + +Additionally, the name, IPv4, and IPv6 caches can be be preloaded by using +the addr_preload option. The default is none which does not preload these +caches. To preload these caches, set this option to acm_hosts and +configure the addr_data_file appropriately. diff --git a/ibacm/acm_notes.txt b/ibacm/acm_notes.txt deleted file mode 100644 index c975555..0000000 --- a/ibacm/acm_notes.txt +++ /dev/null @@ -1,141 +0,0 @@ -Assistant for InfiniBand Communication Management (IB ACM) - -Note: The IB ACM should be considered experimental. - - -Overview --------- -The IB ACM package implements and provides a framework for experimental name, -address, and route resolution services over InfiniBand. It is intended to -address connection setup scalability issues running MPI applications on -large clusters. The IB ACM provides information needed to establish a -connection, but does not implement the CM protocol. - -The librdmacm can invoke IB ACM services when built using the --with-ib_acm -option. The IB ACM services tie in under the rdma_resolve_addr, -rdma_resolve_route, and rdma_getaddrinfo routines. For maximum benefit, -the rdma_getaddrinfo routine should be used, however existing applications -should still see significant connection scaling benefits using the calls -available in librdmacm 1.0.11 and previous releases. - -The IB ACM is focused on being scalable and efficient. The current -implementation limits network traffic, SA interactions, and centralized -services. ACM supports multiple resolution protocols in order to handle -different fabric topologies. - -This release is limited in its handling of dynamic changes. - -The IB ACM package is comprised of two components: the ibacm service -and a test/configuration utility - ib_acme. Both are userspace components -and are available for Linux and Windows. Additional details are given below. - - -Quick Start Guide ------------------ -1. Prerequisites: libibverbs and libibumad must be installed. - The IB stack should be running with IPoIB configured. - These steps assume that the user has administrative privileges. -2. Install the IB ACM package - This installs ibacm, and ib_acme. -3. Run ib_acme -A -O - This will generate IB ACM address and options configuration files. - (acm_addr.cfg and acm_opts.cfg) -4. Run ibacm -D - This will run ibacm as service/daemon. - Because ibacm uses the libibumad interfaces, it should be run with - administrative privileges. -5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v - This will verify that the ibacm service is running. -5. Install librdmacm. - The librdmacm will automatically use the ibacm service. - On failures, the librdmacm will fall back to normal resolution. - - -Details -------- -ib_acme: -The ib_acme program serves a dual role. It acts as a utility to test -ibacm operation and help verify if the ibacm service and selected -protocol is usable for a given cluster configuration. Additionally, -it automatically generates ibacm configuration files to assist with -or eliminate manual setup. - - -acm configuration files: -The ibacm service relies on two configuration files. - -The acm_addr.cfg file contains name and address mappings for each IB -<device, port, pkey> endpoint. Although the names in the acm_addr.cfg -file can be anything, ib_acme maps the host name and IP addresses to -the IB endpoints. - -The acm_opts.cfg file provides a set of configurable options for the -ibacm service, such as timeout, number of retries, logging level, etc. -ib_acme generates the acm_opts.cfg file using static information. A -future enhancement would adjust options based on the current system -and cluster size. - - -ibacm: -The ibacm service is responsible for resolving names and addresses to -InfiniBand path information and caching such data. It is implemented as a -daemon that execute with administrative privileges. - -The ibacm implements a client interface over TCP sockets, which is -abstracted by the librdmacm library. One or more back-end protocols are -used by the ibacm service to satisfy user requests. Although the -ibacm supports standard SA path record queries on the back-end, it -provides an experimental multicast resolution protocol in hope of -achieving greater scalability. The latter is not usable on all fabric -topologies, specifically ones that may not have reversible paths. -Users should use the ib_acme utility to verify that multicast protocol -is usable before running other applications. - -Conceptually, the ibacm service implements an ARP like protocol and either -uses IB multicast records to construct path record data or queries the -SA directly, depending on the selected route protocol. By default, the -ibacm services uses and caches SA path record queries. - -Specifically, all IB endpoints join a number of multicast groups. -Multicast groups differ based on rates, mtu, sl, etc., and are prioritized. -All participating endpoints must be able to communicate on the lowest -priority multicast group. The ibacm assigns one or more names/addresses -to each IB endpoint using the acm_addr.cfg file. Clients provide source -and destination names or addresses as input to the service, and receive -as output path record data. - -The service maps a client's source name/address to a local IB endpoint. -If a client does not provide a source address, then the ibacm service -will select one based on the destination and local routing tables. If the -destination name/address is not cached locally, it sends a multicast -request out on the lowest priority multicast group on the local endpoint. -The request carries a list of multicast groups that the sender can use. -The recipient of the request selects the highest priority multicast group -that it can use as well and returns that information directly to the sender. -The request data is cached by all endpoints that receive the multicast -request message. The source endpoint also caches the response and uses -the multicast group that was selected to construct or obtain path record -data, which is returned to the client. - -The current implementation of the IB ACM has several additional restrictions: -- The ibacm is limited in its handling of dynamic changes; - the ibacm should be stopped and restarted if a cluster is reconfigured. -- Support for IPv6 has not been verified. -- The number of addresses that can be assigned to a single endpoint is - limited to 4. -- The number of multicast groups that an endpoint can support is limited to 2. - -The ibacm contains several internal caches. These include caches for -GID and LID destination addresses. These caches can be optionally -preloaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord -format which is used to preload these caches. The file format is specified -in the ibacm_opts.cfg file via the route_preload setting which should -be set to opensm_full_v1 for this file format. Default format is -none which does not preload these caches. See dump_pr.notes.txt in dump_pr -for more information on the opensm_full_v1 file format and how to configure -OpenSM to generate this file. - -Additionally, the name, IPv4, and IPv6 caches can be be preloaded by using -the addr_preload option. The default is none which does not preload these -caches. To preload these caches, set this option to acm_hosts and -configure the addr_data_file appropriately. -- 2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
