Greetings, I don’t see a way to open a ticket at rdma-core; it was suggested that I send this email instead. I have been chasing a problem in rdma-core-47.1. Originally, I opened a ticket in libfabric, but it was pointed out that mlx5 is not part of libfabric. Full description of the problem plus debug notes are documented at the github repository for libfabric, see issue 9792, please have a look there rather than repeating all of the background information in this email. An application started by pytorch does a fork, then the child process attempts to use libfabric to open a new DAOS infiniband endpoint. The original endpoint is owned and still in use by the parent process. When the parent process created the endpoint (fi_fabric, fi_domain, fi_endpoint calls), the mlx5 driver allocated memory pages for use in SRQ creation, and issued a madvise to say that the pages are DONTFORK. These pages are associated with the domain’s ibv_device which is cached in the driver. After the fork when the child process calls fi_domain for its new endpoint, it gets the ibv_device that was cached at the time it was created by the parent. The child process immediately segfaults when trying to create a SRQ, because the pages associated with that ibv_device are not in the child’s memory. There doesn’t appear to be any way for a child process to create a fresh endpoint because of the caching being done for ibv_devices. Is this the proper way to “open a ticket” against rdma-core? Regards, Kevan
