Hi,
We've been debugging an issue with the rstream application, would be glad to get your help.
This application is part of the OFA logo program and therefore we've been debugging it.
Intermittently we get an error: Connection refused (stale connection ) on the second connect in the test.
(rstream -S all -T a )
It looks like in some cases the server side gets a new connection request before destroying the cm-id,
Leaving the remote id and remote qp in the remote_id_table and remote_qp_table
Attached are two traces (using ftrace)
good_trace when second connect succeeds
bad_trace when second connect attempt fails
I think this can be considered as an application issue, and rstream could be modified to try and re-connect
In case it fails.
Your input on this will be highly appreciated,
Thanks,
Michal
# tracer: function
#
# entries-in-buffer/entries-written: 274/274 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-93015 [002] .... 621086.276018: ib_cm_insert_listen <-rdma_listen
<...>-93015 [002] .... 621086.276019: ib_create_cm_id <-ib_cm_insert_listen
<...>-93015 [002] d... 621086.276988: cm_find_listen <-ib_cm_insert_listen
<...>-93015 [007] .... 621086.277864: ib_cm_insert_listen <-rdma_listen
<...>-93015 [007] .... 621086.277865: ib_create_cm_id <-ib_cm_insert_listen
<...>-93015 [007] d... 621086.278787: cm_find_listen <-ib_cm_insert_listen
<idle>-0 [001] ..s. 621088.046503: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 621088.046527: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 621088.046530: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 621088.046537: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 621088.046539: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 621088.046539: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 621088.046542: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 621088.046549: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 621088.047454: cm_req_handler <-cm_work_handler
kworker/1:1-541 [001] .... 621088.047455: ib_create_cm_id <-cm_req_handler
kworker/1:1-541 [001] .... 621088.048276: ib_init_ah_from_wc <-cm_req_handler
kworker/1:1-541 [001] .... 621088.048314: ib_find_gid_by_filter <-ib_init_ah_from_wc
kworker/1:1-541 [001] .... 621088.048316: cm_create_timewait_info <-cm_req_handler
kworker/1:1-541 [001] d... 621088.049152: cm_insert_remote_id <-cm_req_handler
kworker/1:1-541 [001] d... 621088.049995: cm_insert_remote_qpn <-cm_req_handler
kworker/1:1-541 [001] d... 621088.050848: cm_find_listen <-cm_req_handler
kworker/1:1-541 [001] .... 621088.051709: ib_get_cached_gid <-cm_req_handler
kworker/1:1-541 [001] .... 621088.051709: cm_init_av_by_path <-cm_req_handler
kworker/1:1-541 [001] d... 621088.051710: ib_find_cached_gid <-cm_init_av_by_path
kworker/1:1-541 [001] .... 621088.051711: ib_find_cached_pkey <-cm_init_av_by_path
kworker/1:1-541 [001] .... 621088.051712: ib_init_ah_from_path <-cm_init_av_by_path
kworker/1:1-541 [001] .... 621088.051712: ib_get_client_data <-ib_init_ah_from_path
kworker/1:1-541 [001] .... 621088.051716: ib_find_cached_gid_by_port <-ib_init_ah_from_path
kworker/1:1-541 [001] .... 621088.051717: cm_get_bth_pkey.isra.55 <-cm_req_handler
kworker/1:1-541 [001] .... 621088.051717: ib_get_cached_pkey <-cm_get_bth_pkey.isra.55
kworker/1:1-541 [001] .... 621088.051717: cm_process_work <-cm_req_handler
kworker/1:1-541 [001] .... 621088.051719: ib_get_net_dev_by_params <-cma_get_net_dev
kworker/1:1-541 [001] .... 621088.051729: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/1:1-541 [001] .... 621088.051729: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/1:1-541 [001] .... 621088.051731: ib_send_cm_mra <-cma_req_handler
kworker/1:1-541 [001] .... 621088.051732: cm_copy_private_data <-ib_send_cm_mra
<...>-93015 [004] .... 621088.052003: ib_uverbs_open <-chrdev_open
<...>-93015 [004] .... 621088.052009: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052010: ib_uverbs_get_context <-ib_uverbs_write
<...>-93015 [004] .... 621088.052013: ib_uverbs_alloc_event_file <-ib_uverbs_get_context
<...>-93015 [004] .... 621088.052015: ib_register_event_handler <-ib_uverbs_alloc_event_file
<...>-93015 [004] .... 621088.052019: ib_uverbs_mmap <-mmap_region
<...>-93015 [004] .... 621088.052034: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052034: ib_uverbs_query_device <-ib_uverbs_write
<...>-93015 [004] .... 621088.052042: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052042: ib_uverbs_query_port <-ib_uverbs_write
<...>-93015 [004] .... 621088.052043: ib_query_port <-ib_uverbs_query_port
<...>-93015 [004] .... 621088.052046: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052046: ib_uverbs_alloc_pd <-ib_uverbs_write
<...>-93015 [004] .... 621088.052051: ib_sa_pack_path <-ucma_query
<...>-93015 [004] .... 621088.052052: ib_pack <-ib_sa_pack_path
<...>-93015 [004] .... 621088.052057: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052058: ib_uverbs_create_comp_channel <-ib_uverbs_write
<...>-93015 [004] .... 621088.052058: ib_uverbs_alloc_event_file <-ib_uverbs_create_comp_channel
<...>-93015 [004] .... 621088.052067: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052067: ib_uverbs_create_cq <-ib_uverbs_write
<...>-93015 [004] .... 621088.052068: ib_uverbs_lookup_comp_file <-create_cq.constprop.18
<...>-93015 [004] .... 621088.052069: ib_umem_get <-qedr_create_cq
kworker/1:1-541 [001] .... 621088.052588: cm_free_work <-cm_process_work
kworker/1:1-541 [001] .... 621088.052588: ib_free_recv_mad <-cm_free_work
<...>-93015 [004] .... 621088.052694: ib_umem_page_count <-qedr_create_cq
<...>-93015 [004] .... 621088.052745: ib_uverbs_create_cq_cb <-create_cq.constprop.18
<...>-93015 [004] .... 621088.052761: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052761: ib_uverbs_create_qp <-ib_uverbs_write
<...>-93015 [004] .... 621088.052764: ib_umem_get <-qedr_create_user_qp
<...>-93015 [004] .... 621088.052809: ib_umem_page_count <-qedr_create_user_qp
<...>-93015 [004] .... 621088.052823: ib_umem_get <-qedr_create_user_qp
<...>-93015 [004] .... 621088.052835: ib_umem_page_count <-qedr_create_user_qp
<...>-93015 [004] .... 621088.052839: ib_uverbs_create_qp_cb <-create_qp.isra.16
<...>-93015 [004] .... 621088.052851: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-93015 [004] .... 621088.052851: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-93015 [004] .... 621088.052851: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.052852: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.052857: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052857: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-93015 [004] .... 621088.052858: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-93015 [004] .... 621088.052858: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-93015 [004] .... 621088.052871: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.052871: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621088.052872: ib_umem_get <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053288: ib_umem_page_count <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053363: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.053363: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621088.053364: ib_umem_get <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053367: ib_umem_page_count <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053400: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.053401: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621088.053401: ib_umem_get <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053816: ib_umem_page_count <-qedr_reg_user_mr
<...>-93015 [004] .... 621088.053986: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-93015 [004] .... 621088.053986: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-93015 [004] .... 621088.053986: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.053986: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.053987: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.053987: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-93015 [004] .... 621088.053988: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-93015 [004] .... 621088.053988: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-93015 [004] .... 621088.053989: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-93015 [004] .... 621088.053989: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-93015 [004] .... 621088.053989: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.053989: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.053990: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.053990: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-93015 [004] .... 621088.053990: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-93015 [004] .... 621088.053990: ib_query_gid <-ib_resolve_eth_dmac
<...>-93015 [004] .... 621088.053991: ib_get_cached_gid <-ib_query_gid
<...>-93015 [004] .... 621088.054050: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-93015 [004] .... 621088.054050: ib_get_cached_gid <-qedr_modify_qp
<...>-93015 [004] .... 621088.054115: ib_query_gid <-show_port_gid
<...>-93015 [004] .... 621088.054115: ib_get_cached_gid <-ib_query_gid
<...>-93015 [004] .... 621088.054126: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-93015 [004] .... 621088.054127: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-93015 [004] .... 621088.054127: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.054127: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-93015 [004] .... 621088.054128: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621088.054128: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-93015 [004] .... 621088.054129: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-93015 [004] .... 621088.054129: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-93015 [004] .... 621088.054179: ib_send_cm_rep <-rdma_accept
<...>-93015 [004] d... 621088.054179: cm_alloc_msg <-ib_send_cm_rep
<...>-93015 [004] d... 621088.054180: ib_create_ah <-cm_alloc_msg
<...>-93015 [004] d... 621088.054180: ib_create_send_mad <-cm_alloc_msg
<...>-93015 [004] d... 621088.054181: cm_format_rep <-ib_send_cm_rep
<...>-93015 [004] d... 621088.054182: ib_post_send_mad <-ib_send_cm_rep
<...>-93015 [004] d... 621088.054182: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-93015 [004] d... 621088.054182: ib_send_mad <-ib_post_send_mad
<...>-93015 [004] d... 621088.054184: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-93015 [004] d... 621088.054185: ib_ud_header_init <-qedr_gsi_build_packet
<...>-93015 [004] d... 621088.054185: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-93015 [004] d... 621088.054186: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621088.054186: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621088.054186: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621088.054186: ib_pack <-ib_ud_header_pack
<idle>-0 [000] ..s. 621088.054206: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [000] .... 621088.054210: ib_cq_poll_work <-process_one_work
<...>-84717 [000] .... 621088.054211: ib_mad_send_done <-__ib_process_cq
<...>-84717 [000] .... 621088.054213: ib_mad_complete_send_wr <-ib_mad_send_done
<idle>-0 [000] ..s. 621088.054403: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [000] .... 621088.054405: ib_cq_poll_work <-process_one_work
<...>-84717 [000] .... 621088.054405: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [000] d... 621088.054408: ib_response_mad <-find_mad_agent
<...>-84717 [000] .... 621088.054408: ib_response_mad <-ib_mad_recv_done
<...>-84717 [000] .... 621088.054408: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [000] .... 621088.054410: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/0:0-4 [000] .... 621088.054415: cm_work_handler <-process_one_work
kworker/0:0-4 [000] .... 621088.054417: cm_acquire_id <-cm_work_handler
kworker/0:0-4 [000] d... 621088.054417: cm_get_id <-cm_acquire_id
kworker/0:0-4 [000] d... 621088.056862: ib_cancel_mad <-cm_work_handler
kworker/0:0-4 [000] d... 621088.056863: ib_modify_mad <-ib_cancel_mad
kworker/0:0-4 [000] .... 621088.056872: cm_process_work <-cm_work_handler
kworker/0:0-4 [000] .... 621088.056874: cm_free_work <-cm_process_work
kworker/0:0-4 [000] .... 621088.056874: ib_free_recv_mad <-cm_free_work
<...>-58578 [003] .... 621088.056891: cm_send_handler <-timeout_sends
<...>-58578 [003] .... 621088.056892: cm_free_msg <-cm_send_handler
<...>-58578 [003] .... 621088.056892: ib_destroy_ah <-cm_free_msg
<...>-58578 [003] .... 621088.056893: ib_free_send_mad <-cm_free_msg
<...>-93015 [004] .... 621105.219530: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.219534: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-93015 [004] .... 621105.219536: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-93015 [004] .... 621105.219537: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-93015 [004] .... 621105.219619: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.219620: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621105.219621: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-93015 [004] .... 621105.219643: ib_umem_release <-qedr_dereg_mr
<...>-93015 [004] .... 621105.219795: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.219795: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621105.219796: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-93015 [004] .... 621105.219811: ib_umem_release <-qedr_dereg_mr
<...>-93015 [004] .... 621105.219953: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.219953: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-93015 [004] .... 621105.219953: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-93015 [004] .... 621105.219970: ib_umem_release <-qedr_dereg_mr
<...>-93015 [004] .... 621105.219984: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.219985: ib_uverbs_destroy_qp <-ib_uverbs_write
<...>-93015 [004] .... 621105.219985: ib_destroy_qp <-ib_uverbs_destroy_qp
<...>-93015 [004] .... 621105.220028: ib_umem_release <-qedr_free_qp_resources
<...>-93015 [004] .... 621105.220032: ib_umem_release <-qedr_free_qp_resources
<...>-93015 [004] .... 621105.220034: ib_uverbs_release_uevent <-ib_uverbs_destroy_qp
<...>-93015 [004] .... 621105.220068: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.220069: ib_uverbs_destroy_cq <-ib_uverbs_write
<...>-93015 [004] .... 621105.220069: ib_destroy_cq <-ib_uverbs_destroy_cq
<...>-93015 [004] .... 621105.220086: ib_umem_release <-qedr_destroy_cq
<...>-93015 [004] .... 621105.220089: ib_uverbs_release_ucq <-ib_uverbs_destroy_cq
<idle>-0 [000] ..s. 621105.303491: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [000] .... 621105.303518: ib_cq_poll_work <-process_one_work
<...>-84717 [000] .... 621105.303521: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [000] d... 621105.303528: ib_response_mad <-find_mad_agent
<...>-84717 [000] .... 621105.303530: ib_response_mad <-ib_mad_recv_done
<...>-84717 [000] .... 621105.303530: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [000] .... 621105.303533: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/0:0-4 [000] .... 621105.303541: cm_work_handler <-process_one_work
kworker/0:0-4 [000] .... 621105.304484: cm_acquire_id <-cm_work_handler
kworker/0:0-4 [000] d... 621105.304484: cm_get_id <-cm_acquire_id
kworker/0:0-4 [000] ..s. 621105.305329: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
kworker/0:0-4 [000] .... 621105.305331: cm_process_work <-cm_work_handler
kworker/0:0-4 [000] .... 621105.305334: cm_free_work <-cm_process_work
kworker/0:0-4 [000] .... 621105.305334: ib_free_recv_mad <-cm_free_work
<...>-84717 [003] .... 621105.305360: ib_cq_poll_work <-process_one_work
<...>-84717 [003] .... 621105.305362: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [003] d... 621105.305364: ib_response_mad <-find_mad_agent
<...>-84717 [003] .... 621105.305365: ib_response_mad <-ib_mad_recv_done
<...>-84717 [003] .... 621105.305365: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [003] .... 621105.305369: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/3:1-565 [003] .... 621105.305374: cm_work_handler <-process_one_work
kworker/3:1-565 [003] .... 621105.306234: cm_req_handler <-cm_work_handler
kworker/3:1-565 [003] .... 621105.306234: ib_create_cm_id <-cm_req_handler
kworker/3:1-565 [003] .... 621105.307080: ib_init_ah_from_wc <-cm_req_handler
kworker/3:1-565 [003] .... 621105.307118: ib_find_gid_by_filter <-ib_init_ah_from_wc
kworker/3:1-565 [003] .... 621105.307120: cm_create_timewait_info <-cm_req_handler
<...>-93015 [004] .... 621105.307985: ib_uverbs_event_close <-__fput
<...>-93015 [004] .... 621105.307998: ib_destroy_cm_id <-rdma_destroy_id
<...>-93015 [004] .... 621105.307998: cm_destroy_id <-ib_destroy_cm_id
kworker/3:1-565 [003] d... 621105.307998: cm_insert_remote_id <-cm_req_handler
kworker/3:1-565 [003] d... 621105.308002: cm_insert_remote_qpn <-cm_req_handler
kworker/3:1-565 [003] .... 621105.308010: cm_issue_rej.constprop.63 <-cm_req_handler
kworker/3:1-565 [003] .... 621105.308010: cm_alloc_response_msg.isra.57 <-cm_issue_rej.constprop.63
kworker/3:1-565 [003] .... 621105.308010: ib_create_ah_from_wc <-cm_alloc_response_msg.isra.57
kworker/3:1-565 [003] .... 621105.308010: ib_init_ah_from_wc <-ib_create_ah_from_wc
kworker/3:1-565 [003] .... 621105.308041: ib_find_gid_by_filter <-ib_init_ah_from_wc
kworker/3:1-565 [003] .... 621105.308042: ib_create_ah <-ib_create_ah_from_wc
kworker/3:1-565 [003] .... 621105.308043: ib_create_send_mad <-cm_alloc_response_msg.isra.57
kworker/3:1-565 [003] .... 621105.308044: ib_post_send_mad <-cm_issue_rej.constprop.63
kworker/3:1-565 [003] .... 621105.308045: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
kworker/3:1-565 [003] .... 621105.308045: ib_send_mad <-ib_post_send_mad
kworker/3:1-565 [003] d... 621105.308049: ib_get_cached_gid <-qedr_gsi_build_packet
kworker/3:1-565 [003] d... 621105.308050: ib_ud_header_init <-qedr_gsi_build_packet
kworker/3:1-565 [003] d... 621105.308051: ib_ud_header_pack <-qedr_gsi_build_packet
kworker/3:1-565 [003] d... 621105.308051: ib_pack <-ib_ud_header_pack
kworker/3:1-565 [003] d... 621105.308052: ib_pack <-ib_ud_header_pack
kworker/3:1-565 [003] d... 621105.308053: ib_pack <-ib_ud_header_pack
kworker/3:1-565 [003] d... 621105.308053: ib_pack <-ib_ud_header_pack
kworker/3:1-565 [003] .... 621105.308060: cm_destroy_id <-cm_req_handler
kworker/3:1-565 [003] .... 621105.308061: cm_free_work <-cm_work_handler
kworker/3:1-565 [003] .... 621105.308062: ib_free_recv_mad <-cm_free_work
systemd-journal-899 [000] ..s. 621105.308073: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [003] .... 621105.308078: ib_cq_poll_work <-process_one_work
<...>-84717 [003] .... 621105.308079: ib_mad_send_done <-__ib_process_cq
<...>-84717 [003] .... 621105.308081: ib_mad_complete_send_wr <-ib_mad_send_done
<...>-84717 [003] .... 621105.308082: cm_send_handler <-ib_mad_complete_send_wr
<...>-84717 [003] .... 621105.308082: cm_free_msg <-cm_send_handler
<...>-84717 [003] .... 621105.308082: ib_destroy_ah <-cm_free_msg
<...>-84717 [003] .... 621105.308083: ib_free_send_mad <-cm_free_msg
<...>-93015 [004] .... 621105.311656: ib_send_cm_drep <-cm_destroy_id
<...>-93015 [004] .... 621105.311851: cm_copy_private_data <-ib_send_cm_drep
<...>-93015 [004] d... 621105.311852: cm_enter_timewait <-ib_send_cm_drep
<...>-93015 [004] d... 621105.311852: ib_get_client_data <-cm_enter_timewait
<...>-93015 [004] d... 621105.316273: cm_alloc_msg <-ib_send_cm_drep
<...>-93015 [004] d... 621105.316274: ib_create_ah <-cm_alloc_msg
<...>-93015 [004] d... 621105.316275: ib_create_send_mad <-cm_alloc_msg
<...>-93015 [004] d... 621105.316276: ib_post_send_mad <-ib_send_cm_drep
<...>-93015 [004] d... 621105.316277: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-93015 [004] d... 621105.316277: ib_send_mad <-ib_post_send_mad
<...>-93015 [004] d... 621105.316283: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-93015 [004] d... 621105.316285: ib_ud_header_init <-qedr_gsi_build_packet
<...>-93015 [004] d... 621105.316286: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-93015 [004] d... 621105.316286: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621105.316287: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621105.316287: ib_pack <-ib_ud_header_pack
<...>-93015 [004] d... 621105.316287: ib_pack <-ib_ud_header_pack
<idle>-0 [000] .Ns. 621105.316913: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [003] .... 621105.316942: ib_cq_poll_work <-process_one_work
<...>-84717 [003] .... 621105.316943: ib_mad_send_done <-__ib_process_cq
<...>-84717 [003] .... 621105.316945: ib_mad_complete_send_wr <-ib_mad_send_done
<...>-84717 [003] .... 621105.316946: cm_send_handler <-ib_mad_complete_send_wr
<...>-84717 [003] .... 621105.316946: cm_free_msg <-cm_send_handler
<...>-84717 [003] .... 621105.316946: ib_destroy_ah <-cm_free_msg
<...>-84717 [003] .... 621105.316949: ib_free_send_mad <-cm_free_msg
<...>-93015 [004] .... 621105.316985: ib_uverbs_write <-vfs_write
<...>-93015 [004] .... 621105.316986: ib_uverbs_dealloc_pd <-ib_uverbs_write
kworker/4:1-557 [004] .... 621107.373153: cm_work_handler <-process_one_work
kworker/4:1-557 [004] .... 621107.374070: cm_acquire_id <-cm_work_handler
kworker/4:1-557 [004] d... 621107.374071: cm_get_id <-cm_acquire_id
kworker/4:1-557 [004] .... 621107.374071: cm_free_work <-cm_work_handler
# tracer: function
#
# entries-in-buffer/entries-written: 455/455 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-92046 [007] .... 620495.058232: ib_cm_insert_listen <-rdma_listen
<...>-92046 [007] .... 620495.058235: ib_create_cm_id <-ib_cm_insert_listen
<...>-92046 [007] d... 620495.059297: cm_find_listen <-ib_cm_insert_listen
<...>-92046 [007] .... 620495.060130: ib_cm_insert_listen <-rdma_listen
<...>-92046 [007] .... 620495.060130: ib_create_cm_id <-ib_cm_insert_listen
<...>-92046 [007] d... 620495.060950: cm_find_listen <-ib_cm_insert_listen
<idle>-0 [001] ..s. 620514.389824: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 620514.389853: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620514.389856: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 620514.389863: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 620514.389864: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 620514.389865: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 620514.389868: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 620514.389876: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 620514.390768: cm_req_handler <-cm_work_handler
kworker/1:1-541 [001] .... 620514.390769: ib_create_cm_id <-cm_req_handler
kworker/1:1-541 [001] .... 620514.391564: ib_init_ah_from_wc <-cm_req_handler
kworker/1:1-541 [001] .... 620514.391605: ib_find_gid_by_filter <-ib_init_ah_from_wc
kworker/1:1-541 [001] .... 620514.391607: cm_create_timewait_info <-cm_req_handler
kworker/1:1-541 [001] d... 620514.392415: cm_insert_remote_id <-cm_req_handler
kworker/1:1-541 [001] d... 620514.393231: cm_insert_remote_qpn <-cm_req_handler
kworker/1:1-541 [001] d... 620514.394056: cm_find_listen <-cm_req_handler
kworker/1:1-541 [001] .... 620514.394895: ib_get_cached_gid <-cm_req_handler
kworker/1:1-541 [001] .... 620514.394896: cm_init_av_by_path <-cm_req_handler
kworker/1:1-541 [001] d... 620514.394896: ib_find_cached_gid <-cm_init_av_by_path
kworker/1:1-541 [001] .... 620514.394897: ib_find_cached_pkey <-cm_init_av_by_path
kworker/1:1-541 [001] .... 620514.394898: ib_init_ah_from_path <-cm_init_av_by_path
kworker/1:1-541 [001] .... 620514.394898: ib_get_client_data <-ib_init_ah_from_path
kworker/1:1-541 [001] .... 620514.394903: ib_find_cached_gid_by_port <-ib_init_ah_from_path
kworker/1:1-541 [001] .... 620514.394903: cm_get_bth_pkey.isra.55 <-cm_req_handler
kworker/1:1-541 [001] .... 620514.394904: ib_get_cached_pkey <-cm_get_bth_pkey.isra.55
kworker/1:1-541 [001] .... 620514.394904: cm_process_work <-cm_req_handler
kworker/1:1-541 [001] .... 620514.394906: ib_get_net_dev_by_params <-cma_get_net_dev
kworker/1:1-541 [001] .... 620514.394916: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/1:1-541 [001] .... 620514.394916: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/1:1-541 [001] .... 620514.394918: ib_send_cm_mra <-cma_req_handler
kworker/1:1-541 [001] .... 620514.394919: cm_copy_private_data <-ib_send_cm_mra
<...>-92046 [004] .... 620514.395229: ib_uverbs_open <-chrdev_open
<...>-92046 [004] .... 620514.395237: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395238: ib_uverbs_get_context <-ib_uverbs_write
<...>-92046 [004] .... 620514.395241: ib_uverbs_alloc_event_file <-ib_uverbs_get_context
<...>-92046 [004] .... 620514.395244: ib_register_event_handler <-ib_uverbs_alloc_event_file
<...>-92046 [004] .... 620514.395248: ib_uverbs_mmap <-mmap_region
<...>-92046 [004] .... 620514.395261: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395262: ib_uverbs_query_device <-ib_uverbs_write
<...>-92046 [004] .... 620514.395269: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395269: ib_uverbs_query_port <-ib_uverbs_write
<...>-92046 [004] .... 620514.395269: ib_query_port <-ib_uverbs_query_port
<...>-92046 [004] .... 620514.395272: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395272: ib_uverbs_alloc_pd <-ib_uverbs_write
<...>-92046 [004] .... 620514.395278: ib_sa_pack_path <-ucma_query
<...>-92046 [004] .... 620514.395278: ib_pack <-ib_sa_pack_path
<...>-92046 [004] .... 620514.395283: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395284: ib_uverbs_create_comp_channel <-ib_uverbs_write
<...>-92046 [004] .... 620514.395284: ib_uverbs_alloc_event_file <-ib_uverbs_create_comp_channel
<...>-92046 [004] .... 620514.395293: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395294: ib_uverbs_create_cq <-ib_uverbs_write
<...>-92046 [004] .... 620514.395294: ib_uverbs_lookup_comp_file <-create_cq.constprop.18
<...>-92046 [004] .... 620514.395295: ib_umem_get <-qedr_create_cq
kworker/1:1-541 [001] .... 620514.395752: cm_free_work <-cm_process_work
kworker/1:1-541 [001] .... 620514.395752: ib_free_recv_mad <-cm_free_work
<...>-92046 [004] .... 620514.395921: ib_umem_page_count <-qedr_create_cq
<...>-92046 [004] .... 620514.395952: ib_uverbs_create_cq_cb <-create_cq.constprop.18
<...>-92046 [004] .... 620514.395967: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.395967: ib_uverbs_create_qp <-ib_uverbs_write
<...>-92046 [004] .... 620514.395970: ib_umem_get <-qedr_create_user_qp
<...>-92046 [004] .... 620514.396014: ib_umem_page_count <-qedr_create_user_qp
<...>-92046 [004] .... 620514.396026: ib_umem_get <-qedr_create_user_qp
<...>-92046 [004] .... 620514.396041: ib_umem_page_count <-qedr_create_user_qp
<...>-92046 [004] .... 620514.396046: ib_uverbs_create_qp_cb <-create_qp.isra.16
<...>-92046 [004] .... 620514.396056: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620514.396057: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620514.396057: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.396057: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.396062: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.396062: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620514.396063: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620514.396064: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620514.396077: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.396077: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620514.396078: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.396516: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.396578: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.396578: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620514.396579: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.396581: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.396604: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.396605: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620514.396605: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.397027: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620514.397185: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620514.397185: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620514.397185: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397185: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397186: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.397186: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620514.397186: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620514.397187: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620514.397188: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620514.397188: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620514.397188: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397188: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397189: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.397189: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620514.397189: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620514.397189: ib_query_gid <-ib_resolve_eth_dmac
<...>-92046 [004] .... 620514.397190: ib_get_cached_gid <-ib_query_gid
<...>-92046 [004] .... 620514.397234: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620514.397234: ib_get_cached_gid <-qedr_modify_qp
<...>-92046 [004] .... 620514.397288: ib_query_gid <-show_port_gid
<...>-92046 [004] .... 620514.397288: ib_get_cached_gid <-ib_query_gid
<...>-92046 [004] .... 620514.397300: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620514.397300: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620514.397300: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397301: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620514.397302: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620514.397302: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620514.397302: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620514.397302: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620514.397352: ib_send_cm_rep <-rdma_accept
<...>-92046 [004] d... 620514.397352: cm_alloc_msg <-ib_send_cm_rep
<...>-92046 [004] d... 620514.397353: ib_create_ah <-cm_alloc_msg
<...>-92046 [004] d... 620514.397354: ib_create_send_mad <-cm_alloc_msg
<...>-92046 [004] d... 620514.397354: cm_format_rep <-ib_send_cm_rep
<...>-92046 [004] d... 620514.397355: ib_post_send_mad <-ib_send_cm_rep
<...>-92046 [004] d... 620514.397355: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-92046 [004] d... 620514.397356: ib_send_mad <-ib_post_send_mad
<...>-92046 [004] d... 620514.397358: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-92046 [004] d... 620514.397359: ib_ud_header_init <-qedr_gsi_build_packet
<...>-92046 [004] d... 620514.397360: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-92046 [004] d... 620514.397360: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620514.397360: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620514.397360: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620514.397361: ib_pack <-ib_ud_header_pack
<idle>-0 [001] ..s. 620514.397380: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [001] .... 620514.397388: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620514.397390: ib_mad_send_done <-__ib_process_cq
<...>-84717 [001] .... 620514.397391: ib_mad_complete_send_wr <-ib_mad_send_done
<idle>-0 [001] ..s. 620514.397583: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 620514.397586: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620514.397587: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 620514.397588: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 620514.397589: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 620514.397589: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 620514.397590: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 620514.397595: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 620514.397597: cm_acquire_id <-cm_work_handler
kworker/1:1-541 [001] d... 620514.397597: cm_get_id <-cm_acquire_id
kworker/1:1-541 [001] d... 620514.399956: ib_cancel_mad <-cm_work_handler
kworker/1:1-541 [001] d... 620514.399956: ib_modify_mad <-ib_cancel_mad
kworker/1:1-541 [001] .... 620514.399970: cm_process_work <-cm_work_handler
kworker/1:1-541 [001] .... 620514.399972: cm_free_work <-cm_process_work
kworker/1:1-541 [001] .... 620514.399973: ib_free_recv_mad <-cm_free_work
<...>-58578 [002] .... 620514.399984: cm_send_handler <-timeout_sends
<...>-58578 [002] .... 620514.399985: cm_free_msg <-cm_send_handler
<...>-58578 [002] .... 620514.399985: ib_destroy_ah <-cm_free_msg
<...>-58578 [002] .... 620514.399987: ib_free_send_mad <-cm_free_msg
<...>-92046 [004] .N.. 620531.565375: ib_uverbs_write <-vfs_write
<...>-92046 [004] .N.. 620531.565379: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.565387: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620531.565388: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620531.565466: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.565467: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.565468: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620531.565490: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620531.565647: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.565648: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.565648: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620531.565663: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620531.565814: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.565814: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.565814: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620531.565828: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620531.565842: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.565843: ib_uverbs_destroy_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.565843: ib_destroy_qp <-ib_uverbs_destroy_qp
<...>-92046 [004] .... 620531.565878: ib_umem_release <-qedr_free_qp_resources
<...>-92046 [004] .... 620531.565882: ib_umem_release <-qedr_free_qp_resources
<...>-92046 [004] .... 620531.565884: ib_uverbs_release_uevent <-ib_uverbs_destroy_qp
<...>-92046 [004] .... 620531.565910: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.565910: ib_uverbs_destroy_cq <-ib_uverbs_write
<...>-92046 [004] .... 620531.565911: ib_destroy_cq <-ib_uverbs_destroy_cq
<...>-92046 [004] .... 620531.565927: ib_umem_release <-qedr_destroy_cq
<...>-92046 [004] .... 620531.565930: ib_uverbs_release_ucq <-ib_uverbs_destroy_cq
<idle>-0 [001] ..s. 620531.649224: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 620531.649252: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620531.649255: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 620531.649262: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 620531.649263: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 620531.649264: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 620531.649266: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 620531.649275: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 620531.650187: cm_acquire_id <-cm_work_handler
kworker/1:1-541 [001] d... 620531.650187: cm_get_id <-cm_acquire_id
kworker/1:1-541 [001] ..s. 620531.651005: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
kworker/1:1-541 [001] .... 620531.651007: cm_process_work <-cm_work_handler
kworker/1:1-541 [001] .... 620531.651011: cm_free_work <-cm_process_work
kworker/1:1-541 [001] .... 620531.651011: ib_free_recv_mad <-cm_free_work
<...>-84717 [003] .... 620531.651037: ib_cq_poll_work <-process_one_work
<...>-84717 [003] .... 620531.651038: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [003] d... 620531.651041: ib_response_mad <-find_mad_agent
<...>-84717 [003] .... 620531.651041: ib_response_mad <-ib_mad_recv_done
<...>-84717 [003] .... 620531.651042: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [003] .... 620531.651045: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/3:1-565 [003] .... 620531.651050: cm_work_handler <-process_one_work
kworker/3:1-565 [003] .... 620531.651882: cm_req_handler <-cm_work_handler
kworker/3:1-565 [003] .... 620531.651882: ib_create_cm_id <-cm_req_handler
<...>-92046 [004] .... 620531.652490: ib_uverbs_event_close <-__fput
<...>-92046 [004] .... 620531.652502: ib_destroy_cm_id <-rdma_destroy_id
<...>-92046 [004] .... 620531.652502: cm_destroy_id <-ib_destroy_cm_id
<...>-92046 [004] .... 620531.652505: ib_send_cm_drep <-cm_destroy_id
<...>-92046 [004] .... 620531.652665: cm_copy_private_data <-ib_send_cm_drep
<...>-92046 [004] d... 620531.652665: cm_enter_timewait <-ib_send_cm_drep
<...>-92046 [004] d... 620531.652666: ib_get_client_data <-cm_enter_timewait
<...>-92046 [004] d... 620531.652671: cm_alloc_msg <-ib_send_cm_drep
<...>-92046 [004] d... 620531.652672: ib_create_ah <-cm_alloc_msg
<...>-92046 [004] d... 620531.652673: ib_create_send_mad <-cm_alloc_msg
<...>-92046 [004] d... 620531.652674: ib_post_send_mad <-ib_send_cm_drep
<...>-92046 [004] d... 620531.652675: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-92046 [004] d... 620531.652675: ib_send_mad <-ib_post_send_mad
<...>-92046 [004] d... 620531.652680: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.652681: ib_ud_header_init <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.652683: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.652683: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.652683: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.652684: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.652684: ib_pack <-ib_ud_header_pack
<idle>-0 [001] ..s. 620531.652727: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [001] .... 620531.652735: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620531.652736: ib_mad_send_done <-__ib_process_cq
<...>-84717 [001] .... 620531.652737: ib_mad_complete_send_wr <-ib_mad_send_done
<...>-84717 [001] .... 620531.652738: cm_send_handler <-ib_mad_complete_send_wr
<...>-84717 [001] .... 620531.652739: cm_free_msg <-cm_send_handler
<...>-84717 [001] .... 620531.652739: ib_destroy_ah <-cm_free_msg
<...>-84717 [001] .... 620531.652741: ib_free_send_mad <-cm_free_msg
<...>-92046 [004] .... 620531.652761: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.652762: ib_uverbs_dealloc_pd <-ib_uverbs_write
kworker/3:1-565 [003] .... 620531.657499: ib_init_ah_from_wc <-cm_req_handler
kworker/3:1-565 [003] .... 620531.657545: ib_find_gid_by_filter <-ib_init_ah_from_wc
kworker/3:1-565 [003] .... 620531.657546: cm_create_timewait_info <-cm_req_handler
kworker/3:1-565 [003] d... 620531.658379: cm_insert_remote_id <-cm_req_handler
kworker/3:1-565 [003] d... 620531.659203: cm_insert_remote_qpn <-cm_req_handler
kworker/3:1-565 [003] d... 620531.660034: cm_find_listen <-cm_req_handler
kworker/3:1-565 [003] .... 620531.660863: ib_get_cached_gid <-cm_req_handler
kworker/3:1-565 [003] .... 620531.660863: cm_init_av_by_path <-cm_req_handler
kworker/3:1-565 [003] d... 620531.660864: ib_find_cached_gid <-cm_init_av_by_path
kworker/3:1-565 [003] .... 620531.660865: ib_find_cached_pkey <-cm_init_av_by_path
kworker/3:1-565 [003] .... 620531.660866: ib_init_ah_from_path <-cm_init_av_by_path
kworker/3:1-565 [003] .... 620531.660866: ib_get_client_data <-ib_init_ah_from_path
kworker/3:1-565 [003] .... 620531.660870: ib_find_cached_gid_by_port <-ib_init_ah_from_path
kworker/3:1-565 [003] .... 620531.660871: cm_get_bth_pkey.isra.55 <-cm_req_handler
kworker/3:1-565 [003] .... 620531.660871: ib_get_cached_pkey <-cm_get_bth_pkey.isra.55
kworker/3:1-565 [003] .... 620531.660871: cm_process_work <-cm_req_handler
kworker/3:1-565 [003] .... 620531.660873: ib_get_net_dev_by_params <-cma_get_net_dev
kworker/3:1-565 [003] .... 620531.660885: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/3:1-565 [003] .... 620531.660885: ib_find_cached_gid_by_port <-cma_acquire_dev
kworker/3:1-565 [003] .... 620531.660887: ib_send_cm_mra <-cma_req_handler
kworker/3:1-565 [003] .... 620531.660888: cm_copy_private_data <-ib_send_cm_mra
<...>-92046 [004] .... 620531.661007: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661008: ib_uverbs_alloc_pd <-ib_uverbs_write
<...>-92046 [004] .... 620531.661016: ib_sa_pack_path <-ucma_query
<...>-92046 [004] .... 620531.661016: ib_pack <-ib_sa_pack_path
<...>-92046 [004] .... 620531.661021: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661021: ib_uverbs_create_comp_channel <-ib_uverbs_write
<...>-92046 [004] .... 620531.661022: ib_uverbs_alloc_event_file <-ib_uverbs_create_comp_channel
<...>-92046 [004] .... 620531.661034: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661035: ib_uverbs_create_cq <-ib_uverbs_write
<...>-92046 [004] .... 620531.661035: ib_uverbs_lookup_comp_file <-create_cq.constprop.18
<...>-92046 [004] .... 620531.661036: ib_umem_get <-qedr_create_cq
<...>-92046 [004] .... 620531.661670: ib_umem_page_count <-qedr_create_cq
<...>-92046 [004] .... 620531.661722: ib_uverbs_create_cq_cb <-create_cq.constprop.18
kworker/3:1-565 [003] .... 620531.661726: cm_free_work <-cm_process_work
kworker/3:1-565 [003] .... 620531.661727: ib_free_recv_mad <-cm_free_work
<...>-92046 [004] .... 620531.661733: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661733: ib_uverbs_create_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.661736: ib_umem_get <-qedr_create_user_qp
<...>-92046 [004] .... 620531.661786: ib_umem_page_count <-qedr_create_user_qp
<...>-92046 [004] .... 620531.661798: ib_umem_get <-qedr_create_user_qp
<...>-92046 [004] .... 620531.661815: ib_umem_page_count <-qedr_create_user_qp
<...>-92046 [004] .... 620531.661820: ib_uverbs_create_qp_cb <-create_qp.isra.16
<...>-92046 [004] .... 620531.661826: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620531.661827: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620531.661827: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.661827: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.661830: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661831: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.661831: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620531.661832: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620531.661878: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.661879: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.661880: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.662294: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.662355: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.662356: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.662356: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.662358: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.662449: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.662449: ib_uverbs_reg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620531.662450: ib_umem_get <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.663029: ib_umem_page_count <-qedr_reg_user_mr
<...>-92046 [004] .... 620531.663164: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620531.663164: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620531.663164: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663164: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663165: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.663166: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.663166: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620531.663166: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620531.663167: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620531.663168: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620531.663168: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663168: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663169: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.663169: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.663169: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620531.663169: ib_query_gid <-ib_resolve_eth_dmac
<...>-92046 [004] .... 620531.663169: ib_get_cached_gid <-ib_query_gid
<...>-92046 [004] .... 620531.663185: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620531.663185: ib_get_cached_gid <-qedr_modify_qp
<...>-92046 [004] .... 620531.663300: ib_query_gid <-show_port_gid
<...>-92046 [004] .... 620531.663300: ib_get_cached_gid <-ib_query_gid
<...>-92046 [004] .... 620531.663313: ib_cm_init_qp_attr <-rdma_init_qp_attr
<...>-92046 [004] .... 620531.663313: ib_copy_qp_attr_to_user <-ucma_init_qp_attr
<...>-92046 [004] .... 620531.663314: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663314: ib_copy_ah_attr_to_user <-ib_copy_qp_attr_to_user
<...>-92046 [004] .... 620531.663315: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620531.663315: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620531.663315: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620531.663316: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620531.663365: ib_send_cm_rep <-rdma_accept
<...>-92046 [004] d... 620531.663366: cm_alloc_msg <-ib_send_cm_rep
<...>-92046 [004] d... 620531.663366: ib_create_ah <-cm_alloc_msg
<...>-92046 [004] d... 620531.663366: ib_create_send_mad <-cm_alloc_msg
<...>-92046 [004] d... 620531.663367: cm_format_rep <-ib_send_cm_rep
<...>-92046 [004] d... 620531.663367: ib_post_send_mad <-ib_send_cm_rep
<...>-92046 [004] d... 620531.663368: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-92046 [004] d... 620531.663368: ib_send_mad <-ib_post_send_mad
<...>-92046 [004] d... 620531.663369: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.663370: ib_ud_header_init <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.663370: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-92046 [004] d... 620531.663370: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.663370: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.663370: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620531.663371: ib_pack <-ib_ud_header_pack
<idle>-0 [001] ..s. 620531.663390: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [001] .... 620531.663397: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620531.663398: ib_mad_send_done <-__ib_process_cq
<...>-84717 [001] .... 620531.663399: ib_mad_complete_send_wr <-ib_mad_send_done
<idle>-0 [001] ..s. 620531.663611: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 620531.663614: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620531.663614: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 620531.663616: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 620531.663616: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 620531.663617: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 620531.663618: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 620531.663622: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 620531.663624: cm_acquire_id <-cm_work_handler
kworker/1:1-541 [001] d... 620531.663624: cm_get_id <-cm_acquire_id
kworker/1:1-541 [001] d... 620531.666008: ib_cancel_mad <-cm_work_handler
kworker/1:1-541 [001] d... 620531.666008: ib_modify_mad <-ib_cancel_mad
kworker/1:1-541 [001] .... 620531.666019: cm_process_work <-cm_work_handler
kworker/1:1-541 [001] .... 620531.666020: cm_free_work <-cm_process_work
kworker/1:1-541 [001] .... 620531.666020: ib_free_recv_mad <-cm_free_work
<...>-58578 [007] .... 620531.666039: cm_send_handler <-timeout_sends
<...>-58578 [007] .... 620531.666040: cm_free_msg <-cm_send_handler
<...>-58578 [007] .... 620531.666040: ib_destroy_ah <-cm_free_msg
<...>-58578 [007] .... 620531.666041: ib_free_send_mad <-cm_free_msg
kworker/4:1-557 [004] .... 620533.709492: cm_work_handler <-process_one_work
kworker/4:1-557 [004] .... 620533.710351: cm_acquire_id <-cm_work_handler
kworker/4:1-557 [004] d... 620533.710351: cm_get_id <-cm_acquire_id
kworker/4:1-557 [004] .... 620533.710351: cm_free_work <-cm_work_handler
<...>-92046 [004] .... 620537.587492: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587496: ib_uverbs_modify_qp <-ib_uverbs_write
<...>-92046 [004] .... 620537.587498: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
<...>-92046 [004] .... 620537.587499: ib_modify_qp_is_ok <-qedr_modify_qp
<...>-92046 [004] .... 620537.587572: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587572: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620537.587573: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620537.587596: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620537.587644: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587645: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620537.587645: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620537.587660: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620537.587891: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587891: ib_uverbs_dereg_mr <-ib_uverbs_write
<...>-92046 [004] .... 620537.587891: ib_dereg_mr <-ib_uverbs_dereg_mr
<...>-92046 [004] .... 620537.587905: ib_umem_release <-qedr_dereg_mr
<...>-92046 [004] .... 620537.587911: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587911: ib_uverbs_destroy_qp <-ib_uverbs_write
<...>-92046 [004] .... 620537.587911: ib_destroy_qp <-ib_uverbs_destroy_qp
<...>-92046 [004] .... 620537.587947: ib_umem_release <-qedr_free_qp_resources
<...>-92046 [004] .... 620537.587951: ib_umem_release <-qedr_free_qp_resources
<...>-92046 [004] .... 620537.587953: ib_uverbs_release_uevent <-ib_uverbs_destroy_qp
<...>-92046 [004] .... 620537.587977: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.587977: ib_uverbs_destroy_cq <-ib_uverbs_write
<...>-92046 [004] .... 620537.587978: ib_destroy_cq <-ib_uverbs_destroy_cq
<...>-92046 [004] .... 620537.588000: ib_umem_release <-qedr_destroy_cq
<...>-92046 [004] .... 620537.588002: ib_uverbs_release_ucq <-ib_uverbs_destroy_cq
<...>-92046 [004] .... 620537.627691: ib_uverbs_event_close <-__fput
<...>-92046 [004] .... 620537.627706: ib_destroy_cm_id <-rdma_destroy_id
<...>-92046 [004] .... 620537.627706: cm_destroy_id <-ib_destroy_cm_id
<...>-92046 [004] .... 620537.627707: ib_send_cm_dreq <-cm_destroy_id
<...>-92046 [004] d... 620537.627708: cm_alloc_msg <-ib_send_cm_dreq
<...>-92046 [004] d... 620537.627708: ib_create_ah <-cm_alloc_msg
<...>-92046 [004] d... 620537.627709: ib_create_send_mad <-cm_alloc_msg
<...>-92046 [004] d... 620537.627711: cm_form_tid.isra.59 <-ib_send_cm_dreq
<...>-92046 [004] d... 620537.627711: ib_post_send_mad <-ib_send_cm_dreq
<...>-92046 [004] d... 620537.627712: ib_is_mad_class_rmpp.part.20 <-ib_post_send_mad
<...>-92046 [004] d... 620537.627712: ib_send_mad <-ib_post_send_mad
<...>-92046 [004] d... 620537.627717: ib_get_cached_gid <-qedr_gsi_build_packet
<...>-92046 [004] d... 620537.627719: ib_ud_header_init <-qedr_gsi_build_packet
<...>-92046 [004] d... 620537.627720: ib_ud_header_pack <-qedr_gsi_build_packet
<...>-92046 [004] d... 620537.627720: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620537.627721: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620537.627722: ib_pack <-ib_ud_header_pack
<...>-92046 [004] d... 620537.627722: ib_pack <-ib_ud_header_pack
<idle>-0 [001] ..s. 620537.627772: ib_cq_completion_workqueue <-qedr_ll2_tx_cb
<...>-84717 [001] .... 620537.627781: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620537.627784: ib_mad_send_done <-__ib_process_cq
<...>-84717 [001] .... 620537.627786: ib_mad_complete_send_wr <-ib_mad_send_done
<...>-92046 [004] d... 620537.628786: ib_cancel_mad <-cm_destroy_id
<...>-92046 [004] d... 620537.628787: ib_modify_mad <-ib_cancel_mad
<...>-92046 [004] d... 620537.628793: cm_enter_timewait <-cm_destroy_id
<...>-92046 [004] d... 620537.628793: ib_get_client_data <-cm_enter_timewait
<...>-58578 [002] .... 620537.628835: cm_send_handler <-timeout_sends
<...>-58578 [002] .... 620537.628837: cm_free_msg <-cm_send_handler
<...>-58578 [002] .... 620537.628838: ib_destroy_ah <-cm_free_msg
<...>-58578 [002] .... 620537.628840: ib_free_send_mad <-cm_free_msg
<idle>-0 [001] ..s. 620537.629766: ib_cq_completion_workqueue <-qedr_ll2_rx_cb
<...>-84717 [001] .... 620537.629774: ib_cq_poll_work <-process_one_work
<...>-84717 [001] .... 620537.629776: ib_mad_recv_done <-__ib_process_cq
<...>-84717 [001] d... 620537.629779: ib_response_mad <-find_mad_agent
<...>-84717 [001] .... 620537.629780: ib_response_mad <-ib_mad_recv_done
<...>-84717 [001] .... 620537.629780: cm_recv_handler <-ib_mad_recv_done
<...>-84717 [001] .... 620537.631365: ib_mad_post_receive_mads <-ib_mad_recv_done
kworker/1:1-541 [001] .... 620537.631373: cm_work_handler <-process_one_work
kworker/1:1-541 [001] .... 620537.631376: cm_acquire_id <-cm_work_handler
kworker/1:1-541 [001] d... 620537.631376: cm_get_id <-cm_acquire_id
kworker/1:1-541 [001] .... 620537.633840: cm_free_work <-cm_work_handler
kworker/1:1-541 [001] .... 620537.633841: ib_free_recv_mad <-cm_free_work
<...>-92046 [004] .... 620537.633858: ib_uverbs_write <-vfs_write
<...>-92046 [004] .... 620537.633859: ib_uverbs_dealloc_pd <-ib_uverbs_write
<...>-92046 [004] .... 620537.634494: ib_destroy_cm_id <-rdma_destroy_id
<...>-92046 [004] .... 620537.634494: cm_destroy_id <-ib_destroy_cm_id
<...>-92046 [004] .... 620537.634496: ib_destroy_cm_id <-rdma_destroy_id
<...>-92046 [004] .... 620537.634496: cm_destroy_id <-ib_destroy_cm_id
<...>-92046 [004] .... 620537.634573: ib_uverbs_close <-__fput
<...>-92046 [004] .... 620537.634573: ib_uverbs_cleanup_ucontext <-ib_uverbs_close
<...>-92046 [004] .... 620537.634576: ib_uverbs_event_close <-__fput
<...>-92046 [004] .... 620537.634577: ib_unregister_event_handler <-ib_uverbs_event_close
<...>-92046 [004] .... 620537.634577: ib_uverbs_release_file <-ib_uverbs_event_close
kworker/4:1-557 [004] .... 620539.682427: cm_work_handler <-process_one_work
kworker/4:1-557 [004] .... 620539.683272: cm_acquire_id <-cm_work_handler
kworker/4:1-557 [004] d... 620539.683273: cm_get_id <-cm_acquire_id
kworker/4:1-557 [004] .... 620539.683273: cm_free_work <-cm_work_handler
