On 4/15/2020 8:51 AM, Hannes Reinecke wrote:
Relating to my previous comments: Can you elaborate about the lifetime
rules to the node? It looks as if any access to the node is being
guarded by the main efc->lock.
Which, taken to extremes, would meant that one has to hold that lock for
_any_ access to the node; that would include I/O processing etc, too.
I can't really see how that would scale to the IOPS you are pursuing, so
clearly that will not happen.
So how _do_ you ensure the validity of a node if the lock is not held?
Cheers,
Hannes
This lock is defined for single EQ/RQ. For multi EQ/RQ which we will add
in future, will have more granular locks and performance improvements.
All the State Machine Events are triggered from the EQ processing thread
except vport creation and deletion.
Locking:
Global lock per efc port "efc->lock"
Domain:
efc_domain_cb --> (All the events are executed under efc lock)
Async link up notification(EFC_HW_DOMAIN_FOUND)
Alloc Domain, Start discovery
Alloc Sport
Async link down notification (EFC_HW_DOMAIN_LOST)
Hold frames. (New frames will be moved to pending)
Post Sport Shutdown.
Post All Nodes Shutdown.
Sport:
efc_lport_cb --> (All the events are executed under efc lock)
HW sport registration (Mbox command responses) to complete sport
allocation.
Vport :
efc_sport_vport_new() --> New Vport
efc_vport_sport_alloc() (Protected by efc lock)
Alloc Sport, and start Sport SM
efc_sport_vport_del() --> Delete Vport
Post shutdown event to the Sport. (Protected by efc lock)
Node:
efc_remote_node_cb() --> (All the events are executed under efc lock)
HW node registration (Mbox command responses) to complete node
allocation.
Node lifeCycle:
efc_domain_dispatch_frame:-
EFC lock held
efc_node_find() IF not found, Create one.
when PLOGI is received, Register with hardware
Upon PRLI completion, setup a new session with LIO.
IO path:
Find sport and node under EFC lock.
If node can process commands, start IO processing.
Each IO is added to the node->active_ios.
Node deletion: (RSCN notification or LOGO received ..)
EFC lock held
Disable receiving fcp commands. (node->fcp_enabled)
Disable IO allocation for this node.
Remove LIO session.
Deregister node with HW
Waits for all active_ios to be completed/freed.
Purge pending IOs.
Free the Node.
In IO path, EFC lock is acquired to find the sport and node, release the
EFC lock and continue with IO allocation and processing. Note: There is
still an unsafe area where we check for 'node->hold_frames" without the
lock. Node is assumed to be kept alive until all the IOs under the node
are freed. Adding the refcounting will remove this assumption.
-- james
