Add documentation to describe the various scenarios that the scsi_cmnd may go through in its life time in the mid level driver - aborts, failures, retries, error handling etc. The documentation has lots of details including examples. --- Hello, I have been reading the SCSI for the past few weeks and decided to turn all my notes from past few weeks into a documentation, in the hope that this may be helpful to others. I'd appreciate if you could please review & provide your valuable feedback. Thanks, Rajat Documentation/scsi/life_of_a_scsi_cmnd.txt | 667 +++++++++++++++++++++++++++++ 1 file changed, 667 insertions(+) create mode 100644 Documentation/scsi/life_of_a_scsi_cmnd.txt diff --git a/Documentation/scsi/life_of_a_scsi_cmnd.txt b/Documentation/scsi/life_of_a_scsi_cmnd.txt new file mode 100644 index 0000000..b09b2a2 --- /dev/null +++ b/Documentation/scsi/life_of_a_scsi_cmnd.txt @@ -0,0 +1,667 @@ + ================================== + Life of a SCSI Command (scsi_cmnd) + ================================== + + Rajat Jain <rajatja@xxxxxxxxxx> on 12-May-2015 + +(This document roughly matches the Linux kernel 4.0) + +This documents describes the various phases of a SCSI command (struct scsi_cmnd) +lifecycle, as it flows though different parts of the SCSI mid level driver. It +describes under what conditions and how a scsi_cmnd may be aborted, or retried, +or scheduled for error handling, and how is it recovered, and in general how a +block request is handled by the SCSI mid level driver. It goes into detail about +what functions get called and the purpose for each one of them etc. + +To help explain with an example, it takes example of a scsi_cmnd that goes +through it all - timeout, abort, error handling, retry (also results in +CHECK_CONDITION and gets sense info). The last section traces the path taken by +this example scsi_cmnd in its lifetime. + +TABLE OF CONTENTS + +[1] Lifecycle of a scsi_cmnd +[2] How does a scsi_cmnd get queued to the LLD for processing? +[3] How does a scsi_cmnd complete? + [3.1] Command completing via scsi_softirq_done() + [3.2] Command completing via scsi_times_out()$ +[4] SCSI Error Handling + [4.1] How did we Get here? + [4.2] When does Error Handling actually run? + [4.3] SCSI Error Handler thread +[5] SCSI Commands can be "hijacked" +[6] SCSI Command Aborts + [6.1] When would mid level try to abort a command? + [6.2] How SCSI command abort works? + [6.3] Aborts can fail too +[7] SCSI command Retries + [7.1] When would mid level retry a command? + [7.2] Eligibility criteria for Retry +[8] Example: Following a scsi_cmnd (that results in CHECK_CONDITION) + [8.1] High level view of path taken by example scsi_cmnd + [8.2] Actual Path taken +[9] References + +1. Lifecycle of a scsi_cmnd + ======================== + SCSI Mid level interfaces with the block layer just like any other block + driver. For each block device that SCSI ML adds to the system, it indicates + a bunch of functions to serve the corresponding request queue. + + The following functions are relevant to the scsi_cmnd in its lifetime. Note + that depending on the situations, it may not go thourgh some of these + stages, or may have to go through some stages multiple times. + + scsi_prep_fn() + is called by the blocklayer to prepare the request. This + function actually allocates a new scsi_cmnd for the request (from + scsi_host->cmd_pool) and sets it up. This is where a scsi_smnd is "born". + Note, a new scsi_cmnd is allocated only if the blk req did not already have + one associated with it (req->special != NULL). A req may already have a + scsi_cmnd if the req was tried by SCSI earlier, and it resulted in a + decision to retry later (and hence req was put back on the queue). + + scsi_request_fn() + is the actual function to serve the request queue. It basically checks + whether the host is ready for new commands, and if so, it submits it to the + LLD: + scsi_request_fn() + ->scsi_dispatch_cmd() + ->hostt->queue_command() + In case a scsi_cmnd could not be queued to LLD for some reason, the req + is put back on the original request queue (for retry later). + + scsi_softirq_done() + is the handler that gets called once the LLD indicates command completed. + scsi_done() + ->blk_complete_request() + ->causes softirq + ->blk_done_softirq() + ->scsi_softirq_done() + The most important goal of this function is to determine the course of + further action for this req (based on the scsi_cmnd->result and sense data + if present), and take that course. The options could be to finish off the + request to block layer, requeue it to block layer, or schedule it for error + handling (if that is deemed necessary). This is discussed in much detail + later. + + scsi_times_out() + is the function that gets called if the LLD does not respond with the + result of a scsi_cmnd for a long time, and a time out happens. It tries + to see if the situation can be fixed by LLD timeout handlers (if available) + or aborting the commands. If not, it schedules the commands for EH + (discussed at length later). + + scsi_unprep_fn() + is the function that gets called to unprepare the request. It is supposed + to undo whatever scsi_prep_fn() does. + +2. How does a scsi_cmnd get queued to the LLD for processing? + ========================================================== + The submission part is very simple. Once the scsi_request_fn() gets called + for a block request and it picks up a new block request via + blk_peek_request(), the scsi_cmnd has already been setup and is ready to be + sent to the LLD: + scsi_request_fn() + ->scsi_dispatch_cmd() + ->hostt->queue_command() + +3. How does a scsi_cmnd complete? + ============================== + Once a scsi_cmnd is submitted to the LLD, there are only 2 ways it can get + completed: + + a. Either the LLD responds in time. + (i.e. resulting in scsi_softirq_done() for the command) + + b. Or, the LLD does not respond in time and a timeout out occurred + (i.e. resulting in scsi_times_out() for the command) + + We discuss both these cases below. + + Note 1: There may be scsi_cmnd(s) that are re-tried. But completion of a + re-tried scsi_cmnd is not any different than the completion of a new + scsi_cmnd. Thus irrespective of retries, the scsi_cmnds will always end up + in using one of the above 2 scenarios. + + Note 2: A scsi_cmnd may be "highjacked" during error handling in + scsi_send_eh_cmnd(), to send one of the EH commands (TUR / STU / + REQUEST_SENSE). However, the completion of these EH commands does not land up + in the above two scenarios. This is the only exception. Once the scsi_cmnd is + "un-hijacked", the result of this original scsi_cmnd will still go through + the same 2 scenarios. + +3.1 Command completing via scsi_softirq_done() + ========================================== + This is the case when the LLD responded in time i.e. completed the command. + Note that here "completed" does not mean that the command was successfully + completed. In fact it could have been the case, that the SCSI host hardware + may have failed without even accepting the command. However, the fact that + scsi_softir_done() was called, indicates that there is a "result" available + in a timely fashion. And we'll have to examine this result in order to + decide the next course of action. + + scsi_softirq_done() + | + +---> scsi_decide_disposition() + | Takes a look at the scsi_cmnd->result and sense data to determine + | what is the best course of action to take. While reading this + | function code, one should not confuse SUCCESS as meaning the command + | was successful, or FAILED to mean the command failed etc. The return + | value of this function merely indicates the course of action to take + | + +---> case SUCCESS: + | (Finish off the command to block layer. For e.g, the device may be + | offline, and hence complete the command - the block layer may retry + | on its own later, but that doesn't concern the SCSI ML) + | | + | +---> scsi_finish_command() + | | + | +---> scsi_io_completion() (*see note below) + | | + | +---> blk_finish_request() + | + +---> case RETRY/ADD_TO_MLQUEUE: + | (Requeue the command to request queue. For e.g. the device HW was + | busy, and thus SCSI ML knows that retrying may help) + | | + | +---> scsi_queue_insert() + | | + | +---> blk_requeue_request() + | + +---> case FAILED/default: + (Schedule the scsi_cmnd for EH. For e.g. there was a bus error that + might need bus reset. Or we got CHECK_CONDITION and we need to issue + REQ_SENSE to get more info about the failure. etc) + | + +---> scsi_eh_scmd_add() + Add scsi_cmnd to the host EH queue + scsi_eh_wakeup() + + Note 3: + The scsi_io_completion() has a secondary logic similar to + scsi_decide_disposition() in that it also looks at result and sense data + and figures out what to do with request. It makes similar choices on the + course of action to take. There is a special case in this function that + involves "unprepping" a scsi_cmnd before requeuing it, and we'll discuss + it in sections below. + +3.2 Command completing via scsi_times_out() + ======================================= + This happens when the LLD does not repond in time, the block layer times + out, and as a result calls the timeout function for the request queue for + the SCSI device in question. + + scsi_times_out() + | + +---> scsi_transport_template->eh_timed_out() - Successful? If not... + | (Gives transportt a chance to deal with it) + | + +---> scsi_host_template->eh_timed_out() - Successful? If not... + | (Gives hostt a chance to deal with it) + | + +---> scsi_abort_command() - Successful? If not... + | (Schedule an ABORT of the scsi_cmnd. The abort handler will also + | requeue it if needed) + | + +---> scsi_eh_scmd_add() + (Schedule the scsi_cmnd for EH. This'll definitely work. Because if it + doesn't work, the EH handler will mark the device as offline, which + counts as a good fix :-)) + +4. SCSI Error Handling + =================== + + SCSI Error handling should be thought of the action the mid level decides to + take when it knows that merely retrying a request may not help, and it needs + to do something else (possibly disruptive) in order to fix the issue. For + e.g. a stalled host may require a host reset, and only after that a retry of + the request may complete. + + Note 4: + (Random thoughts): Contrast the "Error Handling" with "Retries". A Retry + is a normal thing to do, when the mid level believes that it has seen an + error which is transient in nature, and will go away on its own without + explicitly doing anything. Thus a retry of a request again makes sense in + this case. (On the other hand a cmnd is scheduled for EH, when it knows + that it needs to do "something" before a retrying a cmnd can give good + results). + + Note 5: + The SCSI mid level maintains a (per-host) list of all the scsi_cmnd(s) + that have been scheduled for EH at that host using scsi_host->eh_cmd_q. + This is the list that gets processed by the EH thread, when it runs. + +4.1 How did we Get here? + -------------------- + + A scsi_cmnd could be marked for EH in the following cases: + + * The command "error completed" i.e. scsi_decide_disposition() returned + FAILED or something that indicates a failure that requires some sort of + error recovery. E.g. device hardware failed, or we have a CHECK_CONDITION. + scsi_softirq_done() + ->scsi_decide_disposition = FAILED + ->scsi_eh_scmd_add() + + * A scsi_cmnd timed out, and attempt to abort it fails. + scsi_times_out() + ->scsi_abort_command() != SUCCESS + ->scsi_eh_scmd_add() + +4.2 When does Error Handling actually run? + ------------------------------------- + + A SCSI error handler thread is scheduled whenever there is a scsi_smnd that + is marked for EH (inserted in the Scsi_Host->eh_cmd_q). Once a scsi_cmnd is + marked for EH, the ML does not accept any more scsi_cmnds for that + particular Scsi_Host. However, the EH thread does not actually run until all + the pending IOs to the LLD for that particular Scsi_Host have either + completed or failed. In other words, the only commands pending at the LLD + for that host are the ones that need EH (host_busy == host_failed). + + The idea is to quiesce the bus, so that EH thread can recover the devices, + as it may require to reset different components in order to do its job. + +4.3 SCSI Error Handler thread + ------------------------- + + scsi_error_handler() + | + +---> transportt->eh_strategy_handler() if exists, else... + | (Use transportt's own error recovery handler, if available) + | + +---> scsi_unjam_host() + | (The SCSI ML error handler described below. Also described in + | Documentation/scsi/scsi_eh.txt. Basic goal is to do whatever + | needs to recover from the current error condition. And requeue the + | eligible commands after recovery) + | + +---> scsi_restart_operations() + (Restart the operations of the SCSI request queue) + | + +---> scsi_run_host_queues() + | + +---> scsi_run_queue() + | + +---> blk_run_queue() + + scsi_unjam_host() + ----------------- + The idea is to create 2 lists: work_q, done_q. + Initially, work_q = <All EH scsi cmds>, done_q = NULL + And then error handle all the requests in work_q by taking sequentially + higher severity action items that may recover the cmnd or device. Keep + moving the requests from work_q to done_q and in the end finish them all + in one go rather than individually finishing them up. + + scsi_unjam_host() + | + +--> Create 2 lists: work_q, done_q + | work_q = <All EH scsi cmds>, done_q = NULL + | + +--> scsi_eh_get_sense() - Are we done? if not... + | (For the commands that have CHECK_CONDITION, get sense_info) + | | + | +--> scsi_request_sense() + | | (Use scsi_send_eh_cmnd() to send a "hijacked" REQ_SENSE cmnd) + | | + | +--> scsi_decide_disposition() + | | + | +--> Arrange to finish the scsi_cmnd if SUCCESS (by setting + | retries=allowed) + | + +--> scsi_eh_abort_cmds() - Are we done? If not... + | (Abort the commands that had timed out) + | | + | +--> scsi_try_to_abort_cmd() + | | (Results in call to hostt->eh_abort_handler() which is responsible + | | making the LLD and the HW forget about the scsi_cmnd) + | | + | +--> scsi_eh_test_devices() + | (Test if the device is responding now by sending appropriate EH + | commands (STU / TEST_UNIT_READY). Again, sending these EH + | commands involves highjacking the original scsi_cmnd, and later + | restoring the context) + | + +--> scsi_eh_ready_devs() - Are we done? if not... + | (Take increasing order of higher severity actions in order to recover) + | | + | +--> scsi_eh_bus_device_reset() + | | (Reset the scsi_device. Results in call to + | | hostt->eh_device_reset_handler()) + | | + | +--> scsi_eh_target_reset() + | | (Reset the scsi_target. Results in call to + | | hostt->eh_target_reset_handler()) + | | + | +--> scsi_eh_bus_reset() + | | (Reset the scsi_device. Results in call to + | | hostt->eh_bus_reset_handler()) + | | + | +--> scsi_eh_host_reset() + | | (Reset the Scsi_Host. Results in call to + | | hostt->eh_host_reset_handler()) + | | + | +--> If nothing has worked - scsi_eh_offline_sdevs() + | (The device is not recoverable, put it offline) + | + +--> scsi_eh_flush_done_q() + (For all the EH commands on the done_q, either requeue them (via + scsi_queue_insert()) if eligible, or finish them up to block layer + (via scsi_finish_command()) + + Note 6: + At each recovery stage we test if we are done (using + scsi_eh_test_devices()), and take the next severity action only if needed. + + Note 7: + The error handler takes care that for multiple scsi_cmnds that can be + recovered by resetting the same component (e.g. same scsi_device), the + device is reset only once. + +5. SCSI Commands can be "hijacked" + =============================== + + As seen above, the EH thread may need to send some EH commands in order to + check the health and responsiveness of the SCSI device: + * TUR - Test Unit Ready + * STU - Start / Stop Unit + * REQUEST_SENSE - To get the Sense data in response to CHECK_CONDITION + + However instead of allocating and setting up a new scsi_cmnd for such + temporary purposes, the EH thread hijacks- the current scsi_cmnd that it is + trying to recover, in order to send the EH commands. This whole process is + done in scsi_send_eh_cmnd(). + + The scsi_send_eh_cmnd saves a context of the current command before hijacking + it, replaces the scsi_done ptr with its own before dipatching it to the LLD, + and restores the context later once it is done. The EH commands sent in this + manner are subject to the same problems of timeouts / abort failures / + completions - but they do not take the route taken by normal commands (i.e. + don't take the scsi_softirq_done() or scsi_times_out() route). Every + thing is handled within scsi_send_eh_cmnd(). This is discussed in following + sections. + +6. SCSI Command Aborts + =================== + + It refers to the scenario where the SCSI mid level wants to have the LLD + driver and the hardware below it forget everything about a scsi_cmnd that + was given to the LLD earlier. The most common reason is that the LLD failed + to respond in time. + +6.1 When would mid level try to abort a command? + -------------------------------------------- + The SCSI ML may try to abort a scsi_cmnd in the following conditions: + + 1. SCSI mid layer times out on a command, and tried to abort it. + scsi_times_out() + -> scsi_abort_command() + What happens if this abort fails? Schedule the command for EH. + + 2. The EH thread tried to abort all the pending commands while trying to + unjam a host. + scsi_unjam_host() + -> scsi_eh_abort_cmds() + + What happens if this abort fails? We move to higher severity recovery + steps (start resetting HW components etc) because that is likely to cause + both LLD and the HW forget aout those commands. + + 3. This is a nasty one. During error recovery, the EH thread may "hijack" + a scsi_cmnd to send a EH command (TUR/STU/REQ_SENSE) to LLD using + scsi_send_eh_cmnd(). If such a "hijacked" EH command times out, the SCSI + EH thread will try to abort it. + scsi_send_eh_cmnd() + -> scsi_abort_eh_cmnd() + -> scsi_try_to_abort_cmd() + + What happens if this abort fails? Similar to the previous case, the + scsi_abort_eh_cmnd() will try to take higher severity actions (reset bus + etc) but will not send EH commands such as TUR etc again in order to + verify if the devices started to respond. + +6.2 How SCSI command abort works? + ----------------------------- + Unlike EH command like TUR, the ABORT is not a SCSI command that mid layer + driver sends to LLD. The LLD provides an eh_abort_handler() function + pointer that is used to abort the command. It is up to the LLD to do + whatever is needed to abort the command. It may require to send some + proprietary command to the HW, or fiddle some bits, or do whatever magic + is necessary. + +6.3 Aborts can fail too + -------------------- + + As with other things, abort attempts can also fail. The SCSI mid layer does + the right thing in such situations as depicted in the section above. + + Note 8: + Once a block layer hands off a command to the SCSI subsystem, there is no + way currently for the block layer to cancel / abort a request. This needs + some work. + +7. SCSI command Retries + ==================== + + The SCSI mid level maintains no queues for the SCSI commands it is processing + (other than the EH command queue). Thus whenever the SCSI ML thinks it needs + to retry a command, it requeues the request back to the corresponding request + queue, so that the retries will be made "naturally" when the request function + picks up the next request for processing. + + When requing such requests back to the request queue, they are put at the + head so that they go before the other (existing) requests in that request + queue. + +7.1 When would mid level retry a command? + ------------------------------------- + + Following are the conditions that will cause a SCSI command to be retried + (by putting the blk request back at the request queue): + + 1. Mid layer times out on a scsi_cmnd, aborts it successfully, and requeues + it. + scsi_times_out() + -> scsi_abort_command() + -> schedules scmd_eh_abort_handler() + -> scsi_queue_insert() + -> blk_requeue_request() + + 2. EH thread, after recovering a host, requeues back all the scsi_cmnds that + are eligible for a retry: + scsi_error_handler() + -> scsi_unjam_host() + -> scsi_eh_flush_done_q() + -> scsi_queue_insert() + -> blk_requeue_request() + + 3. LLD completes the scsi_cmnd, and scsi_decide_disposition() looks at the + scsi_cmnd->result and thinks it needs to be retried (For e.g. because the + bus was busy). + scsi_softirq_done() + -> scsi_decide_disposition() returns NEEDS_RETRY + -> scsi_queue_insert() + -> blk_requeue_request() + + 4. In the scsi_request_fn(), the SCSI ML finds out that the host is busy and + the scsi_cmnd could not be sent to the LLD, hence it requeues the req + back on the queue. + scsi_request_fn() + -> case note_ready: + -> blk_requeue_request() + + 5. scsi_finish_command() that is called from a variety of places to finish + off a request to the block level. However, it calls scsi_io_completion() + that may look at the request and decide to retry it (if it qualifies). + scsi_finish_command() + -> scsi_io_completion() + -> __scsi_queue_insert() + -> blk_requeue_request() + + Note 9: + The case 5 above has a very special case. There may be some cases where + the scsi_io_completion() decides that a blk request has to be retried, + however the scsi_cmnd for this req should be relased and instead a new + scsi_cmnd should be allocated and used for this request at the next + retry. This can be the case for e.g. if it sees an ILLEGAL REQUEST as a + response to a READ10 command, and thinks that it may be because the + device supports only READ6. Thus it may make sense to switch to READ6 + (hence a new scsi_cmnd) at the time of next retry. + +7.2 Eligibility criteria for Retry + ------------------------------ + + Note that SCSI mid level always checks for retry eligibility before it goes + ahead and requeues the command for retries. The eligibility criteria for a + scsi_cmnd includes (some of these may not apply in all situations described + above): + + * retries < allowed (Num of retries should be less than allowed retries) + * no more than host->eh_deadline jiffies spent in EH. + * scsi_noretry_cmd() should return 0 for the command. + * scsi_device must be online + * req->timeout must not have expired + * etc. + +8. Example: Following a scsi_cmnd + ============================== + +8.1 High level view of path taken by example scsi_cmnd + -------------------------------------------------- + We take the example of a block request that for example wants to read a + block off a scsi disk, how ever the LBA address is out of range for the + current device (hypothetically). The ML submits it to LLD, but the HW takes + the command and chokes on it (again hypothetically to trace through the + abort sequence). So the timeout happens and the ML aborts the + command, and requeues it. In the next run, the LLD completes the command + with CHECK_CONDITION. We assume that the SCSI host does not automatically + get the sense info. The ML schedules the cmnd for EH. The EH thread sends + the REQUEST_SENSE to get sense info ILLEGAL_REQUEST, and based on it + completes the request to the block layer. + +8.2 Actual Path taken + ----------------- + + Dispatched: + + scsi_request_fn() + | + +---> blk_peek_request() + | | + | +---> scsi_prep_fn() + | (Allocate and setup scsi_cmnd) + | + +---> scsi_dispatch_cmd() + | + +---> hostt->queue_command() + + Times out: + + scsi_times_out() + | + +---> scsi_abort_command() - returns SUCCESS + | + +---> queue_delayed_work(abort_work) + + Abort Handler: + + scmd_eh_abort_handler() + | + +---> scsi_try_to_abort_cmd() - returns SUCCESS + | | + | +---> hostt->eh_abort_handler() + | + +---> scsi_queue_insert() + | + +---> __scsi_queue_insert() + | + +---> blk_requeue_request() + (the req is requeued, with req->special pointing + to scsi_cmnd) + + Request picked up again: + + scsi_request_fn() + | + +---> blk_peek_request() + | (req->cmd_flags has REQ_DONTPREP set, so does not call + | scsi_prep_fn() again) + | + +---> scsi_dispatch_cmd() + | + +---> hostt->queue_command() + + Command is completed with a CHECK_CONDITION: + + scsi_softirq_done() + | + +---> scsi_decide_disposition() + | (Sees the CHECK_CONDITION) + | | + | +---> scsi_check_sense() - returns FAILED + | | + | +---> scsi_command_normalize_sense() + | (Fails to find a valid sense data) + | + +---> case FAILED: + | + +---> scsi_eh_scmd_add() + Add scsi_cmnd to the host EH queue + | + +---> scsi_eh_wakeup() + + The SCSI Error handler thread runs to get the sense info, and completes the + request once it is done. + + scsi_error_handler() + | + +---> scsi_unjam_host() + | + +---> scsi_eh_get_sense() + | | + | +---> scsi_request_sense() + | | | + | | +---> scsi_send_eh_cmnd() + | | (Highjacks the smnd to send EH command) + | | | + | | +--> scsi_eh_prep_cmnd() + | | | (save context of the existing scsi_cmndi, + | | | allocates a sense buffer, and sets up the + | | | scsi_cmnd for REQUEST_SENSE) + | | | + | | +--> hostt->queuecommand(), and then wait... + | | | (gets the sense data for the cmnd) + | | | + | | +--> scsi_eh_completed_normally() - returns SUCCESS + | | | + | | +--> scsi_eh_restore_cmnd() + | | (restores the context of original scsi_cmnd) + | | + | +---> scsi_decide_disposition() - returns SUCCESS + | | (This time can see the sense info) + | | + | +---> Set scmd->retries = scmd->allowed (to avoid retries) + | | + | +---> scsi_eh_finish_cmd() + | (Puts the scsi_cmnd on the done_q) + | + +---> scsi_eh_flush_done_q() + (Sees that scsi_cmnd is not eligible for retries) + | + +---> scsi_finish_command() + | + +---> scsi_io_completion() + | + +---> scsi_end_request() + | + +---> scsi_put_command() + (Releases the scsi_cmnd) + +9. References + ========== + The following are excellent sources of references: + Documentation/scsi/scsi_eh.txt + http://events.linuxfoundation.org/sites/events/files/slides/SCSI-EH.pdf +-- -- 2.2.0.rc0.207.ga3a616c -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
