Re: confusion about nr of pending I/O requests

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




> Il giorno 18 dic 2018, alle ore 19:50, Jens Axboe <axboe@xxxxxxxxx> ha scritto:
> 
> On 12/18/18 5:45 AM, Paolo Valente wrote:
>> Hi Jens,
>> sorry for the following silly question, but maybe you can solve very
>> quickly a doubt for which I'd spend much more time investigating.
>> 
>> While doing some tests with scsi_debug, I've just seen that (at least)
>> with direct I/O, the maximum number of pending I/O requests (at least
>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of
>> the drive and not to
>> /sys/block/<dev>/queue/nr_requests
>> 
>> For example, after:
>> 
>> sudo modprobe scsi_debug max_queue=4
>> 
>> and with fio executed as follows:
>> 
>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20
>> 
>> I get this periodic trace, where four insertions are followed by four
>> completions, and so on, till the end of the I/O.  This trace is taken
>> with none, but the result is the same with bfq.
>> 
>>             fio-20275 [001] d...  7560.655213:   8,48   I   R 281088 + 8 [fio]
>>             fio-20275 [001] d...  7560.655288:   8,48   I   R 281096 + 8 [fio]
>>             fio-20275 [001] d...  7560.655311:   8,48   I   R 281104 + 8 [fio]
>>             fio-20275 [001] d...  7560.655331:   8,48   I   R 281112 + 8 [fio]
>>          <idle>-0     [001] d.h.  7560.749868:   8,48   C   R 281088 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.749912:   8,48   C   R 281096 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.749928:   8,48   C   R 281104 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.749934:   8,48   C   R 281112 + 8 [0]
>>             fio-20275 [001] d...  7560.750023:   8,48   I   R 281120 + 8 [fio]
>>             fio-20275 [001] d...  7560.750196:   8,48   I   R 281128 + 8 [fio]
>>             fio-20275 [001] d...  7560.750229:   8,48   I   R 281136 + 8 [fio]
>>             fio-20275 [001] d...  7560.750250:   8,48   I   R 281144 + 8 [fio]
>>          <idle>-0     [001] d.h.  7560.842510:   8,48   C   R 281120 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.842551:   8,48   C   R 281128 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.842556:   8,48   C   R 281136 + 8 [0]
>>          <idle>-0     [001] dNh.  7560.842562:   8,48   C   R 281144 + 8 [0]
>> 
>> Shouldn't the total number of pending requests reach
>> /sys/block/<dev>/queue/nr_requests ?
>> 
>> The latter is of course equal to 8.
> 
> With a scheduler, the depth is what the scheduler provides. You cannot
> exceed the hardware queue depth in any situation. You just have 8
> requests available for scheduling, with a max of 4 being inflight on
> the device side.
> 
> If both were 4, for instance, then you would have nothing to schedule
> with, as all of them could reside on the hardware side. That's why
> the scheduler defaults to twice the hardware queue depth.
> 

That's exactly what I expected, thanks.  But not what happened.  Let
me add also dispatch lines to my filtered-trace snippet.  The
repetitive pattern becomes:

             fio-5180  [001] d...   786.931956:   8,48   I  WS 333824 + 1024 [fio]
             fio-5180  [001] d...   786.932010:   8,48   D  WS 333824 + 1024 [fio]
             fio-5180  [001] d...   786.932137:   8,48   I  WS 334848 + 1024 [fio]
             fio-5180  [001] d...   786.932160:   8,48   D  WS 334848 + 1024 [fio]
             fio-5180  [001] d...   786.932318:   8,48   I  WS 335872 + 1024 [fio]
             fio-5180  [001] d...   786.932354:   8,48   D  WS 335872 + 1024 [fio]
             fio-5180  [001] d...   786.932467:   8,48   I  WS 336896 + 1024 [fio]
             fio-5180  [001] d...   786.932489:   8,48   D  WS 336896 + 1024 [fio]
          <idle>-0     [001] d.h.   787.023945:   8,48   C  WS 333824 + 1024 [0]
          <idle>-0     [001] d.h.   787.023978:   8,48   C  WS 334848 + 1024 [0]
          <idle>-0     [001] d.h.   787.024080:   8,48   C  WS 335872 + 1024 [0]
          <idle>-0     [001] d.h.   787.024237:   8,48   C  WS 336896 + 1024 [0]

So, after the four dispatches, there are 0 requests in the scheduler,
and 4 requests inflight.  But *no* new request is inserted into the
scheduler before *all* four inflight requests are completed.

So the total number of requests available seems 4 and not 8.

Am I missing something?

Thanks,
Paolo

> -- 
> Jens Axboe





[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux