> Il giorno 18 dic 2018, alle ore 19:50, Jens Axboe <axboe@xxxxxxxxx> ha scritto:
>
> On 12/18/18 5:45 AM, Paolo Valente wrote:
>> Hi Jens,
>> sorry for the following silly question, but maybe you can solve very
>> quickly a doubt for which I'd spend much more time investigating.
>>
>> While doing some tests with scsi_debug, I've just seen that (at least)
>> with direct I/O, the maximum number of pending I/O requests (at least
>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of
>> the drive and not to
>> /sys/block/<dev>/queue/nr_requests
>>
>> For example, after:
>>
>> sudo modprobe scsi_debug max_queue=4
>>
>> and with fio executed as follows:
>>
>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20
>>
>> I get this periodic trace, where four insertions are followed by four
>> completions, and so on, till the end of the I/O. This trace is taken
>> with none, but the result is the same with bfq.
>>
>> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio]
>> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio]
>> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio]
>> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio]
>> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0]
>> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0]
>> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0]
>> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0]
>> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio]
>> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio]
>> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio]
>> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio]
>> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0]
>> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0]
>> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0]
>> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0]
>>
>> Shouldn't the total number of pending requests reach
>> /sys/block/<dev>/queue/nr_requests ?
>>
>> The latter is of course equal to 8.
>
> With a scheduler, the depth is what the scheduler provides. You cannot
> exceed the hardware queue depth in any situation. You just have 8
> requests available for scheduling, with a max of 4 being inflight on
> the device side.
>
> If both were 4, for instance, then you would have nothing to schedule
> with, as all of them could reside on the hardware side. That's why
> the scheduler defaults to twice the hardware queue depth.
>
That's exactly what I expected, thanks. But not what happened. Let
me add also dispatch lines to my filtered-trace snippet. The
repetitive pattern becomes:
fio-5180 [001] d... 786.931956: 8,48 I WS 333824 + 1024 [fio]
fio-5180 [001] d... 786.932010: 8,48 D WS 333824 + 1024 [fio]
fio-5180 [001] d... 786.932137: 8,48 I WS 334848 + 1024 [fio]
fio-5180 [001] d... 786.932160: 8,48 D WS 334848 + 1024 [fio]
fio-5180 [001] d... 786.932318: 8,48 I WS 335872 + 1024 [fio]
fio-5180 [001] d... 786.932354: 8,48 D WS 335872 + 1024 [fio]
fio-5180 [001] d... 786.932467: 8,48 I WS 336896 + 1024 [fio]
fio-5180 [001] d... 786.932489: 8,48 D WS 336896 + 1024 [fio]
<idle>-0 [001] d.h. 787.023945: 8,48 C WS 333824 + 1024 [0]
<idle>-0 [001] d.h. 787.023978: 8,48 C WS 334848 + 1024 [0]
<idle>-0 [001] d.h. 787.024080: 8,48 C WS 335872 + 1024 [0]
<idle>-0 [001] d.h. 787.024237: 8,48 C WS 336896 + 1024 [0]
So, after the four dispatches, there are 0 requests in the scheduler,
and 4 requests inflight. But *no* new request is inserted into the
scheduler before *all* four inflight requests are completed.
So the total number of requests available seems 4 and not 8.
Am I missing something?
Thanks,
Paolo
> --
> Jens Axboe
