On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <axboe@xxxxxxxxx> wrote: > > On 12/18/18 5:45 AM, Paolo Valente wrote: > > Hi Jens, > > sorry for the following silly question, but maybe you can solve very > > quickly a doubt for which I'd spend much more time investigating. > > > > While doing some tests with scsi_debug, I've just seen that (at least) > > with direct I/O, the maximum number of pending I/O requests (at least > > in the I/O schedulers) is equal, unexpectedly, to the queue depth of > > the drive and not to > > /sys/block/<dev>/queue/nr_requests > > > > For example, after: > > > > sudo modprobe scsi_debug max_queue=4 > > > > and with fio executed as follows: > > > > job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 > > > > I get this periodic trace, where four insertions are followed by four > > completions, and so on, till the end of the I/O. This trace is taken > > with none, but the result is the same with bfq. > > > > fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] > > fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] > > fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] > > fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] > > <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] > > <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] > > <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] > > <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] > > fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] > > fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] > > fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] > > fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] > > <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] > > <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] > > <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] > > <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] > > > > Shouldn't the total number of pending requests reach > > /sys/block/<dev>/queue/nr_requests ? > > > > The latter is of course equal to 8. > > With a scheduler, the depth is what the scheduler provides. You cannot > exceed the hardware queue depth in any situation. You just have 8 > requests available for scheduling, with a max of 4 being inflight on > the device side. > > If both were 4, for instance, then you would have nothing to schedule > with, as all of them could reside on the hardware side. That's why > the scheduler defaults to twice the hardware queue depth. The default twice to hw queue depth might not be reasonable for multi LUN. Maybe it should be set twice of sdev->queue_depth for SCSI or hw queue depth/hctx->nr_active. But either way may become complicated because both can be adjusted runtime. Thanks, Ming Lei
