On 9/2/2021 3:21 PM, Stefan Hajnoczi wrote:
On Wed, Sep 01, 2021 at 04:14:34PM +0300, Max Gurtovoy wrote:
No need to pre-allocate a big buffer for the IO SGL anymore. If a device
has lots of deep queues, preallocation for the sg list can consume
substantial amounts of memory. For HW virtio-blk device, nr_hw_queues
can be 64 or 128 and each queue's depth might be 128. This means the
resulting preallocation for the data SGLs is big.
Switch to runtime allocation for SGL for lists longer than 2 entries.
This is the approach used by NVMe drivers so it should be reasonable for
virtio block as well. Runtime SGL allocation has always been the case
for the legacy I/O path so this is nothing new.
The preallocated small SGL depends on SG_CHAIN so if the ARCH doesn't
support SG_CHAIN, use only runtime allocation for the SGL.
Re-organize the setup of the IO request to fit the new sg chain
mechanism.
No performance degradation was seen (fio libaio engine with 16 jobs and
128 iodepth):
IO size IOPs Rand Read (before/after) IOPs Rand Write (before/after)
-------- --------------------------------- ----------------------------------
512B 318K/316K 329K/325K
4KB 323K/321K 353K/349K
16KB 199K/208K 250K/275K
128KB 36K/36.1K 39.2K/41.7K
Signed-off-by: Max Gurtovoy <mgurtovoy@xxxxxxxxxx>
Reviewed-by: Israel Rukshin <israelr@xxxxxxxxxx>
---
changes from V2:
- initialize vbr->out_hdr.sector during virtblk_setup_cmd
changes from V1:
- Kconfig update (from Christoph)
- Re-order cmd setup (from Christoph)
- use flexible sg pointer in the cmd (from Christoph)
- added perf numbers to commit msg (from Feng Li)
---
drivers/block/Kconfig | 1 +
drivers/block/virtio_blk.c | 155 +++++++++++++++++++++++--------------
2 files changed, 100 insertions(+), 56 deletions(-)
Hi Max,
I can run benchmark to give everyone more confidence about this change.
Should I test this version or are you still planning to make code
changes?
Yes you can test v3.
Thanks,
Max.
Stefan
