On 5/18/21 2:37 PM, John Garry wrote: > On 18/05/2021 12:54, Hannes Reinecke wrote: >>> In yanhui's virt workload(qemu, libaio, dio, high queue depth, single >>> job), the patch can improve throughput much(>50%) when running >>> sequential write(dio, libaio, 16 jobs) to XFS. And it is observed that >>> IO merge is recovered to level of disabling host tags. >>> >> I've found a good testcase for this by using null_blk: >> >> modprobe null_blk submit_queues=32 queue_mode=2 irqmode=0 bs=4096 >> hw_queue_depth=2048 completion_nsec=1 nr_devices=4 shared_tags=1 >> shared_tag_bitmap=1 >> >> and using a simple fio job with libaio and rw=read and numjobs=32 will >> do the trick: >> >> [nullb] >> rw=read >> size=16g >> ioengine=libaio >> direct=1 >> buffered=0 >> group_reporting=1 >> bs=4096 >> numjobs=32 >> >> (of course, the 'numjobs' and 'submit_queues' parameter would need to be >> adjusted to your machine). >> Omitting the 'shared_tag_bitmap' module parameter would yield around 12M >> iops; adding it would see a rather dramatic drop to 172k iops. >> With this patchset it's back to the expected iops value. > > I will consider running this test myself, but do you mean that you get > ~12M with shared_tag_bitmap=1 and this patch? > > I would always expect shared_tag_bitmap=1 to give a drop for null_blk, > as we move from per-submit queue driver tag to all submit queues sharing > the same driver tagset. > There is a slight drop (less than 1M iops), but that is about to be expected; however, as there is a rather high fluctuation in the numbers I didn't include it. But nothing as dramatic as without this patchset :-) > And I am not sure if you are fixing the IO sched from default. > No, I don't; I'm using whatever it's there (ie mq-deadline). Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@xxxxxxx +49 911 74053 688 SUSE Software Solutions Germany GmbH, 90409 Nürnberg GF: F. Imendörffer, HRB 36809 (AG Nürnberg)
