On Fri, Mar 3, 2023 at 1:18 AM Jonathan Derrick <jonathan.derrick@xxxxxxxxx> wrote: > > > > On 3/2/2023 2:05 AM, Xiao Ni wrote: > > On Thu, Mar 2, 2023 at 1:57 AM Jonathan Derrick > > <jonathan.derrick@xxxxxxxxx> wrote: > >> > >> > >> > >> On 3/1/2023 5:36 AM, Xiao Ni wrote: > >>> On Wed, Mar 1, 2023 at 7:10 AM Jonathan Derrick > >>> <jonathan.derrick@xxxxxxxxx> wrote: > >>>> > >>>> Hi Xiao > >>>> > >>>> On 2/26/2023 6:56 PM, Xiao Ni wrote: > >>>>> Hi Jonathan > >>>>> > >>>>> I did a test in my environment, but I didn't see such a big > >>>>> performance difference. > >>>>> > >>>>> The first environment: > >>>>> All nvme devices have 512 logical size, 512 phy size, and 0 optimal size. Then > >>>>> I used your way to rebuild the kernel > >>>>> /sys/block/nvme0n1/queue/physical_block_size 512 > >>>>> /sys/block/nvme0n1/queue/optimal_io_size 4096 > >>>>> cat /sys/block/nvme0n1/queue/logical_block_size 512 > >>>>> > >>>>> without the patch set > >>>>> write: IOPS=68.0k, BW=266MiB/s (279MB/s)(15.6GiB/60001msec); 0 zone resets > >>>>> with the patch set > >>>>> write: IOPS=69.1k, BW=270MiB/s (283MB/s)(15.8GiB/60001msec); 0 zone resets > >>>>> > >>>>> The second environment: > >>>>> The nvme devices' opt size are 4096. So I don't need to rebuild the kernel. > >>>>> /sys/block/nvme0n1/queue/logical_block_size > >>>>> /sys/block/nvme0n1/queue/physical_block_size > >>>>> /sys/block/nvme0n1/queue/optimal_io_size > >>>>> > >>>>> without the patch set > >>>>> write: IOPS=51.6k, BW=202MiB/s (212MB/s)(11.8GiB/60001msec); 0 zone resets > >>>>> with the patch set > >>>>> write: IOPS=53.5k, BW=209MiB/s (219MB/s)(12.2GiB/60001msec); 0 zone resets > >>>>> > >>>> Sounds like your devices may not have latency issues at sub-optimal sizes. > >>>> Can you provide biosnoop traces with and without patches? > >>>> > >>>> Still, 'works fine for me' is generally not a reason to reject the patches. > >>> > >>> Yes, I can. I tried to install the biosnoop in fedora38 but it failed. > >>> These are the rpm packages I've installed: > >>> bcc-tools-0.25.0-1.fc38.x86_64 > >>> bcc-0.25.0-1.fc38.x86_64 > >>> python3-bcc-0.25.0-1.fc38.noarch > >>> > >>> Are there other packages that I need to install? > >>> > >> I've had issues with the packaged versions as well > >> > >> Best to install from source: > >> https://github.com/iovisor/bcc/ > >> https://github.com/iovisor/bcc/blob/master/INSTALL.md#fedora---source > >> > > Hi Jonathan > > > > I did a test without modifying phys_size and opt_size. And I picked up a part > > of the result: > > > > 0.172142 md0_raid10 2094 nvme1n1 W 1225496264 4096 0.01 > > 0.172145 md0_raid10 2094 nvme0n1 W 1225496264 4096 0.01 > > 0.172161 md0_raid10 2094 nvme3n1 W 16 4096 0.01 > > 0.172164 md0_raid10 2094 nvme2n1 W 16 4096 0.01 > > 0.172166 md0_raid10 2094 nvme1n1 W 16 4096 0.01 > > 0.172168 md0_raid10 2094 nvme0n1 W 16 4096 0.01 > > 0.172178 md0_raid10 2094 nvme3n1 W 633254624 4096 0.01 > > 0.172180 md0_raid10 2094 nvme2n1 W 633254624 4096 0.01 > > 0.172196 md0_raid10 2094 nvme3n1 W 16 4096 0.01 > > 0.172199 md0_raid10 2094 nvme2n1 W 16 4096 0.01 > > 0.172201 md0_raid10 2094 nvme1n1 W 16 4096 0.01 > > 0.172203 md0_raid10 2094 nvme0n1 W 16 4096 0.01 > > 0.172213 md0_raid10 2094 nvme3n1 W 1060251672 4096 0.01 > > 0.172215 md0_raid10 2094 nvme2n1 W 1060251672 4096 0.01 > > > > The last column always shows 0.01. Is that the reason I can't see the > > performance > > improvement? What do you think if I use ssd or hdds? > Try reducing your mdadm's --bitmap-chunk first. In above logs, only LBA 16 is > being used, and that's the first bitmap page (and seemingly also the last). You > want to configure it such that you have more bitmap pages. Reducing the > --bitmap-chunk parameter should create more bitmap pages, and you may run into > the scenario predicted by the patch. > Finally, I can see the performance improvement. It's cool. Thanks for this. The raid with 64MB bitmap size write: IOPS=58.4k, BW=228MiB/s (239MB/s)(13.4GiB/60001msec); 0 zone resets (without patch) write: IOPS=69.3k, BW=271MiB/s (284MB/s)(252MiB/931msec); 0 zone resets (with patch) The raid with 8MB bitmap size write: IOPS=11.6k, BW=45.4MiB/s (47.6MB/s)(2724MiB/60002msec); 0 zone resets (without patch) write: IOPS=43.7k, BW=171MiB/s (179MB/s)(10.0GiB/60001msec); 0 zone resets (with patch) Best Regards Xiao
