On 3/2/2023 2:05 AM, Xiao Ni wrote: > On Thu, Mar 2, 2023 at 1:57 AM Jonathan Derrick > <jonathan.derrick@xxxxxxxxx> wrote: >> >> >> >> On 3/1/2023 5:36 AM, Xiao Ni wrote: >>> On Wed, Mar 1, 2023 at 7:10 AM Jonathan Derrick >>> <jonathan.derrick@xxxxxxxxx> wrote: >>>> >>>> Hi Xiao >>>> >>>> On 2/26/2023 6:56 PM, Xiao Ni wrote: >>>>> Hi Jonathan >>>>> >>>>> I did a test in my environment, but I didn't see such a big >>>>> performance difference. >>>>> >>>>> The first environment: >>>>> All nvme devices have 512 logical size, 512 phy size, and 0 optimal size. Then >>>>> I used your way to rebuild the kernel >>>>> /sys/block/nvme0n1/queue/physical_block_size 512 >>>>> /sys/block/nvme0n1/queue/optimal_io_size 4096 >>>>> cat /sys/block/nvme0n1/queue/logical_block_size 512 >>>>> >>>>> without the patch set >>>>> write: IOPS=68.0k, BW=266MiB/s (279MB/s)(15.6GiB/60001msec); 0 zone resets >>>>> with the patch set >>>>> write: IOPS=69.1k, BW=270MiB/s (283MB/s)(15.8GiB/60001msec); 0 zone resets >>>>> >>>>> The second environment: >>>>> The nvme devices' opt size are 4096. So I don't need to rebuild the kernel. >>>>> /sys/block/nvme0n1/queue/logical_block_size >>>>> /sys/block/nvme0n1/queue/physical_block_size >>>>> /sys/block/nvme0n1/queue/optimal_io_size >>>>> >>>>> without the patch set >>>>> write: IOPS=51.6k, BW=202MiB/s (212MB/s)(11.8GiB/60001msec); 0 zone resets >>>>> with the patch set >>>>> write: IOPS=53.5k, BW=209MiB/s (219MB/s)(12.2GiB/60001msec); 0 zone resets >>>>> >>>> Sounds like your devices may not have latency issues at sub-optimal sizes. >>>> Can you provide biosnoop traces with and without patches? >>>> >>>> Still, 'works fine for me' is generally not a reason to reject the patches. >>> >>> Yes, I can. I tried to install the biosnoop in fedora38 but it failed. >>> These are the rpm packages I've installed: >>> bcc-tools-0.25.0-1.fc38.x86_64 >>> bcc-0.25.0-1.fc38.x86_64 >>> python3-bcc-0.25.0-1.fc38.noarch >>> >>> Are there other packages that I need to install? >>> >> I've had issues with the packaged versions as well >> >> Best to install from source: >> https://github.com/iovisor/bcc/ >> https://github.com/iovisor/bcc/blob/master/INSTALL.md#fedora---source >> > Hi Jonathan > > I did a test without modifying phys_size and opt_size. And I picked up a part > of the result: > > 0.172142 md0_raid10 2094 nvme1n1 W 1225496264 4096 0.01 > 0.172145 md0_raid10 2094 nvme0n1 W 1225496264 4096 0.01 > 0.172161 md0_raid10 2094 nvme3n1 W 16 4096 0.01 > 0.172164 md0_raid10 2094 nvme2n1 W 16 4096 0.01 > 0.172166 md0_raid10 2094 nvme1n1 W 16 4096 0.01 > 0.172168 md0_raid10 2094 nvme0n1 W 16 4096 0.01 > 0.172178 md0_raid10 2094 nvme3n1 W 633254624 4096 0.01 > 0.172180 md0_raid10 2094 nvme2n1 W 633254624 4096 0.01 > 0.172196 md0_raid10 2094 nvme3n1 W 16 4096 0.01 > 0.172199 md0_raid10 2094 nvme2n1 W 16 4096 0.01 > 0.172201 md0_raid10 2094 nvme1n1 W 16 4096 0.01 > 0.172203 md0_raid10 2094 nvme0n1 W 16 4096 0.01 > 0.172213 md0_raid10 2094 nvme3n1 W 1060251672 4096 0.01 > 0.172215 md0_raid10 2094 nvme2n1 W 1060251672 4096 0.01 > > The last column always shows 0.01. Is that the reason I can't see the > performance > improvement? What do you think if I use ssd or hdds? Try reducing your mdadm's --bitmap-chunk first. In above logs, only LBA 16 is being used, and that's the first bitmap page (and seemingly also the last). You want to configure it such that you have more bitmap pages. Reducing the --bitmap-chunk parameter should create more bitmap pages, and you may run into the scenario predicted by the patch. > > Best Regards > Xiao >
