On Sat, Sep 2, 2023 at 4:00 AM Roman Mamedov <rm@xxxxxxxxxxx> wrote: > > On Sat, 2 Sep 2023 03:43:42 +0700 > CoolCold <coolthecold@xxxxxxxxx> wrote: > > > > > md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0] > > > > 7501212288 blocks super 1.2 16K chunks 2 far-copies [4/4] [UUUU] > > > > [=>...................] resync = 6.2% (466905632/7501212288) > > > > finish=207.7min speed=564418K/sec > > > > > > Any difference if you use e.g. --chunk=1024? > > Goes up to 1.4GB > > > > md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0] > > 7501209600 blocks super 1.2 1024K chunks 2 far-copies [4/4] [UUUU] > > [>....................] resync = 0.4% (35959488/7501209600) > > finish=86.4min speed=1438382K/sec > > Looks like you have found at least some bottleneck. Does it ever reach the Definitely there is a bottleneck and I very much doubt I'm the first one facing this - NVMe drives with > 1GB/sec are quite widespread. > RAID1 performance at some point if you raise it further to 4096, 8192 or more? I can try, for sake of testing, but in terms of practical outcome - let's imagine with 8MB chunks it reaches maximum - what to do with that knowledge? > > It might also be worth it to try making the RAID with --assume-clean, and then > look at the actual array performance, not just the sync speed. > > > > How about a newer kernel (such as 6.1)? > > Not applicable in my case- there is no test machine unluckily to play > > around with non LTS and reboots. Upgrading to next HWE kernel may > > happen though, which is 5.15.0-82-generic #91-Ubuntu. > > Do you know any specific patches/fixes landed since 5.4? > > No idea. I guessed if you are just setting up a new server, it would be > possible to slip in a reboot or a few. :) Unluckily no - trying to speedup existing DB which is a master node in this setup. Full disclosure: I've actually done a "quick" test with those drives handling the load - created RADI10/f2, added to VG, did pvmove for mysql specific LV and observed. In terms of weighted io time, it performed much better. Now, I've moved data back to "old" drives (pvmove again) and preparing for "proper" not "quick" setup - NVME sector size, XFS sunit/swidth change and so on. > > -- > With respect, > Roman -- Best regards, [COOLCOLD-RIPN]
