Good day! No, no other activities happened during initial sync - at least I have not done anything. In iostat it were only read operations as well. On Sat, 2 Sept 2023, 10:57 Yu Kuai, <yukuai1@xxxxxxxxxxxxxxx> wrote: > > Hi, > > 在 2023/09/02 4:23, CoolCold 写道: > > Good day! > > > > I have 4 NVMe new drives which are planned to replace 2 current NVMe > > drives, serving primarily as MYSQL storage, Hetzner dedicated server > > AX161 if it matters. Drives are SAMSUNG MZQL23T8HCLS-00A07, 3.8TB . > > System - Ubuntu 20.04 / 5.4.0-153-generic #170-Ubuntu > > > > So the strange thing I do observe, is its initial raid sync speed. > > Created with: > > mdadm --create /dev/md3 --run -b none --level=10 --layout=f2 > > --chunk=16 --raid-devices=4 /dev/nvme0n1 /dev/nvme4n1 /dev/nvme3n1 > > /dev/nvme5n1 > > > > sync speed: > > > > md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0] > > 7501212288 blocks super 1.2 16K chunks 2 far-copies [4/4] [UUUU] > > [=>...................] resync = 6.2% (466905632/7501212288) > > finish=207.7min speed=564418K/sec > > > Is there any read/write to the array? Because for raid10, normal io > can't concurrent with sync io, brandwidth will be bad if they both exit, > specially for old kernels. > > Thanks, > Kuai > > > If I try to create RAID1 with just two drives - sync speed is around > > 3.2GByte per second, sysclt is tuned of course: > > dev.raid.speed_limit_max = 8000000 > > > > Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] > > [raid4] [raid10] > > md70 : active raid1 nvme4n1[1] nvme5n1[0] > > 3750606144 blocks super 1.2 [2/2] [UU] > > [>....................] resync = 1.5% (58270272/3750606144) > > finish=19.0min speed=3237244K/sec > > > >>From iostat, drives are basically doing just READs, no writes. > > Quick tests with fio, mounting single drive shows it can do around 30k > > IOPS with 16kb ( fio --rw=write --ioengine=sync --fdatasync=1 > > --directory=test-data --size=8200m --bs=16k --name=mytest ) so likely > > issue are not drives themselves. > > > > Not sure where to look further, please advise. > > >
