Makes some sense, but from my perspective not much sense - in any practical case I can remember, creating a FRESH/NEW array silently assumes you want to have it be filled by zeroes. When you care about actual drive content - there is an "assume-clean" option. Will option like "--fill-with-zeroes" make sense to do "no compare" and just write the same data on all drives? My current drives are relatively small - 3.8TB, but 15TB and 30TB NVMes exist and I hardly can imagine provisioning such server where raid10 sync will take WEEK? Highly likely I'm not the first one facing this behavior and larger market players have already met something of this sort. On Sat, Sep 2, 2023 at 1:13 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote: > > Hi, > > 在 2023/09/02 14:07, CoolCold 写道: > > Good day! > > No, no other activities happened during initial sync - at least I have > > not done anything. In iostat it were only read operations as well. > > > I think this is because resync is different for raid1 and raid10, > > In raid1, just read from one rdev and write to the other rdev. > > In raid10, resync must read from all rdev first, and then compare and > write if contents is different, it is obvious that raid10 will be slower > if contents is different. However, I do feel this behavior is strange, > because contents is likely different in initial sync. > > Thanks, > Kuai > > > > > On Sat, 2 Sept 2023, 10:57 Yu Kuai, <yukuai1@xxxxxxxxxxxxxxx> wrote: > >> > >> Hi, > >> > >> 在 2023/09/02 4:23, CoolCold 写道: > >>> Good day! > >>> > >>> I have 4 NVMe new drives which are planned to replace 2 current NVMe > >>> drives, serving primarily as MYSQL storage, Hetzner dedicated server > >>> AX161 if it matters. Drives are SAMSUNG MZQL23T8HCLS-00A07, 3.8TB . > >>> System - Ubuntu 20.04 / 5.4.0-153-generic #170-Ubuntu > >>> > >>> So the strange thing I do observe, is its initial raid sync speed. > >>> Created with: > >>> mdadm --create /dev/md3 --run -b none --level=10 --layout=f2 > >>> --chunk=16 --raid-devices=4 /dev/nvme0n1 /dev/nvme4n1 /dev/nvme3n1 > >>> /dev/nvme5n1 > >>> > >>> sync speed: > >>> > >>> md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0] > >>> 7501212288 blocks super 1.2 16K chunks 2 far-copies [4/4] [UUUU] > >>> [=>...................] resync = 6.2% (466905632/7501212288) > >>> finish=207.7min speed=564418K/sec > >>> > >> Is there any read/write to the array? Because for raid10, normal io > >> can't concurrent with sync io, brandwidth will be bad if they both exit, > >> specially for old kernels. > >> > >> Thanks, > >> Kuai > >> > >>> If I try to create RAID1 with just two drives - sync speed is around > >>> 3.2GByte per second, sysclt is tuned of course: > >>> dev.raid.speed_limit_max = 8000000 > >>> > >>> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] > >>> [raid4] [raid10] > >>> md70 : active raid1 nvme4n1[1] nvme5n1[0] > >>> 3750606144 blocks super 1.2 [2/2] [UU] > >>> [>....................] resync = 1.5% (58270272/3750606144) > >>> finish=19.0min speed=3237244K/sec > >>> > >>> >From iostat, drives are basically doing just READs, no writes. > >>> Quick tests with fio, mounting single drive shows it can do around 30k > >>> IOPS with 16kb ( fio --rw=write --ioengine=sync --fdatasync=1 > >>> --directory=test-data --size=8200m --bs=16k --name=mytest ) so likely > >>> issue are not drives themselves. > >>> > >>> Not sure where to look further, please advise. > >>> > >> > > > > . > > > -- Best regards, [COOLCOLD-RIPN]
