On 2020/09/13 3:20, Ian S. Worthington wrote: > Many thanks again Damien for that great information. > > I've now run the following tests: > > sudo fio --name START --eta-newline=5s --eta-interval=5s -filename=/dev/sda > --rw=randwrite --size=100t --io_size=14t --ioengine=libaio --iodepth=4 > --direct=1 --numjobs=1 --group_reporting --blocksize=4k --runtime=1m > sudo fio --name FILLSEQ --eta-newline=5s --eta-interval=5s -filename=/dev/sda > --rw=write --size=14t --io_size=14t --ioengine=libaio --iodepth=4 > --direct=1 --numjobs=1 --group_reporting --blocksize=512m > sudo fio --name 10TB1MB --eta-newline=5s --eta-interval=5s -filename=/dev/sda > --rw=randwrite --size=14t --io_size=14t --ioengine=libaio --iodepth=4 > --direct=1 --numjobs=1 --group_reporting --blocksize=1m > sudo fio --name 4K10HR --eta-newline=5s --eta-interval=5s -filename=/dev/sda > --rw=randwrite --size=14t --io_size=14t --ioengine=libaio --iodepth=1 > --direct=1 --numjobs=1 --group_reporting --blocksize=4k --runtime=10h > > Results: > > FILLSEQ: > Purpose: to fill disk as quickly as possible. > Seems FIO IOS counts blocks in some places and maybe split blocks in other > places? > blocksize=512MiB > avg bw=157MiB/s ios=58720040/93692287 msec(26.03 hours), io=14.0TiB > (15.4TB), > 14tib = 14680064 miB > ~/26.03/3600 = 156.7 MiB/s > 58720040 ios/93692287 msecs = 0.6267 ios/ms = 626.73 iops > 157 MiB/s / 626 iops = 0.25 MiB/io = 256KiB/io => 2048 ios/512MiB > 626.73/2048=0.31 blks/sec > > From https://kernel.dk/when-2mb-turns-into-512k.pdf: > > pi@raspberrypi:/ $ cat /sys/block/sda/queue/max_hw_sectors_kb > 256 > pi@raspberrypi:/ $ cat /sys/block/sda/queue/max_sectors_kb > 256 > pi@raspberrypi:/ $ cat /sys/block/sda/queue/max_segments > 2048 > pi@raspberrypi:/ $ cat /sys/block/sda/queue/max_segment_size > 65536 > > This tells us that the maximum size the device can support is 256KB > (max_hw_sectors_kb) > and the maximum size that the kernel allows is 256KB (max_sectors_kb). Which highly depends on the HBA since the hard-disk interface will accept much larger commands. This 256KB on your system seems to be very low. What HBA are you using ? These days, most SAS HBAs have at least 128 or 256 segments, which allow up to 512KB/1MB IOs. SATA/AHCI is limited at 169 segments for up to 32MB max_hw_sectors_kb (and 1280 KB max_sectors_kb). > Additionally, the DMA engine is limited to 2048 segments of IO, > each with a max size of 64KB. That's a lot. I wonder why max_hw_sectors_kb end up so small. > 10TB1MB: > Blksize 1MB randwrite 14TiB > Purpose: Attempt to ensure any CMA cache is full > Avg BW: 104 MiB/s; IOPS: 104 > Samples: BW min: 32 MiB/s; max: 190 MiB/s; avg=104 MiB/s > IOPS min: 32; max: 190; avg: 104 > lat (msec) : 4=0.01%, 10=0.01%, 20=0.86%, 50=98.44%, 100=0.69% > lat (msec) : 250=0.01%, 500=0.01% > > > 4K10HR: > 10 hours of randwrite blksize=4KiB > Purpose: See if we can see any delays caused by forced destaging from CMA > cache to SMA > BW: 1960 KiB/s; IOPS: 489; 67.3 GiB written > Sampling: > BW min: 640 KiB/s; max: 8264 KiB/s > IOPS min: 160; max: 2066 > lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=1.12%, 750=5.86% > lat (usec) : 1000=1.80% > lat (msec) : 2=48.59%, 4=39.72%, 10=2.81%, 20=0.07%, 50=0.02% > lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% > No obvious slow downs in fio log. > > Conclusions: > > So my guess, from the absence of any drastic slow downs during the 4K test is > that either this disk is not an SMR disk, or if it is, I'm not showing it in a > test that writes 4k random for only 10 hours. > >> Getting >> clear information from the drive vendors seems to me like a much easier > solution :) > > Totally agree. I've declined to look until now in case it biased my > analysis. > > The model here is a WD140EMFZ-11A0WA0 which WD don't seem to publish any > information on. HDDScan claim in their blog that its a CMR (PMR, He) drive > manufactured by HGST (https://hddscan.com/blog/2020/hdd-wd-smr.html), but I > have no idea if this information is reliable or not. The list of WD drives using SMR as drive-managed is published. See: https://blog.westerndigital.com/wd-red-nas-drives/ and https://blog.westerndigital.com/wp-content/uploads/2020/07/WD_SMR_SKUs_vDS.pdf Your drive is not on the list :) -- Damien Le Moal Western Digital Research
