On 11/07/2024 01:12, Dave Chinner wrote:
Probably not a lot you can do short of reconfiguring your RAID6
storage devices to handle small IOs better. However, in general,
RAID6 /always sucks/ for small IOs, and the only way to fix this
problem is to use high performance SSDs to give you a massive excess
of write bandwidth to burn on write amplification....
RAID5/6 has the same issues with NVME drives.
Major issue is the bitmap.
5 disk NVMe RAID5, 64K chunk
Test BW IOPS
bitmap internal 64M 700KiB/s 174
bitmap internal 128M 702KiB/s 175
bitmap internal 512M 1142KiB/s 285
bitmap internal 1024M 40.4MiB/s 10.3k
bitmap internal 2G 66.5MiB/s 17.0k
bitmap external 64M 67.8MiB/s 17.3k
bitmap external 1024M 76.5MiB/s 19.6k
bitmap none 80.6MiB/s 20.6k
Single disk 1K 54.1MiB/s 55.4k
Single disk 4K 269MiB/s 68.8k
Tested with fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
