Hi,
Recently we encountered a read performance problem on LSI SAS-3 3108 RAID controller.
The read performance is much better when nr_requests is set as 192 compared with 256,
but after disabling the NoRA (No Readahead) feature of the hardware RAID1, there will
be no difference between nr_requests=192 and nr_requests=256.
One scene is the direct read of one ext4 file with 8GB size. The ext4 fs is stacked
on hardware RAID1 with 3.7TB size, and the RAID1 is composed of two HDDs.
The queue_depth of RAID1 device is set as 256 by driver.
fio --direct=1 --ioengine=libaio --group_reporting=1 --runtime=30 --bs=4k \
--name=1 --numjobs=1 --iodepth=512 --filesize=8G --directory=/tmp/sdd
one ext4 file:
IOPS | nr_requests=128 | nr_requests=192 | nr_requests=256 |
RA | 51k | 51k | 48k |
NoRA | 47k | 46k | 47k |
Another scene is the direct read of two ext4 files with 4GB size.
fio --direct=1 --ioengine=libaio --group_reporting=1 --runtime=30 --bs=4k \
--numjobs=2 --iodepth=256 --filesize=4G --directory=/tmp/sdd
two ext4 files:
IOPS | nr_requests=128 | nr_requests=192 | nr_requests=256 |
RA | 95.7k | 94.5k | 30.7k |
NoRA | 27.3k | 27.1k | 27.2k |
These results show RA feature can boost the read performance, but when nr_requests
is increased, performance is degraded.
However when using a JBOD setup on HDD which has the same model as the HDD
in RAID1 setup, there is no such problem.
one ext4 file:
IOPS | nr_requests=128 | nr_requests=192 | nr_requests=256 |
| 50k | 50.4k | 50.5 |
two ext4 files:
IOPS | nr_requests=128 | nr_requests=192 | nr_requests=256 |
| 46.5k | 46.3k | 46.0k |
So is the performance degradation a known issue of the RAID1 RA feature,
or is there other explanation for it ?
Regards,
Tao
---
Other details of test environment:
(1) linux kernel version
5.6.15
(2) block setup
device/queue_depth:256
queue/scheduler:[mq-deadline] kyber bfq none
(3) driver version
megasas: 07.713.01.00-rc1
[ 3.302270] megasas: 07.713.01.00-rc1
[ 3.302606] megaraid_sas 0000:1c:00.0: BAR:0x1 BAR's base_addr(phys):0x00000000a3500000 mapped virt_addr:0x00000000af230875
[ 3.302608] megaraid_sas 0000:1c:00.0: FW now in Ready state
[ 3.302610] megaraid_sas 0000:1c:00.0: 63 bit DMA mask and 32 bit consistent mask
[ 3.302965] megaraid_sas 0000:1c:00.0: firmware supports msix : (96)
[ 3.305024] megaraid_sas 0000:1c:00.0: requested/available msix 73/73
[ 3.305028] megaraid_sas 0000:1c:00.0: current msix/online cpus : (73/72)
[ 3.305029] megaraid_sas 0000:1c:00.0: RDPQ mode : (disabled)
[ 3.305031] megaraid_sas 0000:1c:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 0
[ 3.305362] megaraid_sas 0000:1c:00.0: Configured max firmware commands: 927
[ 3.308155] megaraid_sas 0000:1c:00.0: Performance mode :Latency
[ 3.308156] megaraid_sas 0000:1c:00.0: FW supports sync cache : Yes
[ 3.308161] megaraid_sas 0000:1c:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 3.352112] megaraid_sas 0000:1c:00.0: FW provided supportMaxExtLDs: 1 max_lds: 64
[ 3.352113] megaraid_sas 0000:1c:00.0: controller type : MR(1024MB)
[ 3.352114] megaraid_sas 0000:1c:00.0: Online Controller Reset(OCR) : Enabled
[ 3.352114] megaraid_sas 0000:1c:00.0: Secure JBOD support : Yes
[ 3.352115] megaraid_sas 0000:1c:00.0: NVMe passthru support : No
[ 3.352115] megaraid_sas 0000:1c:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
[ 3.352116] megaraid_sas 0000:1c:00.0: JBOD sequence map support : Yes
[ 3.352117] megaraid_sas 0000:1c:00.0: PCI Lane Margining support : No
[ 3.375590] megaraid_sas 0000:1c:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[ 3.375591] megaraid_sas 0000:1c:00.0: INIT adapter done
[ 3.388987] megaraid_sas 0000:1c:00.0: pci id : (0x1000)/(0x005d)/(0x19e5)/(0xd207)
[ 3.388988] megaraid_sas 0000:1c:00.0: unevenspan support : no
[ 3.388988] megaraid_sas 0000:1c:00.0: firmware crash dump : yes
[ 3.388989] megaraid_sas 0000:1c:00.0: JBOD sequence map : enabled
(4) megaraid firmware version
Product Name = SAS3108
FW Package Build = 24.16.0-0106
BIOS Version = 6.32.02.0_4.17.08.00_0x06150500
FW Version = 4.660.00-8102
Driver Name = megaraid_sas
Driver Version = 07.713.01.00-rc1
Current Personality = RAID-Mode
(5) hardware RAID1 setup
-------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
-------------------------------------------------------------
1/2 RAID1 Optl RW Yes RWTD - ON 3.637 TB
-------------------------------------------------------------
-----------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------
0:5 6 Onln 1 3.637 TB SATA HDD N N 512B MG04ACA400N U -
0:6 8 Onln 1 3.637 TB SATA HDD N N 512B MG04ACA400N U -
-----------------------------------------------------------------------
(6) hardware JBOD setup
-----------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------
0:4 7 JBOD - 3.638 TB SATA HDD N N 512B MG04ACA400N U -
-----------------------------------------------------------------------
