Avago (LSI) SAS-3 controller, poor performance on CentOS 7

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



I have a handful of new systems where I've seen unexpectedly low disk performance on an Avago SAS controller, when using CentOS 7. It looked like a regression, so I installed CentOS 6 on one of them and reloaded CentOS 7 on the other. Immediately after install, a difference is apparent in the RAID rebuild speed. The CentOS 6 system is initializing its software RAID5 array at somewhere around 120MB/s, while the CentOS 7 system is rebuilding at around 55MB/s.

It took a while to get an older system installed, and I wasn't able to boot CentOS 6 under UEFI, to that's one difference between the systems. I also set the elevator on CentOS 7 to cfq, to match the CentOS 6 system. That didn't have any apparent effect.

Other than that, the kernel reports "MSI-X vectors supported: 96" on C6, but "MSI-X vectors supported: 8" on C7. I don't know if that's a difference that would affect performance.

Does anyone have experience with these controllers? Any pointers while I continue tracking this down?

Output from lspci follows, as well as dmesg output regarding the MPT controller and the RAID array status on C6 and C7.


01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
        Subsystem: LSI Logic / Symbios Logic Device 30e0
        Physical Slot: 1
        Flags: bus master, fast devsel, latency 0, IRQ 24
        I/O ports at e000 [size=256]
        Memory at fb240000 (64-bit, non-prefetchable) [size=64K]
        Memory at fb200000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at fb100000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [1e0] #19
        Capabilities: [1c0] Power Budgeting <?>
        Capabilities: [190] #16
        Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: mpt3sas


CentOS 6:

# dmesg | grep -i mpt
mpt3sas version 09.102.00.00-rh loaded
mpt3sas 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
mpt3sas 0000:01:00.0: setting latency timer to 64
mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32772632 kB) mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 12, max_msix_vectors: -1
mpt3sas 0000:01:00.0: irq 60 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 61 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 62 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 63 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 64 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 65 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 66 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 67 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 60
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 61
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 62
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 63
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 64
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 65
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 66
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 67
mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 68
mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 69
mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 70
mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 71
mpt3sas_cm0: iomem(0x00000000fb240000), mapped(0xffffc90013ee0000), size(65536)
mpt3sas_cm0: ioport(0x000000000000e000), size(256)
mpt3sas_cm0: sending message unit reset !!
mpt3sas_cm0: message unit reset: SUCCESS
mpt3sas_cm0: Allocated physical memory: size(17971 kB)
mpt3sas_cm0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240)
mpt3sas_cm0: Scatter Gather Elements per IO(128)
mpt3sas_cm0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00)
mpt3sas_cm0: Protocol=(
scsi0 : Fusion MPT SAS Host
mpt3sas_cm0: sending port enable !!
mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aee5510), phys(8)
mpt3sas_cm0: port enable: SUCCESS

#  cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md0 : active raid1 sda2[0] sdb2[1]
      999360 blocks super 1.0 [2/2] [UU]

md1 : active raid5 sdf3[6] sde3[4] sdd3[3] sdc3[2] sda3[0] sdb3[1]
19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [====>................] recovery = 22.1% (864374932/3905691648) finish=417.0min speed=121538K/sec
      bitmap: 1/30 pages [4KB], 65536KB chunk

unused devices: <none>


CentOS 7:

# dmesg | grep -i mpt
mpt3sas version 04.100.00.00 loaded
mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32674176 kB)
mpt3sas0: MSI-X vectors supported: 8, no of cores: 12, max_msix_vectors: 8
mpt3sas 0000:01:00.0: irq 45 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 46 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 47 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 49 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 50 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 51 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 52 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 53 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 45
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 46
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 47
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 50
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 51
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 52
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 53
mpt3sas0: iomem(0x00000000fb240000), mapped(0xffffc90013180000), size(65536)
mpt3sas0: ioport(0x000000000000e000), size(256)
mpt3sas0: sending message unit reset !!
mpt3sas0: message unit reset: SUCCESS
mpt3sas0: Allocated physical memory: size(17329 kB)
mpt3sas0: Current Controller Queue Depth(9979),Max Controller Queue Depth(10240)
mpt3sas0: Scatter Gather Elements per IO(128)
mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00)
mpt3sas0: Protocol=(
scsi host0: Fusion MPT SAS Host
mpt3sas0: sending port enable !!
mpt3sas0: host_add: handle(0x0001), sas_addr(0x500605b00aee5ca0), phys(8)
mpt3sas0: port enable: SUCCESS

#  cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sda1[0] sdb1[1]
      194496 blocks super 1.0 [2/2] [UU]

md126 : active raid5 sdf3[6] sda3[0] sde3[4] sdd3[3] sdc3[2] sdb3[1]
19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_] [=========>...........] recovery = 47.6% (1862704880/3905691648) finish=569.6min speed=59772K/sec
      bitmap: 2/30 pages [8KB], 65536KB chunk

md127 : active raid1 sda2[0] sdb2[1]
      999360 blocks super 1.0 [2/2] [UU]

unused devices: <none>


_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos



[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux