slow mdadm reshape, normal?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



is it normal that a raid5 reshape happens at only 7000kB/s on
newer disks? i created the array at 60MB/s stable over the whole array
(accordin to /proc/mdstat), sometimes up to 90MB/s

chunk size is 64kB, disks are 9xst1500 (1.5tb seagate with upgraded 
firmware CC1H /SD1A)

the hardware is a pci 32bit 4port sata and onboard sata2 (6 port sata - non
ide host board) if that explains it?

modules are sata_nv and sata_sil

I have lvm2 and xfs+2TB data on top of md before the growing of two new 
disks from 7 to 9.

perhaps this speed is normal? 

I have tried echoing various things into proc and sys, no luck there

Any ideas?

my apology for the direct mailing, Niel :)
and sorry to everyone for wasting bandwidth with this lengthy mail if its normal :)


Here is the build+reshape log:
-------------------------------------------

while creating the raid5 as normal (the new disks has not been growed yet)

mfs:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7](S) sdb[8](S) sda[0] sdi[9] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks level 5, 64k chunk, algorithm 2 [7/6] [UUUUUU_]
      [================>....]  recovery = 82.5% (1209205112/1465138496) finish=71.5min speed=59647K/sec

unused devices: <none>
mfs:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
cpq:/diskless/mws    284388032 225657236  58730796  80% /
tmpfs                  1817684         0   1817684   0% /lib/init/rw
udev                     10240        84     10156   1% /dev
tmpfs                  1817684         0   1817684   0% /dev/shm
/dev/mapper/st1500-bigdaddy
                     8589803488 2041498436 6548305052  24% /bigdaddy
cpq:/home/diskless/tftp/kernels/src
                     284388096 225657216  58730880  80% /usr/src
mfs:~# umount /bigdaddy/
mfs:~# mdadm --stop /dev/md0
mdadm: fail to stop array /dev/md0: Device or resource busy
mfs:~# mdadm --grow /dev/md0 --raid-devices=9
mdadm: Need to backup 1536K of critical section..
mdadm: /dev/md0: failed to find device 6. Array might be degraded.
 --grow aborted
(reverse-i-search)`for': vi /home/michael/.forward
mfs:~# for i in a b c d e f g h i; do echo sd$i;dd if=/dev/sd$i of=/dev/null bs=1M count=200;done
sda
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,46597 s, 85,0 MB/s
sdb
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66641 s, 126 MB/s
sdc
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,51205 s, 83,5 MB/s
sdd
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,33702 s, 89,7 MB/s
sde
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,62686 s, 129 MB/s
sdf
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66326 s, 126 MB/s
sdg
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,59947 s, 131 MB/s
sdh
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66347 s, 126 MB/s
sdi
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,6167 s, 130 MB/s

mfs:/usr/share/doc/mdadm# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.91
  Creation Time : Tue Jun  9 21:56:04 2009
     Raid Level : raid5
     Array Size : 8790830976 (8383.59 GiB 9001.81 GB)
  Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
   Raid Devices : 9
  Total Devices : 9
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jun 11 00:26:26 2009
          State : clean, recovering
 Active Devices : 9
Working Devices : 9
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

 Reshape Status : 13% complete
  Delta Devices : 2, (7->9)

           UUID : 5f206395:a6c11495:9091a83d:f5070ca0 (local to host mfs)
         Events : 0.139246

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       32        1      active sync   /dev/sdc
       2       8       64        2      active sync   /dev/sde
       3       8       80        3      active sync   /dev/sdf
       4       8       96        4      active sync   /dev/sdg
       5       8      112        5      active sync   /dev/sdh
       6       8      128        6      active sync   /dev/sdi
       7       8       48        7      active sync   /dev/sdd
       8       8       16        8      active sync   /dev/sdb


reshaping (used --grow with two new disks and 9 total)

mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      [>....................]  reshape =  0.4% (6599364/1465138496) finish=3009.7min speed=8074K/sec

unused devices: <none>
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      [>....................]  reshape =  0.4% (6611448/1465138496) finish=3057.7min speed=7947K/sec

unused devices: <none>

mfs:/usr/share/doc/mdadm# echo 20000 > /proc/sys/dev/raid/speed_limit_min

mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      [>....................]  reshape =  0.6% (9161780/1465138496) finish=3611.7min speed=6717K/sec

unused devices: <none>
mfs:/usr/share/doc/mdadm# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      [>....................]  reshape =  2.8% (41790784/1465138496) finish=3786.7min speed=6261K/sec


unused devices: <none>

mfs:/usr/share/doc/mdadm# echo 25000 > /proc/sys/dev/raid/speed_limit_min
mfs:/usr/share/doc/mdadm# echo 400000 >/proc/sys/dev/raid/speed_limit_max

mfs:/usr/share/doc/mdadm# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
      8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      [==>..................]  reshape = 13.6% (200267648/1465138496) finish=3044.2min speed=6922K/sec

mfs:/usr/share/doc/mdadm# lsmod
Module                  Size  Used by
ext3                  106356  0 
jbd                    40164  1 ext3
mbcache                 6872  1 ext3
ipv6                  198960  26 
xfs                   435712  0 
exportfs                3628  1 xfs
raid456               116508  1 
async_xor               1936  1 raid456
async_memcpy            1408  1 raid456
async_tx                2396  3 raid456,async_xor,async_memcpy
xor                    13936  2 raid456,async_xor
md_mod                 72944  2 raid456
dm_mod                 45188  4 
sd_mod                 20916  9 
sata_sil                8180  3 
sata_nv                19468  6 
ehci_hcd               28944  0 
ohci_hcd               19464  0 
libata                148672  2 sata_sil,sata_nv
i2c_nforce2             5804  0 
scsi_mod               91884  2 sd_mod,libata
usbcore               106508  2 ehci_hcd,ohci_hcd
i2c_core               20640  1 i2c_nforce2

mfs:/usr/share/doc/mdadm# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1980   648 ?        Ss   Jun10   0:00 init [2]     
root         2  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S<   Jun10   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   Jun10   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S<   Jun10   0:00 [events/0]
root         8  0.0  0.0      0     0 ?        S<   Jun10   0:00 [events/1]
root         9  0.0  0.0      0     0 ?        S<   Jun10   0:00 [work_on_cpu/0]
root        10  0.0  0.0      0     0 ?        S<   Jun10   0:00 [work_on_cpu/1]
root        11  0.0  0.0      0     0 ?        S<   Jun10   0:00 [khelper]
root        76  0.0  0.0      0     0 ?        S<   Jun10   0:32 [kblockd/0]
root        77  0.0  0.0      0     0 ?        S<   Jun10   0:01 [kblockd/1]
root        78  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kacpid]
root        79  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kacpi_notify]
root       187  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kseriod]
root       236  1.5  0.0      0     0 ?        S<   Jun10  22:28 [kswapd0]
root       278  0.0  0.0      0     0 ?        S<   Jun10   0:00 [aio/0]
root       279  0.0  0.0      0     0 ?        S<   Jun10   0:00 [aio/1]
root       283  0.0  0.0      0     0 ?        S<   Jun10   0:00 [nfsiod]
root       998  0.0  0.0      0     0 ?        S<   Jun10   0:01 [rpciod/0]
root       999  0.0  0.0      0     0 ?        S<   Jun10   0:00 [rpciod/1]
root      1087  0.0  0.0   2144   764 ?        S<s  Jun10   0:00 udevd --daemon
root      2041  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ksuspend_usbd]
root      2045  0.0  0.0      0     0 ?        S<   Jun10   0:00 [khubd]
root      2091  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ata/0]
root      2092  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ata/1]
root      2093  0.0  0.0      0     0 ?        S<   Jun10   0:00 [ata_aux]
root      2106  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_0]
root      2107  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_1]
root      2108  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_2]
root      2109  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_3]
root      2116  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_4]
root      2117  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_5]
root      2190  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_6]
root      2191  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_7]
root      2194  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_8]
root      2195  0.0  0.0      0     0 ?        S<   Jun10   0:00 [scsi_eh_9]
root      2420  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kstriped]
root      2460 18.6  0.0      0     0 ?        S<   Jun10 271:03 [md0_raid5]
root      2482  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kdmflush]
root      2518  0.0  0.0      0     0 ?        S<   Jun10   0:00 [xfs_mru_cache]
root      2522  0.0  0.0      0     0 ?        S<   Jun10   0:03 [xfslogd/0]
root      2523  0.0  0.0      0     0 ?        S<   Jun10   0:00 [xfslogd/1]
root      2524  0.4  0.0      0     0 ?        S<   Jun10   6:45 [xfsdatad/0]
root      2525  0.0  0.0      0     0 ?        S<   Jun10   0:23 [xfsdatad/1]
daemon    2581  0.0  0.0   1764   444 ?        Ss   Jun10   0:00 /sbin/portmap
statd     2592  0.0  0.0   1824   624 ?        Ss   Jun10   0:00 /sbin/rpc.statd
root      2747  0.0  0.0   5272   916 ?        Ss   Jun10   0:00 /usr/sbin/sshd
root      3025  0.0  0.0   3080   600 ?        S    Jun10   0:00 /usr/sbin/smartd --pidfile /var/run/smartd.pid --interval=1800
ntp       3038  0.0  0.0   4132  1044 ?        Ss   Jun10   0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 105:105 -g
root      3048  0.0  0.0   2012   572 ?        Ss   Jun10   0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
root      3060  0.0  0.0      0     0 ?        S<   Jun10   0:00 [lockd]
root      3064  0.0  0.0   1644   504 tty1     Ss+  Jun10   0:00 /sbin/getty 38400 tty1
root      3067  0.0  0.0   1644   496 tty2     Ss+  Jun10   0:00 /sbin/getty 38400 tty2
root      3074  0.0  0.0   7992  1336 ?        Ss   Jun10   0:00 sshd: michael [priv]
michael   3076  0.0  0.0   7992  1084 ?        S    Jun10   0:00 sshd: michael@pts/0
michael   3077  0.0  0.0   4620  1904 pts/0    Ss   Jun10   0:00 -bash
root      3084  0.0  0.0   3636   972 pts/0    S    Jun10   0:00 su
root      3085  0.0  0.0   4132  1740 pts/0    S+   Jun10   0:00 bash
root      3147  0.0  0.0   7992  1336 ?        Ss   Jun10   0:00 sshd: michael [priv]
michael   3149  0.0  0.0   8140  1068 ?        S    Jun10   0:06 sshd: michael@pts/1
michael   3150  0.0  0.0   4632  1924 pts/1    Ss   Jun10   0:00 -bash
root      3283  0.0  0.0   3636   976 pts/1    S    Jun10   0:00 su
root      3284  0.0  0.0   4160  1776 pts/1    S    Jun10   0:00 bash
root      3474  0.0  0.0   7924  1316 ?        Ss   Jun10   0:00 /usr/sbin/nmbd -D
root      3476  0.0  0.0  13388  2116 ?        Ss   Jun10   0:00 /usr/sbin/smbd -D
root      3478  0.0  0.0  13388   892 ?        S    Jun10   0:00 /usr/sbin/smbd -D
106       4608  0.0  0.0   6132   924 ?        Ss   Jun10   0:00 /usr/sbin/exim4 -bd -q30m
root      4702  0.2  0.0      0     0 ?        S    Jun10   2:35 [pdflush]
root      4707  0.2  0.0      0     0 ?        S    Jun10   2:09 [pdflush]
root      4870  0.0  0.0      0     0 ?        S<   Jun10   0:00 [kdmflush]
root      4966  3.4  0.0      0     0 ?        D<   Jun10  16:12 [md0_reshape]
root      5338  0.0  0.0   3580  1012 pts/1    R+   00:25   0:00 ps aux

mfs:/usr/share/doc/mdadm# w
 00:25:16 up 1 day, 12 min,  2 users,  load average: 1,38, 1,30, 1,27
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
michael  pts/0    cpq              Wed00    1:06m  0.17s  0.03s sshd: michael [priv]
michael  pts/1    cpq              Wed00    0.00s  0.40s  0.07s sshd: michael [priv]

Attachment: signature.asc
Description: Digital signature


[Index of Archives]     [Linux RAID Wiki]     [ATA RAID]     [Linux SCSI Target Infrastructure]     [Linux Block]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device Mapper]     [Device Mapper Cryptographics]     [Kernel]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Yosemite Forum]


  Powered by Linux