Hi Charles,
Looks to me like all of the drives are performing roughly the same – there’s certainly not 1 that sticks out (also note this is happening on all 23 nodes in the cluster).
Thanks!
Kelly
[root@r1k1.kafka.log10.blackberry sys] # cat /proc/mdstat
Personalities : [raid10]
md127 : active raid10 sdc[2] sdh[7] sdb[1] sdf[5] sde[13] sdg[12] sdj[9] sdk[10] sda[0] sdl[11] sdd[3] sdi[8]
23441323008 blocks super 1.2 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
[>....................] check = 0.0% (618944/23441323008) finish=108288.4min speed=3607K/sec
unused devices: <none>
[root@r1k1.kafka.log10.blackberry sys] # iostat -xdmc 1 10
Linux 3.10.0-327.18.2.el7.x86_64 (r1k1.kafka.log10.blackberry) 05/26/16 _x86_64_ (32 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
12.76 0.07 2.48 0.16 0.00 84.53
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.01 0.56 0.26 26.44 0.06 11.30 871.39 9.67 362.22 5.14 365.71 6.68 17.83
sdk 0.01 0.56 0.26 26.56 0.06 11.30 867.70 9.53 355.45 5.05 358.84 6.58 17.65
sdc 0.01 0.46 0.26 26.34 0.06 11.29 874.67 9.73 365.89 4.86 369.38 6.81 18.11
sdd 0.01 0.46 0.20 26.34 0.07 11.29 876.98 10.40 391.99 5.33 394.93 7.17 19.02
sda 0.01 0.49 0.26 26.53 0.06 11.29 868.24 9.48 353.91 4.96 357.36 6.57 17.61
sdj 0.01 0.56 0.20 26.44 0.07 11.30 873.73 10.04 376.87 5.48 379.68 6.91 18.40
sdl 0.01 0.56 0.20 26.56 0.07 11.30 869.99 9.77 365.16 5.92 367.92 6.72 17.99
sdh 0.01 0.57 0.21 26.79 0.07 11.30 862.30 9.65 357.60 5.27 360.31 6.63 17.90
sde 0.01 0.47 0.26 26.13 0.06 11.29 881.38 10.60 401.47 6.62 405.41 7.35 19.41
sdf 0.01 0.47 0.20 26.13 0.07 11.29 883.71 9.53 361.85 5.24 364.64 6.73 17.73
sdg 0.01 0.57 0.26 26.79 0.06 11.30 859.99 10.15 375.20 5.26 378.82 6.86 18.57
sdb 0.01 0.49 0.20 26.53 0.07 11.29 870.69 9.85 368.48 5.35 371.23 6.79 18.15
md127 0.00 0.00 2.51 156.82 0.77 67.77 881.06 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
25.51 0.03 4.37 1.05 0.00 69.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 1.00 8.00 30.00 0.50 14.18 791.16 1.06 28.03 0.50 35.37 6.97 26.50
sdk 0.00 0.00 8.00 30.00 0.50 14.52 809.47 0.93 24.32 0.00 30.80 7.87 29.90
sdc 0.00 1.00 9.00 32.00 0.56 15.21 787.90 1.13 27.54 0.67 35.09 6.90 28.30
sdd 0.00 1.00 10.00 32.00 0.62 15.21 772.19 1.29 30.69 0.70 40.06 6.76 28.40
sda 0.00 0.00 8.00 38.00 0.50 15.54 714.09 1.40 30.35 0.38 36.66 7.91 36.40
sdj 0.00 1.00 8.00 30.00 0.50 14.18 791.16 1.05 27.68 0.50 34.93 7.00 26.60
sdl 0.00 0.00 8.00 30.00 0.50 14.52 809.47 0.90 23.61 0.25 29.83 7.66 29.10
sdh 0.00 1.00 13.00 34.00 0.81 14.11 650.04 1.17 24.98 0.31 34.41 6.60 31.00
sde 0.00 0.00 16.00 31.00 1.00 14.54 676.94 1.20 25.45 0.31 38.42 7.13 33.50
sdf 0.00 0.00 16.00 31.00 1.00 14.54 676.94 1.19 25.38 0.31 38.32 5.57 26.20
sdg 0.00 1.00 13.00 34.00 0.81 14.11 650.04 1.22 25.98 0.31 35.79 6.70 31.50
sdb 0.00 0.00 8.00 38.00 0.50 15.54 714.09 1.31 28.41 0.25 34.34 8.02 36.90
md127 0.00 0.00 0.00 198.00 0.00 86.59 895.60 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
21.31 0.00 2.99 0.00 0.00 75.69
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 8.00 7.00 0.50 3.50 546.13 0.13 8.47 6.25 11.00 8.47 12.70
sdk 0.00 0.00 8.00 8.00 0.50 3.98 574.00 0.16 9.88 1.00 18.75 10.06 16.10
sdc 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.12 7.25 0.62 13.88 7.25 11.60
sdd 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.12 7.44 0.50 14.38 7.44 11.90
sda 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.13 8.00 0.50 15.50 8.00 12.80
sdj 0.00 0.00 8.00 7.00 0.50 3.50 546.13 0.18 12.20 9.25 15.57 12.20 18.30
sdl 0.00 0.00 8.00 9.00 0.50 4.48 600.47 0.11 6.94 1.00 12.22 6.59 11.20
sdh 0.00 0.00 8.00 9.00 0.50 3.51 482.82 0.10 6.12 0.50 11.11 6.12 10.40
sde 0.00 0.00 8.00 9.00 0.50 4.00 542.59 0.16 9.65 0.25 18.00 9.65 16.40
sdf 0.00 0.00 8.00 9.00 0.50 4.00 542.59 0.13 7.65 0.25 14.22 7.65 13.00
sdg 0.00 0.00 8.00 9.00 0.50 3.51 482.82 0.13 7.59 0.50 13.89 7.59 12.90
sdb 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.11 6.62 2.12 11.12 6.62 10.60
md127 0.00 0.00 0.00 49.00 0.00 23.00 961.14 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
18.70 4.21 4.24 0.00 0.00 72.85
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 8.00 15.00 0.50 6.09 586.78 0.22 9.17 0.25 13.93 6.26 14.40
sdk 0.00 0.00 8.00 14.00 0.50 5.55 563.27 0.25 11.68 2.38 17.00 7.59 16.70
sdc 0.00 0.00 8.00 13.00 0.50 6.50 682.67 0.15 7.00 0.25 11.15 6.00 12.60
sdd 0.00 0.00 8.00 13.00 0.50 6.50 682.67 0.17 7.95 0.25 12.69 6.86 14.40
sda 0.00 0.00 8.00 14.00 0.50 6.50 652.00 0.26 11.77 0.62 18.14 7.86 17.30
sdj 0.00 0.00 8.00 15.00 0.50 6.09 586.78 0.34 14.35 2.00 20.93 9.87 22.70
sdl 0.00 0.00 8.00 13.00 0.50 5.05 541.33 0.25 11.86 0.50 18.85 7.57 15.90
sdh 0.00 0.00 10.00 17.00 0.62 7.14 589.04 0.33 12.19 0.60 19.00 7.41 20.00
sde 0.00 0.00 8.00 18.00 0.50 6.68 565.85 0.31 11.77 0.25 16.89 7.00 18.20
sdf 0.00 0.00 8.00 18.00 0.50 6.68 565.85 0.42 16.12 2.25 22.28 9.96 25.90
sdg 0.00 0.00 10.00 17.00 0.62 7.14 589.04 0.33 12.30 0.60 19.18 6.59 17.80
sdb 0.00 0.00 8.00 14.00 0.50 6.50 652.00 0.27 12.14 2.25 17.79 8.00 17.60
md127 0.00 0.00 0.00 91.00 0.00 38.47 865.76 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
16.69 0.03 3.08 0.03 0.00 80.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 18.00 14.00 1.12 7.00 520.00 0.15 4.84 0.50 10.43 4.62 14.80
sdk 0.00 0.00 16.00 14.00 1.00 7.00 546.13 0.14 4.77 0.38 9.79 4.77 14.30
sdc 0.00 0.00 16.00 13.00 1.00 6.50 529.66 0.14 5.00 0.38 10.69 5.00 14.50
sdd 0.00 0.00 16.00 13.00 1.00 6.50 529.66 0.15 5.10 0.38 10.92 5.10 14.80
sda 0.00 0.00 16.00 18.00 1.00 7.54 514.59 0.21 6.12 1.31 10.39 6.26 21.30
sdj 0.00 0.00 18.00 14.00 1.12 7.00 520.00 0.13 4.25 0.50 9.07 4.03 12.90
sdl 0.00 0.00 16.00 14.00 1.00 7.00 546.13 0.13 4.47 0.31 9.21 4.47 13.40
sdh 7.00 0.00 10.00 13.00 1.06 6.50 673.39 0.10 4.57 0.50 7.69 4.57 10.50
sde 6.00 0.00 10.00 13.00 1.00 6.50 667.83 0.15 6.35 0.60 10.77 6.35 14.60
sdf 6.00 0.00 10.00 13.00 1.00 6.50 667.83 0.14 6.22 0.60 10.54 6.22 14.30
sdg 7.00 0.00 10.00 13.00 1.06 6.50 673.39 0.10 4.39 0.50 7.38 4.39 10.10
sdb 0.00 0.00 16.00 19.00 1.00 7.57 501.71 0.13 3.77 0.31 6.68 3.77 13.20
md127 0.00 0.00 0.00 85.00 0.00 40.57 977.60 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
22.73 0.00 5.91 0.06 0.00 71.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 242.00 0.00 1048.00 1.00 80.62 0.01 157.43 4.38 4.18 4.17 8.00 0.37 38.50
sdk 334.00 0.00 954.00 0.00 80.50 0.00 172.81 7.27 7.62 7.62 0.00 0.45 43.20
sdc 294.00 0.00 994.00 0.00 80.50 0.00 165.86 5.56 5.59 5.59 0.00 0.40 39.80
sdd 249.00 0.00 1039.00 0.00 80.50 0.00 158.68 4.11 3.95 3.95 0.00 0.37 38.00
sda 268.00 0.00 1020.00 11.00 80.50 0.18 160.26 5.47 5.31 5.14 21.36 0.58 60.20
sdj 253.00 0.00 1037.00 1.00 80.62 0.01 159.10 4.42 4.26 4.26 3.00 0.37 38.80
sdl 257.00 0.00 1031.00 0.00 80.50 0.00 159.91 5.13 4.98 4.98 0.00 0.37 38.30
sdh 224.00 0.00 1064.00 1.00 80.50 0.00 154.81 3.80 3.57 3.57 10.00 0.36 38.30
sde 247.00 0.00 1041.00 0.00 80.50 0.00 158.37 4.96 4.77 4.77 0.00 0.37 38.60
sdf 220.00 0.00 1068.00 0.00 80.50 0.00 154.37 3.70 3.47 3.47 0.00 0.33 35.40
sdg 242.00 0.00 1046.00 1.00 80.50 0.00 157.47 5.05 4.82 4.81 13.00 0.39 40.80
sdb 239.00 0.00 1049.00 10.00 80.50 0.15 155.97 4.77 4.51 4.43 12.10 0.45 47.90
md127 0.00 0.00 0.00 13.00 0.00 0.17 26.46 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
28.14 0.03 6.00 0.00 0.00 65.83
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 5.00 12.00 37.00 0.75 0.29 43.27 1.13 23.04 0.67 30.30 4.98 24.40
sdk 0.00 17.00 16.00 34.00 1.00 0.34 54.88 2.95 59.02 0.56 86.53 3.98 19.90
sdc 0.00 4.00 8.00 33.00 0.50 0.25 37.66 1.14 27.88 0.25 34.58 5.66 23.20
sdd 0.00 4.00 8.00 33.00 0.50 0.25 37.66 0.52 12.83 0.25 15.88 4.02 16.50
sda 0.00 3.00 16.00 21.00 1.00 0.17 64.86 0.26 7.14 1.06 11.76 3.92 14.50
sdj 0.00 5.00 12.00 37.00 0.75 0.29 43.27 0.84 17.24 0.42 22.70 4.47 21.90
sdl 0.00 17.00 16.00 34.00 1.00 0.34 54.88 2.98 59.56 0.56 87.32 3.92 19.60
sdh 0.00 4.00 8.00 26.00 0.50 0.20 41.88 0.67 19.71 1.75 25.23 4.50 15.30
sde 0.00 4.00 8.00 22.00 0.50 0.19 47.20 0.39 12.83 2.38 16.64 3.93 11.80
sdf 0.00 4.00 8.00 22.00 0.50 0.19 47.20 0.35 11.60 2.25 15.00 3.67 11.00
sdg 0.00 4.00 8.00 26.00 0.50 0.20 41.88 0.67 19.62 1.50 25.19 4.12 14.00
sdb 0.00 3.00 16.00 21.00 1.00 0.17 64.86 0.42 11.27 1.00 19.10 6.73 24.90
md127 0.00 0.00 0.00 210.00 0.00 1.44 14.06 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
35.57 0.00 10.34 0.00 0.00 54.08
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 9.00 11.00 0.56 0.04 62.00 0.14 7.00 1.78 11.27 6.50 13.00
sdk 0.00 0.00 8.00 11.00 0.50 0.05 58.95 0.10 5.26 0.88 8.45 5.26 10.00
sdc 0.00 0.00 16.00 16.00 1.00 0.07 68.75 0.17 5.44 0.44 10.44 5.44 17.40
sdd 0.00 0.00 16.00 16.00 1.00 0.07 68.75 0.17 5.38 1.00 9.75 5.38 17.20
sda 0.00 0.00 8.00 16.00 0.50 0.06 48.08 0.20 8.54 0.75 12.44 5.42 13.00
sdj 0.00 0.00 9.00 11.00 0.56 0.04 62.00 0.13 6.65 0.44 11.73 5.90 11.80
sdl 0.00 0.00 8.00 11.00 0.50 0.05 58.95 0.12 6.16 1.62 9.45 6.16 11.70
sdh 0.00 0.00 16.00 30.00 1.00 0.11 49.63 0.32 6.85 0.44 10.27 4.39 20.20
sde 0.00 0.00 16.00 6.00 1.00 0.02 95.27 0.10 4.41 0.44 15.00 4.41 9.70
sdf 0.00 0.00 16.00 6.00 1.00 0.02 95.27 0.14 6.59 4.06 13.33 6.55 14.40
sdg 0.00 0.00 16.00 29.00 1.00 0.11 50.56 0.31 6.80 0.44 10.31 4.82 21.70
sdb 0.00 0.00 8.00 16.00 0.50 0.06 48.08 0.24 10.17 0.62 14.94 6.75 16.20
md127 0.00 0.00 0.00 89.00 0.00 0.36 8.24 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
35.50 0.03 7.43 0.00 0.00 57.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 74.00 0.00 21.00 7.00 5.94 1.53 546.00 0.47 16.71 17.57 14.14 4.89 13.70
sdk 70.00 0.00 26.00 10.00 6.00 1.41 421.56 0.41 10.94 11.73 8.90 4.33 15.60
sdc 77.00 0.00 11.00 9.00 5.50 1.57 723.60 0.64 32.00 42.64 19.00 10.65 21.30
sdd 77.00 0.00 11.00 9.00 5.50 1.57 723.60 1.19 59.60 96.36 14.67 12.10 24.20
sda 71.00 1.00 24.00 11.00 5.94 1.53 437.09 0.51 14.46 14.38 14.64 5.09 17.80
sdj 74.00 0.00 21.00 7.00 5.94 1.53 546.00 0.58 20.79 20.57 21.43 7.04 19.70
sdl 70.00 0.00 26.00 11.00 6.00 1.91 437.84 0.39 10.54 11.04 9.36 4.32 16.00
sdh 77.00 0.00 11.00 7.00 5.50 1.52 798.67 0.43 24.17 33.82 9.00 6.61 11.90
sde 77.00 0.00 11.00 6.00 5.50 1.52 845.18 0.58 34.24 36.91 29.33 13.71 23.30
sdf 77.00 0.00 11.00 6.00 5.50 1.52 845.18 0.60 35.35 45.36 17.00 10.06 17.10
sdg 77.00 0.00 11.00 8.00 5.50 1.52 757.05 0.43 22.95 32.00 10.50 6.89 13.10
sdb 71.00 1.00 24.00 11.00 5.94 1.53 437.09 0.60 17.14 13.67 24.73 9.03 31.60
md127 0.00 0.00 0.00 52.00 0.00 9.57 376.96 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
27.06 0.03 6.00 0.00 0.00 66.91
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdi 14.00 0.00 10.00 9.00 1.50 4.06 599.58 0.13 6.84 2.60 11.56 6.63 12.60
sdk 14.00 0.00 10.00 10.00 1.50 5.00 665.60 0.13 7.05 2.50 11.60 6.35 12.70
sdc 14.00 0.00 11.00 10.00 1.56 4.01 543.24 0.15 7.33 1.00 14.30 7.24 15.20
sdd 14.00 0.00 11.00 10.00 1.56 4.01 543.24 0.15 7.14 1.00 13.90 7.05 14.80
sda 14.00 0.00 11.00 10.00 1.56 4.20 561.52 0.12 5.38 0.91 10.30 5.43 11.40
sdj 14.00 0.00 10.00 9.00 1.50 4.06 599.58 0.26 13.68 3.60 24.89 13.47 25.60
sdl 14.00 0.00 10.00 9.00 1.50 4.50 646.74 0.13 6.63 1.30 12.56 6.47 12.30
sdh 13.00 0.00 11.00 9.00 1.50 4.00 563.60 0.11 5.70 1.18 11.22 5.55 11.10
sde 14.00 0.00 10.00 8.00 1.50 4.00 625.78 0.09 4.78 1.10 9.38 4.67 8.40
sdf 14.00 0.00 10.00 8.00 1.50 4.00 625.78 0.14 8.06 4.00 13.12 7.17 12.90
sdg 13.00 0.00 11.00 9.00 1.50 4.00 563.60 0.14 7.00 1.91 13.22 6.80 13.60
sdb 14.00 0.00 11.00 10.00 1.56 4.20 561.52 0.17 7.67 1.73 14.20 7.71 16.20
md127 0.00 0.00 0.00 56.00 0.00 25.27 924.14 0.00 0.00 0.00 0.00 0.00 0.00
On 2016-05-25, 5:43 PM, "centos-bounces@xxxxxxxxxx on behalf of cpolish@xxxxxxxxxxxx" <centos-bounces@xxxxxxxxxx on behalf of cpolish@xxxxxxxxxxxx> wrote:
>On 2016-05-25 19:13, Kelly Lesperance wrote:
>> Hdparm didn’t get far:
>>
>> [root@r1k1 ~] # hdparm -tT /dev/sda
>>
>> /dev/sda:
>> Timing cached reads: Alarm clock
>> [root@r1k1 ~] #
>
>Hi Kelly,
>
>Try running 'iostat -xdmc 1'. Look for a single drive that has
>substantially greater await than ~10msec. If all the drives
>except one are taking 6-8msec, but one is very much more, you've
>got a drive that drags down the whole array's performance.
>
>Ignore the very first output from the command - it's an
>average of the disk subsystem since boot.
>
>Post a representative output along with the contents /proc/mdstat.
>
>Good luck,
>--
>Charles Polisher
>
>_______________________________________________
>CentOS mailing list
>CentOS@xxxxxxxxxx
>https://lists.centos.org/mailman/listinfo/centos
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
