Hi experts.In one of my ceph cluster, some of my OSDs has dramatically slow IOPs when executing tell bench command. All of the OSDs are SSD and we have 2 types of SSD disks: 1) ordinary SSD 2) Enterprise SSDs
In my cluster, there are several OSDs of type ordinary SSD with very slow iops.The result of tell bench is as follows:
ceph tell osd.6 bench -f plain{ "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 14.029725770000001, "bytes_per_sec": 76533343.673473671, "iops": 2.24697105252115}
There are no heavy disk load on my storage server when executing iostat command:avg-cpu: %user %nice %system %iowait %steal %idle 14.79 0.00 3.96 3.44 0.00 77.81
Device: tps kB_read/s kB_wrtn/s sda 9.83 1.12 114.09 sdd 14.52 303.60 239.83 sdf 24.14 608.69 296.31
sdg 22.02 580.63 271.68
I zap & wipe OSD.6 several times but nothing changed and the IOPs is in the order of 2. But in my cluster there are several OSDs with the same hardware conditions but results 300 IOPs!
What is the problem? Thanks ceph experts for helping me to find the problem.
Extra information about the cluster listed in the followings:
My Cluster information: services: mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-2,overcloud-controller-1 (age 3d) mgr: overcloud-controller-1(active, since 8d), standbys: overcloud-controller-0, overcloud-controller-2 osd: 52 osds: 52 up (since 3h), 38 in (since 4d) flags noout data: pools: 5 pools, 1825 pgs objects: 2.73M objects, 11 TiB usage: 32 TiB used, 31 TiB / 63 TiB avail pgs: 1825 active+clean io: client: 1.1 MiB/s rd, 6.6 MiB/s wr, 1.02k op/s rd, 686 op/s wr
9 super-ssd 1.00000 1.00000 1.7 TiB 1.2 TiB 1.1 TiB 7 MiB 7.0 GiB 606 GiB 66.11 1.04 204 up10 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 10 MiB 5.8 GiB 427 GiB 76.12 1.19 230 up15 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 9.4 MiB 8.8 GiB 431 GiB 75.91 1.19 230 up39 super-ssd 1.00000 1.00000 1.7 TiB 1.4 TiB 1.3 TiB 4.8 MiB 5.7 GiB 405 GiB 77.34 1.21 233 up11 super-ssd 1.00000 0.95001 1.7 TiB 1.3 TiB 1.3 TiB 7.8 MiB 6.1 GiB 473 GiB 73.55 1.15 221 up
12 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 9.9 MiB 7.0 GiB 421 GiB 76.44 1.20 230 up40 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 13 MiB 6.0 GiB 458 GiB 74.40 1.17 227 up41 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 2.6 MiB 5.9 GiB 463 GiB 74.10 1.16 227 up13 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 7.7 MiB 4.1 GiB 472 GiB 73.63 1.16 225 up14 super-ssd 1.00000 0.90002 1.7 TiB 1.3 TiB 1.3 TiB 8.5 MiB 5.7 GiB 447 GiB 75.01 1.18 226 up42 super-ssd 1.00000 1.00000 1.7 TiB 1.5 TiB 1.5 TiB 9.8 MiB 4.9 GiB 279 GiB 84.39 1.32 255 up43 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 7.3 MiB 5.0 GiB 494 GiB 72.38 1.14 219 up44 super-ssd 1.00000 0.95001 1.7 TiB 1.3 TiB 1.3 TiB 11 MiB 7.4 GiB 466 GiB 73.92 1.16 226 up45 super-ssd 1.00000 0.95001 1.7 TiB 1.3 TiB 1.3 TiB 3.2 MiB 6.4 GiB 488 GiB 72.73 1.14 217 up46 super-ssd 1.00000 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 1.8 MiB 8.0 GiB 315 GiB 82.38 1.29 249 up47 super-ssd 1.00000 1.00000 1.7 TiB 1.2 TiB 1.2 TiB 4.4 MiB 5.7 GiB 532 GiB 70.23 1.10 213 up48 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 4.6 MiB 4.6 GiB 422 GiB 76.42 1.20 234 up49 super-ssd 1.00000 0.95001 1.7 TiB 1.4 TiB 1.4 TiB 2.3 MiB 6.4 GiB 387 GiB 78.35 1.23 234 up50 super-ssd 1.00000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 9.6 MiB 8.9 GiB 429 GiB 76.03 1.19 231 up51 super-ssd 1.00000 0.70000 1.7 TiB 1.0 TiB 1.0 TiB 6.0 MiB 5.6 GiB 737 GiB 58.79 0.92 179 up17 super-ssd 0.09999 1.00000 500 GiB 154 GiB 153 GiB 281 KiB 1024 MiB 346 Gi
B 30.74 0.48 26 up18 super-ssd 0.39999 1.00000 931 GiB 527 GiB 525 GiB 5.8 MiB 1.6 GiB 405 GiB 56.53 0.89 90 up31 ssd 0.50000 1.00000 931 GiB 261 GiB 260 GiB 2.8 MiB 1.1 GiB 671 GiB 27.99 0.44 40 up20 ssd 0.43660 1.00000 447 GiB 271 GiB 270 GiB 5.2 MiB 1.0 GiB 176 GiB 60.64 0.95 40 up21 ssd 0.90959 1.00000 931 GiB 466 GiB 464 GiB 4.4 MiB 2.0 GiB 466 GiB 49.99 0.78 69 up36 ssd 0.50000 1.00000 931 GiB 324 GiB 323 GiB 4.0 MiB 1.0 GiB 608 GiB 34.75 0.55 49 up 0 ssd 0.50000 1.00000 931 GiB 321 GiB 320 GiB 5.5 MiB 1.1 GiB 610 GiB 34.49 0.54 49 up
2 ssd 0.50000 1.00000 931 GiB 313 GiB 312 GiB 9.7 MiB 1.2 GiB 619 GiB 33.60 0.53 52 up 4 ssd 0.50000 1.00000 931 GiB 248 GiB 246 GiB 2.0 MiB 1.3 GiB 684 GiB 26.61 0.42 39 up23 ssd 0.90959 1.00000 931 GiB 476 GiB 475 GiB 7.2 MiB 1.3 GiB 455 GiB 51.13 0.80 69 up26 ssd 0.50000 1.00000 931 GiB 265 GiB 264 GiB 3.5 MiB 1.0 GiB 666 GiB 28.47 0.45 39 up28 ssd 0.39999 1.00000 931 GiB 256 GiB 255 GiB 1.4 MiB 1.4 GiB 675 GiB 27.50 0.43 40 up29 ssd 0.50000 1.00000 931 GiB 239 GiB 238 GiB 9.7 MiB 1.1 GiB 692 GiB 25.69 0.40 37 up30 ssd 0.90959 1.00000 931 GiB 460 GiB 459 GiB 5.2 MiB 1.3 GiB 471 GiB 49.39 0.77 69 up38 ssd 0.90959 1.00000 931 GiB 499 GiB 498 GiB 7.3 MiB 1.4 GiB 432 GiB 53.58 0.84 76 up16 ssd 0.90959 1.00000 931 GiB 532 GiB 531 GiB 2.6 MiB 1.4 GiB 400 GiB 57.07 0.90 78 up19 ssd 0.50000 1.00000 931 GiB 360 GiB 359 GiB 3.9 MiB 1.3 GiB 572 GiB 38.64 0.61 53 up22 ssd 0.50000 1.00000 931 GiB 308 GiB 307 GiB 4.4 MiB 1.0 GiB 624 GiB 33.03 0.52 49 up
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
