slow requests on a specific osd

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On our three-node 24 OSDs ceph 10.2.10 cluster, we have started seeing slow requests on a specific OSD, during the the two-hour nightly xfs_fsr run from 05:00 - 07:00. This started after we applied the meltdown patches.

The specific osd.10 also has the highest space utilization of all OSDs cluster-wide, with 45%, while the others are mostly around 40%. All OSDs are the same 4TB platters with journal on ssd, all with weight 1.

Smart info for osd.10 shows nothing interesting I think:

Current Drive Temperature:     27 C
Drive Trip Temperature:        60 C

Manufactured in week 04 of year 2016
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  53
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  697
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 1933129649
  Blocks received from initiator = 869206640
  Blocks read from cache and sent to initiator = 2149311508
  Number of read and write commands whose size <= segment size = 676356809
  Number of read and write commands whose size > segment size = 12734900

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 13625.88
  number of minutes until next internal SMART test = 8

Now my question:
Could it be that osd.10 just happens to contain some data chunks that are heavily needed by the VMs around that time, and that the added load of an xfs_fsr is simply too much for it to handle?

In that case, how about reweighting that osd.10 to "0", wait until all data has moved off osd.10, and then setting it back to "1". Would this result in *exactly* the same situation as before, or would it at least cause the data to have spread move better across the other OSDs?

(with the idea that better data spread across OSDs brings also better distribution of load between the OSDs)

Or other ideas to check out?

MJ
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux