Hi List,
Three of our Ceph OSDs got unreasonably high latency right after the first second of the new year (2017/01/01 00:00:00 UTC, I have attached the metrics and I am in UTC+8 timezone). There is exactly a pg (size=3) just contains these 3 OSDs.
The OSD apply latency is usually up to 25 minutes, and I can also see this large number randomly when I execute "ceph osd perf" command. But the 3 OSDs does not have strange behavior and are performing fine so far.
I have no idea how "ceph osd perf" is implemented, but does it have relation to the leap second this year? Since the cluster is not on production, and the developers were all celebrating new year at that time, I can not think of other possibilities.
Do your cluster also get this interestingly unexpected new year's gift too?
Three of our Ceph OSDs got unreasonably high latency right after the first second of the new year (2017/01/01 00:00:00 UTC, I have attached the metrics and I am in UTC+8 timezone). There is exactly a pg (size=3) just contains these 3 OSDs.
The OSD apply latency is usually up to 25 minutes, and I can also see this large number randomly when I execute "ceph osd perf" command. But the 3 OSDs does not have strange behavior and are performing fine so far.
I have no idea how "ceph osd perf" is implemented, but does it have relation to the leap second this year? Since the cluster is not on production, and the developers were all celebrating new year at that time, I can not think of other possibilities.
Do your cluster also get this interestingly unexpected new year's gift too?
Sincerely,
Craig Chi
Craig Chi
Attachment:
osd_apply_latency.png
Description: PNG image
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
