Re: Very high latency values

["Michael J. Kidd" <michael.kidd@xxxxxxxxxxx>]

Hello Dan,

  A couple of quick things...

* Latency is shown as a sum of all measured latencies over a period of time, and the count of operations included.  So to calculate the average per op latency, you must divide the sum by the count.  The result will be in milliseconds.

* The latency values you're showing there are from 'recoverystate_perf', meaning they're not relevant to normal operations of the OSDs.  For that, I'd recommend doing a perf dump against the OSD admin socket and looking at the latency values under the "osd" section.

* I've not seen any documentation on each counter, aside from occasional mailing list posts about specific counters..

Hope this helps!

Michael J. Kidd
Sr. Storage Consultant
Inktank Professional Services

On Fri, Mar 7, 2014 at 11:39 AM, Dan Ryder (daryder) <daryder@xxxxxxxxx> wrote:

Hello,

 

I’m working with two different Ceph clusters, and in both clusters, I’m seeing very high latency values.

 

Here’s part of a sample perf dump:

 

"recoverystate_perf": { "initial_latency": { "avgcount": 338,

          "sum": 0.069851000},

      "started_latency": { "avgcount": 1647,

          "sum": 322317122.940019000},

      "reset_latency": { "avgcount": 1985,

          "sum": 195.935076000},

      "start_latency": { "avgcount": 1985,

          "sum": 0.234355000},

      "primary_latency": { "avgcount": 266,

          "sum": 10819570.688122000},

 

You can see both started latency and primary latency have extremely high values.

 

Some info about the cluster:

All nodes are on the same subnet - 2 VMs, 1 physical node

VM1 is just a Monitor, VM2 is Monitor and OSD, Physical node is just an OSD.

 

 

One additional question, are these latency values in milliseconds? Is there any documentation on the units for perf dump command?

I’ve looked around but haven’t seen anything.

 

Thanks,

Dan

 

 

Dan Ryder ENGINEER.SOFTWARE ENGINEERING CSMTG Performance/Analytics daryder@xxxxxxxxx Phone: +1 919 392 7438	Cisco Systems, Inc. 7100-8 Kit Creek Road PO Box 14987 27709-4987 Research Triangle Park United States Cisco.com

 

 Think before you print.

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.

For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com