On Wed, Mar 1, 2017 at 11:34 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote:
On Wed, 1 Mar 2017, Stephen Blinick wrote:
> Hello, I'm chasing down a situation where there's periodic slow requests
> occurring. While the specific version in this case is 0.80.7 Firefly, I
> think this log format is the same in newer versions. I can verify.
> There's a host of symptoms going on, but one strange anomaly I found that I
> wasn't able to chase down or find in search has to do with the Op parameters
> for an osd_op being logged as a slow request.
>
> Specifically, in some cases the byte range for an op of various types (i.e.
> read, writefull) is sometimes negative. Here's an example of two slow
> request log entries:
>
> #1
> 2017-02-28 18:39:09.943169 osd.27 10.0.1.84:6822/2402845 5255 : [WRN] slow
> request 16.440574 seconds old, received at 2017-02-28 18:38:53.502539:
> osd_op(client.2000529.0:496 ObjectNameOne [writefull 0~4194304] 3.3f26fcc9
> ondisk+write e691) v4 currently commit sent
>
> #2
> 2017-02-28 18:39:05.959253 osd.40 10.0.1.88:6831/2187230 6180 : [WRN] slow
> request 8.470175 seconds old, received at 2017-02-28 18:38:57.489045:
> osd_op(client.1941470.0:21164213 ObjectNameTwo [read 3670016~524288] This is quirky Ceph convention for printing extents as offset~length (it's
> 3.3a50c331 ack+read e691) v4 currently started
>
> As you can see, some of them show the byte range A~B where B is lower than
> A. I'm mostly interested to find out if this is an indication of any
not start~end). So these look fine.
Ahh I should have guessed. This makes a lot more sense. I'll update the parser accordingly. Thanks!
Firefly 0.80.7? You should really upgrade. Twice (to hammer and then to
jewel).
Indeed! We already have for the most part, but this system is in production, so that always adds friction to any upgrade. Definitely makes problem debug more of a 'forensic' exercise :)
sage
> problem. This is an EC pool, 3+2.
>
> Thanks,
>
> Stephen
>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
