Infernalis 9.2.1, Centos 72. My cluster is in recovery and i've noticed a lot of 'waiting for rw locks'. Some of these can last quite a long time. Any idea what can cause this?
Because this is a RGW bucket index file, this causes backup effects -- since the index can't be updated, S3 updates to other objects using that index fail because the client can't get the index. This is a problem with radosgw's implementation of the index metadata - when the cluster has issues, it affects more than just the object itself. It's basically twice the failure points for each object, if you're using radosgw since both the index and the rados object have to be valid and on good OSDs.
2016-05-06 21:22:40.193804 7f9b6bc22700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.484007 secs
2016-05-06 21:22:40.193810 7f9b6bc22700 0 log_channel(cluster) log [WRN] : slow request 30.484007 seconds old, received at 2016-05-06 21:22:09.709765: osd_op(client.45236419.0:481981 .dir.default.42048218.27.16 [call rgw.bucket_complete_op] 11.b9b84dfe ack+ondisk+write+known_if_redirected e105350) currently waiting for rw locks
From dump_ops_in_flight:
"description": "osd_op(client.45297841.0:387094 .dir.default.42048218.27.16 [call rgw.bucket_prepare_op] 11.b9b84dfe ondisk+write+known_if_redirected e105350)",
"initiated_at": "2016-05-06 21:25:51.911180",
"age": 5.121787,
"duration": 5.158513,
"type_data": [
"delayed",
{
"client": "client.45297841",
"tid": 387094
},
[
{
"time": "2016-05-06 21:25:51.911180",
"event": "initiated"
},
{
"time": "2016-05-06 21:25:51.911280",
"event": "queued_for_pg"
},
{
"time": "2016-05-06 21:25:51.940233",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:51.940265",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:52.059897",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:52.059925",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:52.262025",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:52.262056",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:52.358435",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:52.358958",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:52.806910",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:52.806930",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:52.947345",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:52.947357",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:53.131842",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:53.131860",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:53.323012",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:53.323031",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:53.800726",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:53.800744",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:54.260684",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:54.260716",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:54.794254",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:54.794270",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:55.198757",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:55.198775",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.019812",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.019830",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.119945",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.120256",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.167566",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.167581",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.199080",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.199092",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.217557",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.217571",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.251740",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.251873",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.319140",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.319230",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.350690",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.350708",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.370192",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.370222",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.407052",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.407083",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.441077",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.441094",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.506323",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.506343",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.535121",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.535143",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.589898",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.589916",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.675119",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.675150",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.720990",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.721014",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.767841",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.767984",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.792779",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.792801",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.863373",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.863481",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.921674",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.921875",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:56.975822",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:56.975847",
"event": "waiting for rw locks"
},
{
"time": "2016-05-06 21:25:57.030892",
"event": "reached_pg"
},
{
"time": "2016-05-06 21:25:57.030935",
"event": "waiting for rw locks"
}
]
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
