Hello Jason,
the command
# rados -p cephstor6 rm rbd_data.21aafa6b8b4567.0000000000000aaa
hangs as well. Doing absolutely nothing... waiting forever.
Greets,
Stefan
Am 17.05.2017 um 17:05 schrieb Jason Dillaman:
> OSD 23 notes that object rbd_data.21aafa6b8b4567.0000000000000aaa is
> waiting for a scrub. What happens if you run "rados -p <rbd pool> rm
> rbd_data.21aafa6b8b4567.0000000000000aaa" (capturing the OSD 23 logs
> during this)? If that succeeds while your VM remains blocked on that
> remove op, it looks like there is some problem in the OSD where ops
> queued on a scrub are not properly awoken when the scrub completes.
>
> On Wed, May 17, 2017 at 10:57 AM, Stefan Priebe - Profihost AG
> <s.priebe@xxxxxxxxxxxx> wrote:
>> Hello Jason,
>>
>> after enabling the log and generating a gcore dump, the request was
>> successful ;-(
>>
>> So the log only contains the successfull request. So i was only able to
>> catch the successful request. I can send you the log on request.
>>
>> Luckily i had another VM on another Cluster behaving the same.
>>
>> This time osd.23:
>> # ceph --admin-daemon
>> /var/run/ceph/ceph-client.admin.22969.140085040783360.asok
>> objecter_requests
>> {
>> "ops": [
>> {
>> "tid": 18777,
>> "pg": "2.cebed0aa",
>> "osd": 23,
>> "object_id": "rbd_data.21aafa6b8b4567.0000000000000aaa",
>> "object_locator": "@2",
>> "target_object_id": "rbd_data.21aafa6b8b4567.0000000000000aaa",
>> "target_object_locator": "@2",
>> "paused": 0,
>> "used_replica": 0,
>> "precalc_pgid": 0,
>> "last_sent": "1.83513e+06s",
>> "attempts": 1,
>> "snapid": "head",
>> "snap_context": "28a43=[]",
>> "mtime": "2017-05-17 16:51:06.0.455475s",
>> "osd_ops": [
>> "delete"
>> ]
>> }
>> ],
>> "linger_ops": [
>> {
>> "linger_id": 1,
>> "pg": "2.f0709c34",
>> "osd": 23,
>> "object_id": "rbd_header.21aafa6b8b4567",
>> "object_locator": "@2",
>> "target_object_id": "rbd_header.21aafa6b8b4567",
>> "target_object_locator": "@2",
>> "paused": 0,
>> "used_replica": 0,
>> "precalc_pgid": 0,
>> "snapid": "head",
>> "registered": "1"
>> }
>> ],
>> "pool_ops": [],
>> "pool_stat_ops": [],
>> "statfs_ops": [],
>> "command_ops": []
>> }
>>
>> OSD Logfile of OSD 23 attached.
>>
>> Greets,
>> Stefan
>>
>> Am 17.05.2017 um 16:26 schrieb Jason Dillaman:
>>> On Wed, May 17, 2017 at 10:21 AM, Stefan Priebe - Profihost AG
>>> <s.priebe@xxxxxxxxxxxx> wrote:
>>>> You mean the request no matter if it is successful or not? Which log
>>>> level should be set to 20?
>>>
>>>
>>> I'm hoping you can re-create the hung remove op when OSD logging is
>>> increased -- "debug osd = 20" would be nice if you can turn it up that
>>> high while attempting to capture the blocked op.
>>>
>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
