On Fri, Jan 22, 2016 at 2:26 PM, Oliver Dzombic <info@xxxxxxxxxxxxxxxxx> wrote: > Hi Greg, > > from the client the list is huge: > > Thats the situation while the dd's are stuck. > > [root@cn201 ~]# ceph daemon /var/run/ceph/ceph-client.admin.asok > objecter_requests > { > "ops": [ > { > "tid": 12, > "pg": "6.7230bd94", > "osd": 1, > "object_id": "10000000017.00000004", > "object_locator": "@6", > "target_object_id": "10000000017.00000004", > "target_object_locator": "@6", > "paused": 0, > "used_replica": 0, > "precalc_pgid": 0, > "last_sent": "2016-01-22 23:11:28.788800", > "attempts": 95, > "snapid": "head", > "snap_context": "1=[]", > "mtime": "2016-01-21 23:41:18.001327", > "osd_ops": [ > "write 0~4194304 [5@0]" > ] Yeah, so all of these are operations which have been sent out to the OSDs but which haven't been acknowledged — about 300MB worth. Given that this one says 95 attempts, the OSD is dropping the write request without responding for some reason. You can look at the OSD logs to figure out why (you may need to turn log levels up; not sure). -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com