Ceph 9.2.1. Shortly after updating 9.2.0 to 9.2.1 all radosgws are refusing to start up, it's stuck on this 'notify' object:
[root@sm-cld-mtl-033 ceph]# ceph daemon /var/run/ceph/ceph-client.<>.asok objecter_requests
{
"ops": [
{
"tid": 13,
"pg": "4.88aa5c95",
"osd": 64,
"object_id": "notify.2",
"object_locator": "@4",
"target_object_id": "notify.2",
"target_object_locator": "@4",
"paused": 0,
"used_replica": 0,
"precalc_pgid": 0,
"last_sent": "2016-03-02 17:25:34.946304",
"attempts": 1,
"snapid": "head",
"snap_context": "0=[]",
"mtime": "2016-03-02 17:25:34.946149",
"osd_ops": [
"create 0~0"
]
}
-bash-4.2$ ceph pg map 4.88aa5c95
osdmap e66042 pg 4.88aa5c95 (4.95) -> up [64,99] acting [64,99]
on 64...
2016-03-02 17:20:52.671251 7fc42c437700 0 log_channel(cluster) log [WRN] : slow request 120.123788 seconds old, received at 2016-03-02 17:18:52.547397: osd_op(client.38825908.0:8291171 notify.2 [watch ping cookie 74579472 gen 106] 4.88aa5c95 ondisk+write+known_if_redirected e66040) currently waiting for peered
on 99 the object seems to exist... (though it's zero bytes)
-rw-r--r-- 1 root root 0 Jan 13 16:37 __head_00000095__4
-rw-r--r-- 1 root root 0 Mar 1 22:10 notify.2__head_88AA5C95__4
[root@sm-cld-mtl-025 4.95_head]# pwd
/var/lib/ceph/osd/ceph-99/current/4.95_head
On 64, that dir is empty.
We had one osd which went bad and was removed, which was involved in this pg.
Any next steps here? Are these 'notify' objects safe to nuke? I tried a repair/scrub on it, didnt seem to have an effect or log anywhere.
Any assistance is appreciated...
-Ben
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
