2018-04-23 15:16:04.101660 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.290071.4_XXXXXXX/XXXX/XX/XX/XXXXXXX.ZIP
2018-04-23 15:16:04.104231 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_1
2018-04-23 15:16:04.105541 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_2
2018-04-23 15:16:04.176235 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_3
2018-04-23 15:16:04.178435 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_4
2018-04-23 15:16:04.250883 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_5
2018-04-23 15:16:04.297912 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_6
2018-04-23 15:16:04.298803 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_7
2018-04-23 15:16:04.320202 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_8
2018-04-23 15:16:04.340124 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_9
2018-04-23 15:16:04.383924 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_10
2018-04-23 15:16:04.386865 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_11
2018-04-23 15:16:04.389067 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_12
2018-04-23 15:16:04.413938 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_13
2018-04-23 15:16:04.487977 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.bxz6tqhZzqZozTFkxPVspHfIhhVxaj5_14
2018-04-23 15:16:04.544235 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_1
2018-04-23 15:16:04.546978 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_2
2018-04-23 15:16:04.598644 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_3
2018-04-23 15:16:04.629519 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_4
2018-04-23 15:16:04.700492 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_5
2018-04-23 15:16:04.765798 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_6
2018-04-23 15:16:04.772774 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_7
2018-04-23 15:16:04.846379 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_8
2018-04-23 15:16:04.935023 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_9
2018-04-23 15:16:04.937229 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_10
2018-04-23 15:16:04.968289 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_11
2018-04-23 15:16:05.005194 7f1fdcc29a00 0 gc::process: removing .rgw.buckets:default.175209462.16__shadow_.06ry24pXQW8yH8EJpoqjEtZF6M6tiUv_12
We seem completely unable to get this deleted, and nothing else of immediate concern is flagging up as a potential cause of all RGWs become unresponsive at the same time. On the bucket containing this object (the one we originally tried to purge), I have attempted a further purge passing the “—bypass-gc” parameter to it, but this also resulted in all rgws becoming unresponsive within 30 minutes and so I terminated the operation and restarted the rgws again.
The bucket we attempted to remove has no shards and I have attached the details below. 90% of the contents of the bucket have already been successfully removed to our knowledge, and the bucket had no sharding (old bucket, sharding is now active for new buckets).
root@ceph-rgw-1:~# radosgw-admin --id rgw.ceph-rgw-1 bucket stats --bucket=xxxxxxxxxxxx
{
"bucket": "xxxxxxxxxxxx",
"pool": ".rgw.buckets",
"index_pool": ".rgw.buckets.index",
"id": "default.290071.4",
"marker": "default.290071.4",
"owner": "yyyyyyyyyy",
"ver": "0#107938549",
"master_ver": "0#0",
"mtime": "2014-10-24 14:58:48.955805",
"max_marker": "0#",
"usage": {
"rgw.none": {
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 0
},
"rgw.main": {
"size_kb": 186685939,
"size_kb_actual": 189914068,
"num_objects": 1419528
},
"rgw.multimeta": {
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 24
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
}
If anyone has any thoughts, they’d be greatly appreciated!
Kind Regards,
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
