Wido,
I've been looking into this large omap objects problem on a couple of our clusters today and came across your script during my research.
The script has been running for a few hours now and I'm already over 100,000 'orphaned' objects!
It appears that ever since upgrading to Luminous (12.2.5 initially, followed by 12.2.8) this cluster has been resharding the large bucket indexes at least once a day and not cleaning up the previous bucket indexes:
for instance in $(radosgw-admin metadata list bucket.instance | jq -r '.[]' | grep go-test-dashboard); do
mtime=$(radosgw-admin metadata get bucket.instance:${instance} | grep mtime)
num_shards=$(radosgw-admin metadata get bucket.instance:${instance} | grep num_shards)
echo "${instance}: ${mtime} ${num_shards}"
done | column -t | sort -k3
go-test-dashboard:default.188839135.327804: "mtime": "2018-06-01 22:35:28.693095Z", "num_shards": 0,
go-test-dashboard:default.617828918.2898: "mtime": "2018-06-02 22:35:40.438738Z", "num_shards": 46,
go-test-dashboard:default.617828918.4: "mtime": "2018-06-02 22:38:21.537259Z", "num_shards": 46,
go-test-dashboard:default.617663016.10499: "mtime": "2018-06-03 23:00:04.185285Z", "num_shards": 46,
[...snip...]
go-test-dashboard:default.891941432.342061: "mtime": "2018-11-28 01:41:46.777968Z", "num_shards": 7,
go-test-dashboard:default.928133068.2899: "mtime": "2018-11-28 20:01:49.390237Z", "num_shards": 46,
go-test-dashboard:default.928133068.5115: "mtime": "2018-11-29 01:54:17.788355Z", "num_shards": 7,
go-test-dashboard:default.928133068.8054: "mtime": "2018-11-29 20:21:53.733824Z", "num_shards": 7,
go-test-dashboard:default.891941432.359004: "mtime": "2018-11-29 20:22:09.201965Z", "num_shards": 46,
The num_shards is typically around 46, but looking at all 288 instances of that bucket index, it has varied between 3 and 62 shards.
Have you figured anything more out about this since you posted this originally two weeks ago?
Thanks,
Bryan
From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Wido den Hollander <wido@xxxxxxxx>
Date: Thursday, November 15, 2018 at 5:43 AM
To: Ceph Users <ceph-users@xxxxxxxx>
Subject: Removing orphaned radosgw bucket indexes from pool
Hi,
Recently we've seen multiple messages on the mailinglists about people
seeing HEALTH_WARN due to large OMAP objects on their cluster. This is
due to the fact that starting with 12.2.6 OSDs warn about this.
I've got multiple people asking me the same questions and I've done some
digging around.
Somebody on the ML wrote this script:
for bucket in `radosgw-admin metadata list bucket | jq -r '.[]' | sort`; do
actual_id=`radosgw-admin bucket stats --bucket=${bucket} | jq -r '.id'`
for instance in `radosgw-admin metadata list bucket.instance | jq -r
'.[]' | grep ${bucket}: | cut -d ':' -f 2`
do
if [ "$actual_id" != "$instance" ]
then
radosgw-admin bi purge --bucket=${bucket} --bucket-id=${instance}
radosgw-admin metadata rm bucket.instance:${bucket}:${instance}
fi
done
done
That partially works, but 'orphaned' objects in the index pool do not work.
So I wrote my own script [0]:
#!/bin/bash
INDEX_POOL=$1
if [ -z "$INDEX_POOL" ]; then
echo "Usage: $0 <index pool>"
exit 1
fi
INDEXES=$(mktemp)
METADATA=$(mktemp)
trap "rm -f ${INDEXES} ${METADATA}" EXIT
radosgw-admin metadata list bucket.instance|jq -r '.[]' > ${METADATA}
rados -p ${INDEX_POOL} ls > $INDEXES
for OBJECT in $(cat ${INDEXES}); do
MARKER=$(echo ${OBJECT}|cut -d '.' -f 3,4,5)
grep ${MARKER} ${METADATA} > /dev/null
if [ "$?" -ne 0 ]; then
echo $OBJECT
fi
done
It does not remove anything, but for example, it returns these objects:
.dir.eb32b1ca-807a-4867-aea5-ff43ef7647c6.10406917.5752
.dir.eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6162
.dir.eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6186
The output of:
$ radosgw-admin metadata list|jq -r '.[]'
Does not contain:
- eb32b1ca-807a-4867-aea5-ff43ef7647c6.10406917.5752
- eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6162
- eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6186
So for me these objects do not seem to be tied to any bucket and seem to
be leftovers which were not cleaned up.
For example, I see these objects tied to a bucket:
- b32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6160
- eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6188
- eb32b1ca-807a-4867-aea5-ff43ef7647c6.10289105.6167
But notice the difference: 6160, 6188, 6167, but not 6162 nor 6186
Before I remove these objects I want to verify with other users if they
see the same and if my thinking is correct.
Wido
[0]: https://gist.github.com/wido/6650e66b09770ef02df89636891bef04
_______________________________________________
ceph-users mailing list
mailto:ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
