Re: compacting omap doubles its size

[David Turner <drakonstein@xxxxxxxxx>]

Sorry for the late response on this, but life has been really busy over the holidays.

We compact our omaps offline with the ceph-kvstore-tool.  Here [1] is a copy of the script that we use for our clusters.  You might need to modify things a bit for your environment.  I don't remember which version this functionality was added to ceph-kvstore-tool, but it exists in 12.2.4.  We need to do this because our OSDs get marked out when they try to compact their own omaps online.  We run this script monthly and then ad-hoc as we find OSDs compacting their own omaps live.

[1] https://gist.github.com/drakonstein/4391c0b268a35b64d4f26a12e5058ba9

On Thu, Nov 29, 2018 at 6:15 PM Tomasz Płaza <tomasz.plaza@xxxxxxxxxx> wrote:

Hi,

I have a ceph 12.2.8 cluster on filestore with rather large omap dirs
(avg size is about 150G). Recently slow requests became a problem, so
after some digging I decided to convert omap from leveldb to rocksdb.
Conversion went fine and slow requests rate went down to acceptable
level. Unfortunately  conversion did not shrink most of omap dirs, so I
tried online compaction:

Before compaction: 50G    /var/lib/ceph/osd/ceph-0/current/omap/

After compaction: 100G    /var/lib/ceph/osd/ceph-0/current/omap/

Purge and recreate: 1.5G /var/lib/ceph/osd/ceph-0/current/omap/


Before compaction: 135G    /var/lib/ceph/osd/ceph-5/current/omap/

After compaction: 260G    /var/lib/ceph/osd/ceph-5/current/omap/

Purge and recreate: 2.5G /var/lib/ceph/osd/ceph-5/current/omap/


For me compaction which makes omap bigger is quite weird and
frustrating. Please help.


P.S. My cluster suffered from ongoing index reshards (it is disabled
now) and on many buckets with 4m+ objects I have a lot of old indexes:

634   bucket1
651   bucket2

...
1231 bucket17
1363 bucket18


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com