On 12/10/2014 07:30 PM, Kevin Sumner wrote:
The mons have grown another 30GB each overnight (except for 003?), which
is quite worrying. I ran a little bit of testing yesterday after my
post, but not a significant amount.
I wouldn’t expect compact on start to help this situation based on the
name since we don’t (shouldn’t?) restart the mons regularly, but there
appears to be no documentation on it. We’re pretty good on disk space
on the mons currently, but if that changes, I’ll probably use this to
see about bringing these numbers in line.
This is an issue that has been seen on larger clusters, and it usually
takes a monitor restart, with 'mon compact on start = true' or manual
compaction 'ceph tell mon.FOO compact' to bring the monitor back to a
sane disk usage level.
However, I have not been able to reproduce this in order to track the
source. I'm guessing I lack the scale of the cluster, or the
appropriate workload (maybe both).
What kind of workload are you running the cluster through? You mention
cephfs, but do you have any more info you can share that could help us
reproducing this state?
Sage also fixed an issue that could potentially cause this (depending on
what is causing it in the first place) [1,2,3]. This bug, #9987, is due
to a given cached value not being updated, leading to the monitor not
removing unnecessary data, potentially causing this growth. This cached
value would be set to its proper value when the monitor is restarted
though, so a simple restart would have all this unnecessary data blown away.
Restarting the monitor ends up masking the true cause of the store
growth: whether from #9987 or from obsolete data kept by the monitor's
backing store (leveldb), either due to misuse of leveldb or due to
leveldb's nature (haven't been able to ascertain which may be at fault,
partly due to being unable to reproduce the problem).
If you are up to it, I would suggest the following approach in hope to
determine what may be at fault:
1) 'ceph tell mon.FOO compact' -- which will force the monitor to
compact its store. This won't close leveldb, so it won't have much
effect on the store size if it happens to be leveldb holding on to some
data (I could go into further detail, but I don't think this is the
right medium).
1.a) you may notice the store increasing in size during this period;
it's expected.
1.b) compaction may take a while, but in the end you'll hopefully see a
significant reduction in size.
2) Assuming that failed, I would suggest doing the following:
2.1) grab ceph-kvstore-tool from the ceph-test package
2.2) stop the monitor
2.3) run 'ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db list >
store.dump'
2.4) run (for above store's location, let's call it $STORE:
for m in osdmap pgmap; do
for k in first_committed last_committed; do
ceph-kvstore-tool $STORE get $m $k >> store.dump
done
done
ceph-kvstore-tool $STORE get pgmap_meta last_osdmap_epoch >> store.dump
ceph-kvstore-tool $STORE get pgmap_meta version >> store.dump
2.5) send over the results of the dump
2.6) if you were to compress the store as well and send me a link to
grab it I would appreciate it.
3) Next you could simply restart the monitor (without 'mon compact on
start = true'); if the monitor's store size decreases, then there's a
fair chance that you've been bit by #9987. Otherwise, it may be
leveldb's clutter. You should also note that leveldb may itself compact
automatically on start, so it's hard to say for sure what fixed what.
4) If store size hasn't gone back to sane levels by now, you may wish to
restart with 'mon compact on start = true' and see if it helps. If it
doesn't, then we may have a completely different issue in our hands.
Now, assuming your store size went down on step 3, and if you are
willing, it would be interesting to see if Sage's patches helps out in
any way. The patches have not been backported to the giant branch yet,
so you would have to apply them yourself. For them to work you would
have to run the patched monitor as the leader. I would suggest leaving
the other monitors running an unpatched version so they could act as the
control group.
Let us know if any of this helps.
Cheers!
-Joao
[1] - http://tracker.ceph.com/issues/9987
[2] - 093c5f0cabeb552b90d944da2c50de48fcf6f564
[3] - 3fb731b722c50672a5a9de0c86a621f5f50f2d06
:: ~ » ceph health detail | grep 'too big'
HEALTH_WARN mon.cluster4-monitor001 store is getting too big! 77365 MB
>= 15360 MB; mon.cluster4-monitor002 store is getting too big! 87868 MB
>= 15360 MB; mon.cluster4-monitor003 store is getting too big! 30359 MB
>= 15360 MB; mon.cluster4-monitor004 store is getting too big! 93414 MB
>= 15360 MB; mon.cluster4-monitor005 store is getting too big! 88232 MB
>= 15360 MB
mon.cluster4-monitor001 store is getting too big! 77365 MB >= 15360 MB
-- 72% avail
mon.cluster4-monitor002 store is getting too big! 87868 MB >= 15360 MB
-- 70% avail
mon.cluster4-monitor003 store is getting too big! 30359 MB >= 15360 MB
-- 85% avail
mon.cluster4-monitor004 store is getting too big! 93414 MB >= 15360 MB
-- 69% avail
mon.cluster4-monitor005 store is getting too big! 88232 MB >= 15360 MB
-- 71% avail
--
Kevin Sumner
kevin@xxxxxxxxx <mailto:kevin@xxxxxxxxx>
On Dec 9, 2014, at 6:20 PM, Haomai Wang <haomaiwang@xxxxxxxxx
<mailto:haomaiwang@xxxxxxxxx>> wrote:
Maybe you can enable "mon_compact_on_start=true" when restarting mon,
it will compact data
On Wed, Dec 10, 2014 at 6:50 AM, Kevin Sumner <kevin@xxxxxxxxx
<mailto:kevin@xxxxxxxxx>> wrote:
Hi all,
We recently upgraded our cluster to Giant from. Since then, we’ve been
driving load tests against CephFS. However, we’re getting “store is
getting
too big” warnings from the monitors and the mons have started
consuming way
more disk space, 40GB-60GB now as opposed to ~10GB pre-upgrade. Is this
expected? Is there anything I can do to ease the store’s size?
Thanks!
:: ~ » ceph status
cluster f1aefa73-b968-41e0-9a28-9a465db5f10b
health HEALTH_WARN mon.cluster4-monitor001 store is getting too big!
45648 MB >= 15360 MB; mon.cluster4-monitor002 store is getting too big!
56939 MB >= 15360 MB; mon.cluster4-monitor003 store is getting too big!
28647 MB >= 15360 MB; mon.cluster4-monitor004 store is getting too big!
60655 MB >= 15360 MB; mon.cluster4-monitor005 store is getting too big!
57335 MB >= 15360 MB
monmap e3: 5 mons at
{cluster4-monitor001=17.138.96.12:6789/0,cluster4-monitor002=17.138.96.13:6789/0,cluster4-monitor003=17.138.96.14:6789/0,cluster4-monitor004=17.138.96.15:6789/0,cluster4-monitor005=17.138.96.16:6789/0},
election epoch 34938, quorum 0,1,2,3,4
cluster4-monitor001,cluster4-monitor002,cluster4-monitor003,cluster4-monitor004,cluster4-monitor005
mdsmap e6538: 1/1/1 up {0=cluster4-monitor001=up:active}
osdmap e49500: 501 osds: 470 up, 469 in
pgmap v1369307: 98304 pgs, 3 pools, 4933 GB data, 1976 kobjects
16275 GB used, 72337 GB / 93366 GB avail
98304 active+clean
client io 3463 MB/s rd, 18710 kB/s wr, 7456 op/s
--
Kevin Sumner
kevin@xxxxxxxxx <mailto:kevin@xxxxxxxxx>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Best Regards,
Wheat
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
