> Op 7 november 2016 om 13:17 schreef Wido den Hollander <wido@xxxxxxxx>: > > > > > Op 4 november 2016 om 2:05 schreef Joao Eduardo Luis <joao@xxxxxxx>: > > > > > > On 11/03/2016 06:18 PM, wido@xxxxxxxx wrote: > > > > > >> Personally, I don't like this solution one bit, but I can't see any other way without a patched monitor, or maybe ceph_monstore_tool. > > >> > > >> If you are willing to wait till tomorrow, I'll be happy to kludge a sanitation feature onto ceph_monstore_tool that will clean those versions for you (latency being due to coding + testing + building). > > >> > > > > > > Yes, no rush. I will wait. Thanks! > > > > Hey Wido, > > > > I just pushed 'wip-monstore-tool-sanitize-paxos.hammer' to upstream's > > ceph.git. It should be built by morning. > > > > This is a half-assed feature to ceph-monstore-tool, so you should test > > it first on a copy of your monitors, just to make sure I didn't forget > > any corner case. > > > > The command you want is > > > > ceph-monstore-tool <path-to-monstore> sanitize-paxos -- --dry-run > > > > and, if you're happy with whatever it spits out, then drop --dry-run. > > > > Should this work for you, next step will be to merge this onto master > > and, hopefully, backport to previous versions. We'll see. > > > > I tried, but it doesn't seem to work. > To wrap this up, after having a off the list discussion with Joao this was fixed. The fix is in this branch: https://github.com/jecluis/ceph/tree/wip-monstore-tool-copy.hammer Using 'store-copy-conservative' from the old to new path the store got trimmed and went down from 36GB to ~100M. Ran this on all three Monitors one by one and it fixed it. Joao will work on getting this back in upstream. Wido > # time ./ceph-monstore-tool /var/lib/ceph/mon/ceph-mon1 sanitize-paxos -- > > It ran for an hour (without dry-run) and outputted: > > paxos [175000202 .. 175000831] > found surplus paxos versions that ought to be removed: > - [ 225964 .. 226320] > - [ 260032 .. 260401] > - [ 891292 .. 891642] > ... > ... > ... > - [ 35210293 .. 35210571] > - [ 35213765 .. 35213766] > removing paxos versions [225964 .. 226320] > removing paxos versions [260032 .. 260401] > ... > ... > ... > removing paxos versions [35188867 .. 35189115] > removing paxos versions [35210293 .. 35210571] > removing paxos versions [35213765 .. 35213766] > all done! > > real 57m18.904s > user 56m26.928s > sys 0m9.884s > > > I ran it on the Monitor which will become primary again after it syncs. After running the tool the store is still 36GB in size, grows to 50GB during the sync and then trims down again to 36GB. > > Wido > > > HTH, > > > > -Joao > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |