> Op 3 november 2016 om 13:09 schreef Joao Eduardo Luis <joao@xxxxxxx>:
>
>
> On 11/03/2016 09:40 AM, Wido den Hollander wrote:
> > root@mon3:/var/lib/ceph/mon# ceph-monstore-tool ceph-mon3 dump-keys|awk '{print $1}'|uniq -c
> > 96 auth
> > 1143 logm
> > 3 mdsmap
> > 1 mkfs
> > 1 mon_sync
> > 6 monitor
> > 3 monmap
> > 1158 osdmap
> > 358364 paxos
> > 656 pgmap
> > 6 pgmap_meta
> > 168 pgmap_osd
> > 6144 pgmap_pg
> > root@mon3:/var/lib/ceph/mon#
> >
> > So there are 358k Paxos entries in the Mon store.
> >
> > Any suggestions on how to trim those from the MON store(s)?
>
> Can you check the value of paxos:first_committed in the store?
>
Here you go:
root@mon3:~# ceph-monstore-tool /var/lib/ceph/mon/ceph-mon3 show-versions --map-type paxos
first committed: 174349108
last committed: 174349609
root@mon3:~#
Doesn't seem like a lot of keys in there?
> Also, things that would be useful:
>
> - values of 'paxos_mon' and 'paxos_trim_min', if defined in ceph.conf
none, running with the default values. Hammer 0.94.9
> - logs with 'debug paxos = 10'
>
Below is a snippet which I hope helps. What I noticed is that when I stop mon.3, query the monstore with the tool and start it again it will be in 'synchronizing' state for about 10 minutes with these logs:
2016-11-03 13:50:22.254059 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout
2016-11-03 13:50:22.273625 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync mon_sync(chunk cookie 3556769801 lc 174353374 bl 775321 bytes last_key paxos,28090389) v2
2016-11-03 13:50:22.273642 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync_chunk mon_sync(chunk cookie 3556769801 lc 174353374 bl 775321 bytes last_key paxos,28090389) v2
2016-11-03 13:50:22.297303 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout
2016-11-03 13:50:22.304582 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync mon_sync(chunk cookie 3556769801 lc 174353374 bl 2526379 bytes last_key paxos,28090390) v2
2016-11-03 13:50:22.304601 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync_chunk mon_sync(chunk cookie 3556769801 lc 174353374 bl 2526379 bytes last_key paxos,28090390) v2
2016-11-03 13:50:22.346193 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout
Here is the debug_paxos = 10 output, hope this is enough:
2016-11-03 14:23:32.761779 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357166) handle_begin paxos(begin lc 174357166 fc 0 pn 10700 opn 0) v3
2016-11-03 14:23:32.761785 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357166) accepting value for 174357167 pn 10700
2016-11-03 14:23:32.771145 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357166) handle_commit on 174357167
2016-11-03 14:23:32.771151 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) store_state [174357167..174357167]
2016-11-03 14:23:32.772336 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876166) refresh
2016-11-03 14:23:32.772443 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh
2016-11-03 14:23:32.772544 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh
2016-11-03 14:23:32.772642 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) refresh
2016-11-03 14:23:32.772859 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh
2016-11-03 14:23:32.772954 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh
2016-11-03 14:23:32.772963 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876166) post_refresh
2016-11-03 14:23:32.772969 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh
2016-11-03 14:23:32.772972 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh
2016-11-03 14:23:32.772974 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) post_refresh
2016-11-03 14:23:32.772977 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh
2016-11-03 14:23:32.772980 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh
2016-11-03 14:23:32.773016 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357167) handle_lease on 174357167 now 2016-11-03 14:23:37.772394
2016-11-03 14:23:32.773168 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357167) is_readable = 1 - now=2016-11-03 14:23:32.773170 lease_expire=2016-11-03 14:23:37.772394 has v0 lc 174357167
2016-11-03 14:23:33.766493 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357167) handle_begin paxos(begin lc 174357167 fc 0 pn 10700 opn 0) v3
2016-11-03 14:23:33.766511 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) accepting value for 174357168 pn 10700
2016-11-03 14:23:33.791042 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) handle_commit on 174357168
2016-11-03 14:23:33.791048 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) store_state [174357168..174357168]
2016-11-03 14:23:33.794594 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) refresh
2016-11-03 14:23:33.799217 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh
2016-11-03 14:23:33.799259 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh
2016-11-03 14:23:33.799313 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) refresh
2016-11-03 14:23:33.799385 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh
2016-11-03 14:23:33.799443 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh
2016-11-03 14:23:33.799449 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) post_refresh
2016-11-03 14:23:33.799451 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh
2016-11-03 14:23:33.799452 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh
2016-11-03 14:23:33.799453 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) post_refresh
2016-11-03 14:23:33.799454 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh
2016-11-03 14:23:33.799455 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh
2016-11-03 14:23:33.799475 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357168) handle_lease on 174357168 now 2016-11-03 14:23:38.797509
2016-11-03 14:23:33.799496 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357168) is_readable = 1 - now=2016-11-03 14:23:33.799498 lease_expire=2016-11-03 14:23:38.797509 has v0 lc 174357168
2016-11-03 14:23:33.804757 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357168) handle_begin paxos(begin lc 174357168 fc 0 pn 10700 opn 0) v3
2016-11-03 14:23:33.804764 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) accepting value for 174357169 pn 10700
2016-11-03 14:23:33.815222 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) handle_commit on 174357169
2016-11-03 14:23:33.815228 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357169) store_state [174357169..174357169]
2016-11-03 14:23:33.816307 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) refresh
2016-11-03 14:23:33.816353 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh
2016-11-03 14:23:33.816402 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh
2016-11-03 14:23:33.816458 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114632) refresh
2016-11-03 14:23:33.816587 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh
2016-11-03 14:23:33.816626 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh
2016-11-03 14:23:33.816629 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) post_refresh
2016-11-03 14:23:33.816631 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh
2016-11-03 14:23:33.816632 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh
2016-11-03 14:23:33.816633 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114632) post_refresh
2016-11-03 14:23:33.816634 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh
2016-11-03 14:23:33.816635 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh
2016-11-03 14:23:33.816653 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357169) handle_lease on 174357169 now 2016-11-03 14:23:38.816459
2016-11-03 14:23:33.816673 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357169) is_readable = 1 - now=2016-11-03 14:23:33.816674 lease_expire=2016-11-03 14:23:38.816459 has v0 lc 174357169
Wido
> -Joao
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
