> Op 3 november 2016 om 13:09 schreef Joao Eduardo Luis <joao@xxxxxxx>: > > > On 11/03/2016 09:40 AM, Wido den Hollander wrote: > > root@mon3:/var/lib/ceph/mon# ceph-monstore-tool ceph-mon3 dump-keys|awk '{print $1}'|uniq -c > > 96 auth > > 1143 logm > > 3 mdsmap > > 1 mkfs > > 1 mon_sync > > 6 monitor > > 3 monmap > > 1158 osdmap > > 358364 paxos > > 656 pgmap > > 6 pgmap_meta > > 168 pgmap_osd > > 6144 pgmap_pg > > root@mon3:/var/lib/ceph/mon# > > > > So there are 358k Paxos entries in the Mon store. > > > > Any suggestions on how to trim those from the MON store(s)? > > Can you check the value of paxos:first_committed in the store? > Here you go: root@mon3:~# ceph-monstore-tool /var/lib/ceph/mon/ceph-mon3 show-versions --map-type paxos first committed: 174349108 last committed: 174349609 root@mon3:~# Doesn't seem like a lot of keys in there? > Also, things that would be useful: > > - values of 'paxos_mon' and 'paxos_trim_min', if defined in ceph.conf none, running with the default values. Hammer 0.94.9 > - logs with 'debug paxos = 10' > Below is a snippet which I hope helps. What I noticed is that when I stop mon.3, query the monstore with the tool and start it again it will be in 'synchronizing' state for about 10 minutes with these logs: 2016-11-03 13:50:22.254059 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout 2016-11-03 13:50:22.273625 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync mon_sync(chunk cookie 3556769801 lc 174353374 bl 775321 bytes last_key paxos,28090389) v2 2016-11-03 13:50:22.273642 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync_chunk mon_sync(chunk cookie 3556769801 lc 174353374 bl 775321 bytes last_key paxos,28090389) v2 2016-11-03 13:50:22.297303 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout 2016-11-03 13:50:22.304582 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync mon_sync(chunk cookie 3556769801 lc 174353374 bl 2526379 bytes last_key paxos,28090390) v2 2016-11-03 13:50:22.304601 7f93c56ab700 10 mon.3@2(synchronizing) e1 handle_sync_chunk mon_sync(chunk cookie 3556769801 lc 174353374 bl 2526379 bytes last_key paxos,28090390) v2 2016-11-03 13:50:22.346193 7f93c56ab700 10 mon.3@2(synchronizing) e1 sync_reset_timeout Here is the debug_paxos = 10 output, hope this is enough: 2016-11-03 14:23:32.761779 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357166) handle_begin paxos(begin lc 174357166 fc 0 pn 10700 opn 0) v3 2016-11-03 14:23:32.761785 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357166) accepting value for 174357167 pn 10700 2016-11-03 14:23:32.771145 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357166) handle_commit on 174357167 2016-11-03 14:23:32.771151 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) store_state [174357167..174357167] 2016-11-03 14:23:32.772336 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876166) refresh 2016-11-03 14:23:32.772443 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh 2016-11-03 14:23:32.772544 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh 2016-11-03 14:23:32.772642 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) refresh 2016-11-03 14:23:32.772859 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh 2016-11-03 14:23:32.772954 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh 2016-11-03 14:23:32.772963 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876166) post_refresh 2016-11-03 14:23:32.772969 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh 2016-11-03 14:23:32.772972 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh 2016-11-03 14:23:32.772974 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) post_refresh 2016-11-03 14:23:32.772977 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh 2016-11-03 14:23:32.772980 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh 2016-11-03 14:23:32.773016 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357167) handle_lease on 174357167 now 2016-11-03 14:23:37.772394 2016-11-03 14:23:32.773168 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357167) is_readable = 1 - now=2016-11-03 14:23:32.773170 lease_expire=2016-11-03 14:23:37.772394 has v0 lc 174357167 2016-11-03 14:23:33.766493 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357167) handle_begin paxos(begin lc 174357167 fc 0 pn 10700 opn 0) v3 2016-11-03 14:23:33.766511 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) accepting value for 174357168 pn 10700 2016-11-03 14:23:33.791042 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357167) handle_commit on 174357168 2016-11-03 14:23:33.791048 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) store_state [174357168..174357168] 2016-11-03 14:23:33.794594 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) refresh 2016-11-03 14:23:33.799217 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh 2016-11-03 14:23:33.799259 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh 2016-11-03 14:23:33.799313 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) refresh 2016-11-03 14:23:33.799385 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh 2016-11-03 14:23:33.799443 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh 2016-11-03 14:23:33.799449 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) post_refresh 2016-11-03 14:23:33.799451 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh 2016-11-03 14:23:33.799452 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh 2016-11-03 14:23:33.799453 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114631) post_refresh 2016-11-03 14:23:33.799454 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh 2016-11-03 14:23:33.799455 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh 2016-11-03 14:23:33.799475 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357168) handle_lease on 174357168 now 2016-11-03 14:23:38.797509 2016-11-03 14:23:33.799496 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357168) is_readable = 1 - now=2016-11-03 14:23:33.799498 lease_expire=2016-11-03 14:23:38.797509 has v0 lc 174357168 2016-11-03 14:23:33.804757 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357168) handle_begin paxos(begin lc 174357168 fc 0 pn 10700 opn 0) v3 2016-11-03 14:23:33.804764 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) accepting value for 174357169 pn 10700 2016-11-03 14:23:33.815222 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357168) handle_commit on 174357169 2016-11-03 14:23:33.815228 7f93c56ab700 10 mon.3@2(peon).paxos(paxos updating c 174356638..174357169) store_state [174357169..174357169] 2016-11-03 14:23:33.816307 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) refresh 2016-11-03 14:23:33.816353 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) refresh 2016-11-03 14:23:33.816402 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) refresh 2016-11-03 14:23:33.816458 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114632) refresh 2016-11-03 14:23:33.816587 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) refresh 2016-11-03 14:23:33.816626 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) refresh 2016-11-03 14:23:33.816629 7f93c56ab700 10 mon.3@2(peon).paxosservice(pgmap 101875437..101876167) post_refresh 2016-11-03 14:23:33.816631 7f93c56ab700 10 mon.3@2(peon).paxosservice(mdsmap 1..1) post_refresh 2016-11-03 14:23:33.816632 7f93c56ab700 10 mon.3@2(peon).paxosservice(osdmap 119122..119704) post_refresh 2016-11-03 14:23:33.816633 7f93c56ab700 10 mon.3@2(peon).paxosservice(logm 71113900..71114632) post_refresh 2016-11-03 14:23:33.816634 7f93c56ab700 10 mon.3@2(peon).paxosservice(monmap 1..1) post_refresh 2016-11-03 14:23:33.816635 7f93c56ab700 10 mon.3@2(peon).paxosservice(auth 90002..90095) post_refresh 2016-11-03 14:23:33.816653 7f93c56ab700 10 mon.3@2(peon).paxos(paxos active c 174356638..174357169) handle_lease on 174357169 now 2016-11-03 14:23:38.816459 2016-11-03 14:23:33.816673 7f93c56ab700 5 mon.3@2(peon).paxos(paxos active c 174356638..174357169) is_readable = 1 - now=2016-11-03 14:23:33.816674 lease_expire=2016-11-03 14:23:38.816459 has v0 lc 174357169 Wido > -Joao > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com