hi Corin. We run latest hammer on CentOS7.2, with 3 mons and have not
seen this problem. I'm not sure if there are any other possible
differences between the healthy nodes and the one that has excessive
consumption of memory? thx will
On Fri, Nov 18, 2016 at 6:35 PM, Corin Langosch
<corin.langosch@xxxxxxxxxxx> wrote:
> Hi,
>
> about 2 weeks ago I upgraded a rather small cluster from ceph 0.94.2 to
> 0.94.9. The upgrade went fine, the cluster is running stable. But I just
> noticed that one monitor is already eating 20 GB of memory, growing slowly
> over time. The other 2 mons look fine. The disk space used by the
> problematic mon looks fine too.
>
> ceph -w
> cluster 4ac0e21b-6ea2-4ac7-8114-122bd9ba55d6
> health HEALTH_OK
> monmap e5: 3 mons at
> {a=10.0.0.5:6789/0,b=10.0.0.6:6789/0,c=10.0.0.7:6789/0}
> election epoch 856, quorum 0,1,2 a,b,c
> osdmap e27838: 19 osds: 9 up, 9 in
> pgmap v104865438: 4096 pgs, 1 pools, 3225 GB data, 809 kobjects
> 6463 GB used, 1955 GB / 8419 GB avail
> 4096 active+clean
> client io 1484 kB/s rd, 14217 kB/s wr, 2026 op/s
>
>
> ps aux | grep ceph
> root 5958 1.2 15.7 21194296 20821228 ? Sl Nov05 238:33
> /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 7457 38.5 0.3 1660028 509008 ? Ssl Nov05 7125:00
> /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 7981 25.8 0.4 1661064 543684 ? Ssl Nov05 4775:44
> /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c
> /etc/ceph/ceph.conf --cluster ceph
>
>
> ps aux | grep ceph
> root 12704 2.0 0.1 532184 93468 ? Sl Nov05 374:16
> /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 14587 32.3 0.8 1682720 581752 ? Ssl Nov05 5970:53
> /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 14919 28.8 0.7 1680144 526052 ? Ssl Nov05 5328:17
> /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c
> /etc/ceph/ceph.conf --cluster ceph
>
>
> ps aux | grep ceph
> root 19114 1.3 0.2 508000 157724 ? Sl Nov05 256:53
> /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 20155 34.3 0.7 1652784 516072 ? Ssl Nov05 6341:37
> /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c
> /etc/ceph/ceph.conf --cluster ceph
> root 20508 33.2 0.7 1666276 510496 ? Ssl Nov05 6135:04
> /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c
> /etc/ceph/ceph.conf --cluster ceph
>
>
> dpkg -l | grep ceph
> ii ceph 0.94.9-1trusty
> amd64 distributed storage and file system
> ii ceph-common 0.94.9-1trusty
> amd64 common utilities to mount and interact with a ceph storage
> cluster
> ii ceph-fs-common 0.94.9-1trusty
> amd64 common utilities to mount and interact with a ceph file system
> ii ceph-fuse 0.94.9-1trusty
> amd64 FUSE-based client for the Ceph distributed file system
> ii ceph-mds 0.94.9-1trusty
> amd64 metadata server for the ceph distributed file system
> ii ceph-test 0.94.9-1trusty
> amd64 Ceph test and benchmarking tools
> ii libcephfs1 0.94.9-1trusty
> amd64 Ceph distributed file system client library
> ii python-cephfs 0.94.9-1trusty
> amd64 Python libraries for the Ceph libcephfs library
>
> /var/lib/ceph/mon/ceph-c# du -h
> 108K ./auth
> 88K ./auth_gv
> 2.1M ./logm
> 2.0M ./logm_gv
> 24K ./mdsmap
> 4.0K ./mdsmap_gv
> 4.0K ./mkfs
> 40K ./monmap
> 20K ./monmap_gv
> 4.2M ./osdmap
> 15M ./osdmap_full
> 4.2M ./osdmap_gv
> 13M ./pgmap
> 2.0M ./pgmap_gv
> 201M ./store.db
> 243M .
>
> Is this a known bug? Any workaround or fix available?
>
> Thanks
> Corin
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
