> Do you have statistics on the size of the OSDMaps or count of them
> which were being maintained by the OSDs?
No, I don't think so. How can I find this information?
Memory consumption started to climb again:
https://icecube.wisc.edu/~vbrik/graph-3.png
Some more info (not sure if relevant or not):
I increased size of the swap on the servers to 10GB and it's being
completely utilized, even though there is still quite a bit of free memory.
It appears that memory is highly fragmented on the NUMA node 0 of all
the servers. Some of the servers have no free pages higher than order 0.
(Memory on NUMA node 1 of the servers appears much less fragmented.)
The servers have 192GB of RAM, 2 NUMA nodes.
Vlad
On 10/4/19 6:09 PM, Gregory Farnum wrote:
Do you have statistics on the size of the OSDMaps or count of them
which were being maintained by the OSDs? I'm not sure why having noout
set would change that if all the nodes were alive, but that's my bet.
-Greg
On Thu, Oct 3, 2019 at 7:04 AM Vladimir Brik
<vladimir.brik@xxxxxxxxxxxxxxxx> wrote:
And, just as unexpectedly, things have returned to normal overnight
https://icecube.wisc.edu/~vbrik/graph-1.png
The change seems to have coincided with the beginning of Rados Gateway
activity (before, it was essentially zero). I can see nothing in the
logs that would explain what happened though.
Vlad
On 10/2/19 3:43 PM, Vladimir Brik wrote:
Hello
I am running a Ceph 14.2.2 cluster and a few days ago, memory
consumption of our OSDs started to unexpectedly grow on all 5 nodes,
after being stable for about 6 months.
Node memory consumption: https://icecube.wisc.edu/~vbrik/graph.png
Average OSD resident size: https://icecube.wisc.edu/~vbrik/image.png
I am not sure what changed to cause this. Cluster usage has been very
light (typically <10 iops) during this period, and the number of objects
stayed about the same.
The only unusual occurrence was the reboot of one of the nodes the day
before (a firmware update). For the reboot, I ran "ceph osd set noout",
but forgot to unset it until several days later. Unsetting noout did not
stop the increase in memory consumption.
I don't see anything unusual in the logs.
Our nodes have SSDs and HDDs. Resident set size of SSD ODSs is about
3.7GB. Resident set size of HDD OSDs varies from about 5GB to 12GB. I
don't know why there is such a big spread. All HDDs are 10TB, 72-76%
utilized, with 101-104 PGs.
Does anybody know what might be the problem here and how to address or
debug it?
Thanks very much,
Vlad
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
