Hi everyone,
yesterday evening one of my rgw nodes died again, radosgw was killed by the kernel oom killer.
[Thu Aug 12 22:10:04 2021] Out of memory: Killed process 1376 (radosgw) total-vm:70747176kB, anon-rss:63900544kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:131008kB oom_score_adj:0
[Thu Aug 12 22:10:09 2021] oom_reaper: reaped process 1376 (radosgw), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
The radosgw was eating up all the 64GB system memory.
A few hours before this happened, mempool dump showed a total usage of only 2.1 GB of ram, while in fact radosgw was using already 84.7% of 64GB.
"total": {
"items": 88757980,
"bytes": 2147532284
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1376 ceph 20 0 58.8g 52.7g 17824 S 48.2 84.7 20158:04 radosgw
It seems the radowgw loses track of some memory, like there is a memory leak.
Some additional information. I am running on CentOS 8.4, kernel 4.18. As already mentioned, Ceph 14.2.22. radosgw is the only notable service running on this machine.
Any suggestions on this? Are there maybe any tuning settings? How could I debug this further?
________________________________________
Von: Martin Traxl <martin.traxl@xxxxxxxx>
Gesendet: Dienstag, 10. August 2021 15:15
An: ceph-users@xxxxxxx
Betreff: Re: RGW memory consumption
I should mention we are using the S3 interface, so it is S3 traffic.
________________________________________
Von: Martin Traxl <martin.traxl@xxxxxxxx>
Gesendet: Dienstag, 10. August 2021 14:35
An: ceph-users@xxxxxxx
Betreff: RGW memory consumption
Hi everyone,
we are running a ceph nautilus 14.2.22 cluster with 6 osd-nodes (each 12x8TB hdd, 2x ssd for rgw index pool), 3 mon-nodes and 3 dedicated rgw-nodes. The rgw-nodes have 64GB ram each. We have a job running every day at 4 am. It is running for about 1 hour, it is read only, read load is peaking up to about 970MB/s (megabyte per second). This is causing an additional ~5% of ram usage every day on the rgw-nodes. This ram usage is never freed up. So we have to restart the rgws regularly. Maybe there is some kind of memory leak ongoing.
For example right now on one of my rgw-nodes:
$ free -m
total used free shared buff/cache available
Mem: 63686 44084 6363 4 13238 18887
top tells me, radosgw is using 67.8% of the machines memory.
A mempool dump on the radosgw shows me
"total": {
"items": 71162589,
"bytes": 1712643376
which is only 1,7 Gbyte. Almost all of it used as "buffer_anon".
Are there any setting that might help tuning the memory consumption? Do you need further information about my setup?
Thank you,
Martin
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
