To me it looks like a memory leak which wasn't present in 16.2.11 (the
previous Ceph version on this cluster). The usage hasn't changed, so
it must be Ceph. I've been watching podman stats for a bit, the rgw
process uses more and more memory, until it caps at around 5 or 6 GB,
then it respawns and runs for a couple of hours. But there's no OOM
killer or anything, the host has plenty free RAM. This host does also
run OSDs, but only for barely used HDD pools.
Zitat von Eugen Block <eblock@xxxxxx>:
Hi,
yesterday I upgraded a customer cluster to Reef (18.2.4). The
upgrade went quite well, nothing happened for hours, until it did.
One of the two RGW daemons has crashed twice in the last 12 hours.
Here's one backtrace:
---snip---
Feb 06 23:19:58 storage09 conmon[2501983]: radosgw:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/redhat-linux-build/boost/include/boost/context/posix/protected_fixedsize_stack.hpp:70: boost::context::stack_context boost::context::basic_protected_fixedsize_stack<traitsT>::allocate() [with traitsT = boost::context::stack_traits]: Assertion `0 == result'
failed.
Feb 06 23:19:58 storage09 conmon[2501983]: *** Caught signal (Aborted) **
Feb 06 23:19:58 storage09 conmon[2501983]: in thread 7f77a10b2640
thread_name:radosgw
Feb 06 23:19:58 storage09 conmon[2501983]: ceph version 18.2.4
(e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
Feb 06 23:19:58 storage09 conmon[2501983]: 1:
/lib64/libc.so.6(+0x3e6f0) [0x7f78adeba6f0]
Feb 06 23:19:58 storage09 conmon[2501983]: 2:
/lib64/libc.so.6(+0x8b94c) [0x7f78adf0794c]
Feb 06 23:19:58 storage09 conmon[2501983]: 3: raise()
Feb 06 23:19:58 storage09 conmon[2501983]: 4: abort()
Feb 06 23:19:58 storage09 conmon[2501983]: 5:
/lib64/libc.so.6(+0x2871b) [0x7f78adea471b]
Feb 06 23:19:58 storage09 conmon[2501983]: 6:
/lib64/libc.so.6(+0x37386) [0x7f78adeb3386]
Feb 06 23:19:58 storage09 conmon[2501983]: 7:
/usr/bin/radosgw(+0x361cb2) [0x561ed9ef3cb2]
Feb 06 23:19:58 storage09 conmon[2501983]: 8:
/usr/bin/radosgw(+0x361db8) [0x561ed9ef3db8]
Feb 06 23:19:58 storage09 conmon[2501983]: 9:
/usr/bin/radosgw(+0x36e15e) [0x561ed9f0015e]
Feb 06 23:19:58 storage09 conmon[2501983]: 10:
/usr/bin/radosgw(+0x357558) [0x561ed9ee9558]
Feb 06 23:19:58 storage09 conmon[2501983]: 11:
/usr/bin/radosgw(+0x34546c) [0x561ed9ed746c]
Feb 06 23:19:58 storage09 conmon[2501983]: 12:
/usr/bin/radosgw(+0x358f0a) [0x561ed9eeaf0a]
Feb 06 23:19:58 storage09 conmon[2501983]: 13:
/usr/bin/radosgw(+0xb705de) [0x561eda7025de]
Feb 06 23:19:58 storage09 conmon[2501983]: 14:
/usr/bin/radosgw(+0x3c6aed) [0x561ed9f58aed]
Feb 06 23:19:58 storage09 conmon[2501983]: 15:
/lib64/libstdc++.so.6(+0xdbad4) [0x7f78ae258ad4]
Feb 06 23:19:58 storage09 conmon[2501983]: 16:
/lib64/libc.so.6(+0x89c02) [0x7f78adf05c02]
Feb 06 23:19:58 storage09 conmon[2501983]: 17:
/lib64/libc.so.6(+0x10ec40) [0x7f78adf8ac40]
---snip---
I didn't find anything helpful in the tracker, only this report on
this list [0] without a response from a year ago. The other daemon
on a different host seems to be stable for now. This is no
multi-site deployment, just two RGWs for a single zone.
Any comments/pointers are appreciated! I can file a tracker issue if
this is something new.
Thanks!
Eugen
[0] https://www.spinics.net/lists/ceph-users/msg80956.html
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
