Hello,
I'm trying to understand the memory requirements for a Ceph node,
particularly when it is undergoing recovery.
Comments, suggestions, pointers are all welcome.
(This is my second attempt at sending this email; it appeared to get
eaten the first time — probably because it had a 1MB .heap file attached.)
Background:
==========
I've got a fairly tortured prototype Ceph cluster. It was left
unattended for several months, as I'd been needed to work elsewhere —
but now I'm returning to it, with an eye to continue to building
production services on it if I have sufficient confidence in its
capabilities.
In the intervening time, several root filesystems on cluster nodes went
full (because of poorly configured logging, as well as MONs being
co-located with OSDs for expediency) and several drives were also
unceremoniously pulled out for reuse elsewhere.
A subsequent recovery is proving problematic: if all OSDs are started
concurrently, they are substantially exceeding the amount of RAM
available on the hosts during peering, and are being killed off by the
kernel OOM killer.
(And then subsequently being restarted by Upstart, resulting in
thrashing for a while, up until something unknown goes awry and the
machine stops sending telemetry and no-longer responds to SSH. That's a
separate problem.)
Looking at tcmalloc-accounted heap statistics, I've seen individual OSDs
using 9GB+ of RAM; looking at RSS sizes of individual machines, I've
seen process-images exceeding 16GB. On 12-disk machines with 32GB of
RAM each, this is problematic.
So, I've started looking at the data-structures and algorithms that
govern OSD recovery. I've found the following references:
http://ceph.com/docs/master/dev/placement-group/
http://ceph.com/docs/master/dev/peering/
http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/
http://ceph.com/docs/master/dev/osd_internals/map_message_handling/
http://dachary.org/?p=2061
… and hope to develop an understanding of an upper bound on memory
utilization that an efficient implementation of the algorithms described
would require.
I've also been trying to collect memory profiles for OSD processes as
they're operating, to compare theory with reality.
Memory profiling:
================
For example, having found an OSD using ~6GB of memory, I turned on heap
profiling, and dumped its state using `ceph tell osd.N heap
start_profiler; ceph tell osd.N heap dump`:
------------------------------------------------
MALLOC: 6167528240 ( 5881.8 MiB) Bytes in use by application
MALLOC: + 18309120 ( 17.5 MiB) Bytes in page heap freelist
MALLOC: + 39689152 ( 37.9 MiB) Bytes in central cache freelist
MALLOC: + 4750960 ( 4.5 MiB) Bytes in transfer cache freelist
MALLOC: + 25223840 ( 24.1 MiB) Bytes in thread cache freelists
MALLOC: + 27603096 ( 26.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 6283104408 ( 5992.0 MiB) Actual memory used (physical + swap)
MALLOC: + 2080768 ( 2.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 6285185176 ( 5994.0 MiB) Virtual address space used
MALLOC:
MALLOC: 374907 Spans in use
MALLOC: 335 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
However, the heap dumps so generated only appear to show memory
allocations (made? touched?) since heap profiling was enabled:
google-pprof --text /usr/bin/ceph-osd osd.25.profile.0001.heap
Using local file /usr/bin/ceph-osd.
Using local file osd.25.profile.0001.heap.
Total: 0.0 MB
0.0 46.7% 46.7% 0.0 59.0% SimpleMessenger::add_accept_pipe
[...]
Note the "Total: 0.0MB", which differs wildly from the stats reported by
tcmalloc, and the RSS of the process reported by the kernel.
So, for testing purposes, I selectively started up ~20% of the OSDs,
each invoked with the setting
CEPH_HEAP_PROFILER_INIT=1
… defined in their environmentment to cause the heap profiler to be
started at OSD start-time. This has a significant CPU and memory
overhead.
Also set were the cluster flags:
noout,nobackfill,norecover,noscrub,nodeep-scrub
… to avoid commingling memory requirements due to peering with other
factors.
I've produced a number of .heap files which show >= 1000MB of memory
allocated in an RB tree as a result of
PG::RecoveryState::RecoveryMachine::send_notify, PG::read_info and
MOSDPGNotify::decode_payload (or descendants).
An example heapfile from a fairly typical OSD can currently be fetched from:
http://people.ds.cam.ac.uk/dwm37/tmp/osd.0.profile.0124.heap
This was produced by the binaries from the Ceph 'trusty' repository;
`ceph -v` returns:
ceph version 0.92 (00a3ac3b67d93860e7f0b6e07319f11b14d0fec0)
Running pprof in interactive mode and running `top30 --cum` on this
heapfile reports:
Total: 2172.3 MB
1705.9 78.5% 78.5% 1748.4 80.5% __gnu_cxx::new_allocator::construct (inline)
0.0 0.0% 78.5% 1600.7 73.7% std::_Rb_tree::_M_create_node (inline)
0.0 0.0% 78.5% 1367.9 63.0% start_thread
0.0 0.0% 78.5% 1367.6 63.0% ioperm
0.0 0.0% 78.5% 963.4 44.4% ThreadPool::worker
0.0 0.0% 78.5% 963.3 44.3% ThreadPool::WorkThread::entry
0.0 0.0% 78.5% 951.0 43.8% OSD::process_peering_events
0.0 0.0% 78.5% 950.9 43.8% OSD::PeeringWQ::_process
0.0 0.0% 78.5% 949.8 43.7% PG::RecoveryState::handle_event (inline)
0.0 0.0% 78.5% 949.8 43.7% boost::statechart::detail::send_function::operator (inline)
0.0 0.0% 78.5% 949.8 43.7% boost::statechart::simple_state::react_impl
0.0 0.0% 78.5% 949.8 43.7% boost::statechart::state_machine::process_event (inline)
0.0 0.0% 78.5% 949.8 43.7% boost::statechart::state_machine::send_event
0.0 0.0% 78.5% 949.8 43.7% local_react (inline)
0.0 0.0% 78.5% 949.8 43.7% local_react_impl (inline)
0.0 0.0% 78.5% 949.8 43.7% operator (inline)
0.0 0.0% 78.5% 949.8 43.7% react (inline)
0.0 0.0% 78.5% 948.5 43.7% std::vector::push_back (inline)
0.0 0.0% 78.5% 948.3 43.7% PG::RecoveryState::RecoveryMachine::send_notify
0.0 0.0% 78.5% 947.1 43.6% std::vector::_M_insert_aux
0.0 0.0% 78.5% 947.0 43.6% _Rb_tree (inline)
0.0 0.0% 78.5% 947.0 43.6% map (inline)
0.0 0.0% 78.5% 947.0 43.6% std::_Rb_tree::_M_clone_node (inline)
0.0 0.0% 78.5% 947.0 43.6% std::_Rb_tree::_M_copy
0.0 0.0% 78.5% 809.8 37.3% construct (inline)
0.0 0.0% 78.5% 808.4 37.2% std::pair::pair
0.0 0.0% 78.5% 804.2 37.0% __libc_start_main
0.0 0.0% 78.5% 804.2 37.0% _start
0.0 0.0% 78.5% 804.2 37.0% main
0.0 0.0% 78.5% 803.6 37.0% OSD::init
This appears to show a large amount of memory — nearly a gigabyte —
allocated by boost::statechart, which is slightly surprising as the FAQ
for boost::statechart quotes a ~1KB memory footprint per state-machine:
http://www.boost.org/doc/libs/1_35_0/libs/statechart/doc/faq.html#EmbeddedApplications
Perhaps something unexpected is happening here? I'm almost hoping that
perhaps statechart is perhaps being subtly misused or misconfigured in
some way that, if fixed, would result in a significant drop in memory
utilization…!
Quantifying problem-size:
========================
Given that it appears to be the log-merging stage of PG recovery that
seems to be expensive, I queried the statistics of those PGs which
seemed to be taking a long time to peer, via `ceph pg <pgid> query`.
These showed that (at least a handful) of those PG's recovery_state
past_intervals list contained on the order of ~200-300 entries.
(I have no feel as to whether this is excessive.)
Unused memory:
=============
One thing I note is that I still sometimes see OSDs with large fractions
of their memory allocation sitting on the tcmalloc freelist, e.g.:
osd.0 tcmalloc heap stats:------------------------------------------------
MALLOC: 2226810584 ( 2123.7 MiB) Bytes in use by application
MALLOC: + 1421361152 ( 1355.5 MiB) Bytes in page heap freelist
MALLOC: + 41864920 ( 39.9 MiB) Bytes in central cache freelist
MALLOC: + 5215680 ( 5.0 MiB) Bytes in transfer cache freelist
MALLOC: + 18508944 ( 17.7 MiB) Bytes in thread cache freelists
MALLOC: + 16216216 ( 15.5 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 3729977496 ( 3557.2 MiB) Actual memory used (physical + swap)
MALLOC: + 32792576 ( 31.3 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 3762770072 ( 3588.5 MiB) Virtual address space used
MALLOC:
MALLOC: 144565 Spans in use
MALLOC: 225 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
This is despite having:
TCMALLOC_RELEASE_RATE=10
… set in the environment of each OSD process. This doesn't help with
contention for RAM between processes!
(I have mentioned this before, though hadn't at that time yet tried
running OSDs with TCMALLOC_RELEASE_RATE. See also:
http://www.spinics.net/lists/ceph-devel/msg18769.html
… for history.
Note for anyone intending to reproduce this experiment: Upstart
overrides should be written to a file named
/etc/init/ceph-{osd,mon}.override, not ceph-{osd,mon}.conf.override as I
incorrectly specified previously.)
Leak detection:
==============
Not yet being familiar with the the data-structures or algorithms that
govern PG recovery, it's not clear to me whether this is memory usage
that is expected or not for a 120-OSD cluster with 2048 PGs — or
whether there might be some variety of leak (or inefficient memory-use
pattern.)
It doesn't help that I'm not a C++ hacker. :-)
Reading around the subject, I came across `leaksanitiser`, a clang/LLVM:
facility:
https://code.google.com/p/address-sanitizer/wiki/LeakSanitizer
… as well as ticket #9756, which suggests using Clang's other static
analysis capabilities to help flag potentially problematic code:
http://tracker.ceph.com/issues/9756
I might spend some time this weekend to see if I can help advance that
ticket.
(I note that http://ceph.com/gitbuilders.cgi now returns 404; perhaps
that has been superceded by some RedHat-internal facility?)
Cheers,
David
--
David McBride <dwm37@xxxxxxxxx>
Unix Specialist, University Information Services
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
