Hi Mark,
I suspect that BlueStore's buffer cache holds the memory. Could you
please try to set 'bluestore buffer cache size = 4194304' in your config
file and check what's happening again.
And another question - how many pools do you have for your test case? In
Bluestore each collection can allocate up to 512M for the buffer cache
by default. Not sure if collection == pool but there is some correlation
between them hence this might be the case.
Thanks,
Igor
On 30.06.2016 22:30, Mark Nelson wrote:
Not sure how related this is, but for both reads and random reads I'm
seeing very large memory spikes (not necessarily leaks). I've
included both the massif output and a screenshot from massif-visualizer.
The spikes appear to be due to:
bufferptr p = buffer::create_page_aligned(len)
in KernelDevice::read on line ~477 in KernelDevice.cc. Running via
massif slows things down enough that the node doesn't go out of memory
and the tests keep running, but if I attempt these tests without
massif, very quickly the OSD will spike to ~64GB of memory and then
get killed by the kernel.
There were a number of recent commits that modified KernelDevice.cc as
well as buffer.cc:
https://github.com/ceph/ceph/commits/master/src/os/bluestore/KernelDevice.cc
https://github.com/ceph/ceph/commits/master/src/common/buffer.cc
Still looking through them for signs of what might be amiss.
Mark
On 06/27/2016 09:11 AM, Igor Fedotov wrote:
Hi All,
let me share some observations I collected while running
ceph_test_objectstore against the bluestore.
Initially I started this investigations due to a failure in
SyntheticMatrixCompressionAlgorithm test case. The issue appeared while
running the the whole test suite and had a pretty odd symptom: failed
test case was run with settings that aren't provided for it: compression
= none, compression algorithm = snappy. No attempts to run for zlib
despite the fact zlib is the first in the list. When running single test
case everything is OK.
Additional investigations led me to the fact that RAM at my VM decreases
almost to zero while running the test suite and that probably prevents
from setting desired config params. Hence I proceeded with mem leak
investigation.
Since Synthetic test cases are pretty complex I switched to less complex
one - Many4KWriteNoCSumTest. As it performs writes only against the
single object this eliminates other ops, compression, csum, multiple
object handlings etc from under the suspicion.
Currently I can see ~6Gb mem consumption when doing ~3000 random writes
(up to 4K) over 4M object. Counting Bluestore's Buffer, Blob and Onode
objects shows that they aren't grow unexpectedly over the time for the
test case.
Then I changed the test case to perform fixed length(64K) writes - mem
consumption for 3000 writes reduced to 500M but I can see that Buffer
count is permanently growing - one buffer per single write. Thus
original issue is rather specific for small writes. But probably there
is another issue with buffer cache for big ones.
That's all what I have so far.
Any comments/ideas are appreciated.
Thanks,
Igor
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
