"_": buffer::list(len=302, buffer::ptr(0~302 0x559318e74d80 in raw
0x559318e74d80 len 488 nref 1) ),
"_user.rgw.acl": buffer::list(len=147, buffer::ptr(448~147 0x55931677c4c0 in
raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.content_type": buffer::list(len=25, buffer::ptr(616~25
0x55931677c568 in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.etag": buffer::list(len=33, buffer::ptr(654~33 0x55931677c58e in
raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.idtag": buffer::list(len=44, buffer::ptr(14~44 0x55931958e00e in
raw 0x55931958e000 len 4194304 nref 2) ),
"_user.rgw.manifest": buffer::list(len=300, buffer::ptr(136~300 0x55931677c388
in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.pg_ver": buffer::list(len=8, buffer::ptr(0~8 0x559319124000 in raw
0x559319124000 len 4008 nref 1) ),
"_user.rgw.source_zone": buffer::list(len=4, buffer::ptr(1122~4 0x55931677c762
in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.tail_tag": buffer::list(len=44, buffer::ptr(75~44 0x55931958e04b in
raw 0x55931958e000 len 4194304 nref 2) ),
"_user.rgw.x-amz-content-sha256": buffer::list(len=65, buffer::ptr(716~65
0x55931677c5cc in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.x-amz-date": buffer::list(len=17, buffer::ptr(800~17 0x55931677c620
in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.x-amz-meta-s3cmd-attrs": buffer::list(len=173, buffer::ptr(848~173
0x55931677c650 in raw 0x55931677c300 len 1126 nref 9) ),
"_user.rgw.x-amz-storage-class": buffer::list(len=9, buffer::ptr(1049~9
0x55931677c719 in raw 0x55931677c300 len 1126 nref 9) ),
"snapset": buffer::list(len=35, buffer::ptr(0~35 0x559319127000 in raw
0x559319127000 len 4008 nref 1) )
Theoretically with 300 pg per osd and EC 8+3 and
osd_pg_object_context_cache_count=64
and rgw_obj_stripe_size=4M
this cache can consume up to 300/11*64*4M = 6.9G
just because of this side effect of shared buffer::raw.
We not see so high used memory just because rgw not set xattrs on
all rados objects parts of big S3 object.
But with synthetic test with all s3 objects of size 4M it can be easily
achieved.
Thanks,
Aleksei
On 06/29/2018 03:30 AM, Gregory Farnum wrote:
Can you talk more about how you identified this as an issue and came
up with the potential solutions you've identified?
Naively, if I'm told that larger objects make the OSD take up more
memory, it sounds to me like the OSD is probably providing more
throughput, and that if you want it to use up less memory you just
ought to change the amount of outstanding IO it lets in to the system.
-Greg
On Thu, Jun 28, 2018 at 1:29 AM, Aleksei Gutikov
<aleksey.gutikov@xxxxxxxxxx> wrote:
NOTE: rgw_max_chunk_size must be equal to rgw_obj_stripe_size, so I mean
both when refer to one.
For example when I changed rgw_obj_stripe_size from 4M to 16M OSD memory
usage increased approx 2.5 times.
This issue was reproduced with erasure-coded pools.
OSD command dump_mempools show that only anon pool bytes increased.
Further investigations show that whole buffer::raw object received from
network
(created in alloc_aligned_buffer() in AsyncConnection.cc:623)
The whole 4M or 16M buffer::raw objects preserved with nref>0 in
PrimaryLogPG::object_contexts
in ObjectContext::attr_cache.
This issue was reproduced on both luminous and master branches.
I see at least two types of improvement:
1) memcpy relatively small parts of buffer::raw when create new
buffer::ptr
For just example with next compile-time configuration parameters:
BUFFER_MIN_SIZE_COPY_FROM = 64k
BUFFER_MAX_SIZE_TO_COPY = 16k
BUFFER_MIN_RATIO_TO_COPY = 128
will copy up to 512 bytes from 64k raw object
or will copy up to 16k from 4M object
will not copy from 63k raw object
Pros: will improve all issues of this type (preservation of buffer::raw
objects)
Cons: unknown impact, memory fragmentation for example
2) Improvements related particularly to PrimaryLogPG::object_contexts
2.1) Set osd_pg_object_context_cache_count into 1 or 0
Cons: cache will not actually work
2.2) Recreate bufferlists of attr_cache entries during inserting into
cache to copy attrs and free huge buffer later.
Pros: minimal impact on any other subsystems
Cons: will improve only this particular case
2.3) Limit object_contexts with total used memory also in addition to
osd_pg_object_context_cache_count.
Cons: cache will probably will not work because each entry will
occupy
lot of memory and all entries will be skipped.
2.4) Remove object_contexts completely, create contexts every time on
fly.
Cons: object_contexts not looks like spare part that can be safely
removed
We tested osd_pg_object_context_cache_count=1 as hotfix
and it improved OSD memory usage significantly without dependency from
rgw_obj_stripe_size.
Can, please, somebody clarify a little bit about purpose of
PrimaryLogPG::object_contexts.
And, maybe suggest something about fixing this issue.
--
Best regards,
Aleksei Gutikov
Software Engineer | synesis.ru | Minsk. BY
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best regards,
Aleksei Gutikov
Software Engineer | synesis.ru | Minsk. BY
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
