On Wed, 26 Feb 2020 at 23:56, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
> Have you tried dumping the mempools? ...
> One reason this can happen for example is if you
> have a huge number of PGs (like many thousands per OSD).
We are relying on the pg autoscaler to set the PGs, and so far it
seems to do the right thing.
root@cnx-16:~# ceph daemon osd.158 dump_mempools
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 4794632,
"bytes": 38357056
},
"bluestore_cache_data": {
"items": 142,
"bytes": 2265088
},
"bluestore_cache_onode": {
"items": 14217,
"bytes": 9326352
},
"bluestore_cache_other": {
"items": 1886849,
"bytes": 44817175
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 7,
"bytes": 5096
},
"bluestore_writing_deferred": {
"items": 59,
"bytes": 296579
},
"bluestore_writing": {
"items": 49,
"bytes": 172642
},
"bluefs": {
"items": 2133,
"bytes": 65512
},
"buffer_anon": {
"items": 14614,
"bytes": 4312428946
},
"buffer_meta": {
"items": 8751,
"bytes": 770088
},
"osd": {
"items": 95,
"bytes": 1236520
},
"osd_mapbl": {
"items": 50,
"bytes": 8278656
},
"osd_pglog": {
"items": 65858895,
"bytes": 5338210037
},
"osdmap": {
"items": 174218,
"bytes": 5109376
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 72754711,
"bytes": 9761339123
}
}
}
OSD.158 pglog is 5GB (the process is using 11GB) - and we are seeing
lots of these:
OSD_SLOW_PING_TIME_BACK Long heartbeat ping times on back interface
seen, longest is 12252.480 msec
Slow heartbeat ping on back interface from osd.261 to osd.158 12252.480 msec
Slow heartbeat ping on back interface from osd.312 to osd.158 10729.374 msec
Slow heartbeat ping on back interface from osd.216 to osd.158 9827.114 msec
Slow heartbeat ping on back interface from osd.265 to osd.158 8931.290 msec
Slow heartbeat ping on back interface from osd.168 to osd.158 8889.351 msec
Slow heartbeat ping on back interface from osd.159 to osd.158 8389.767 msec
Slow heartbeat ping on back interface from osd.290 to osd.158 8283.217 msec
Slow heartbeat ping on back interface from osd.221 to osd.158 8190.528 msec
Slow heartbeat ping on back interface from osd.197 to osd.158 7973.119 msec
Slow heartbeat ping on back interface from osd.96 to osd.158 7901.591 msec
Truncated long network list. Use ceph daemon mgr.#
dump_osd_network for more information
so something is leaking memory and/or traversing some data structure
is limiting the ability of the OSD to accept/transmit responses to
neighbors.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
