On 31.03.2021 20:27, Paul Ramsey wrote:
On Mar 31, 2021, at 11:24 AM, Tom Lane <tgl@xxxxxxxxxxxxx> wrote:
If that's not it, then the leak must be accumulating through plain
old malloc calls. There's not much of that in the core backend
(although if you use ispell text search dictionaries, maybe [1] is
relevant), so my suspicions would next fall on any extensions you
might be using.
Would be interested in the queries being run. We have a reproduceable leak in <-> geography operator that we have been unable to track down.
I tried to get any idea where the memory is by feeding my core file into
chap. I ended up not knowing how to proceed further. I found a
std::vector which was roughly 70 MB in size and a lot of allocations
chap could not identify where they belonged to:
chap> summarize used
Unrecognized allocations have 23098683 instances taking
0xe6973688(3,868,669,576) bytes.
Unrecognized allocations of size 0x68 have 8214926 instances taking
0x32ec61b0(854,352,304) bytes.
Unrecognized allocations of size 0x28 have 4890673 instances taking
0xba907a8(195,626,920) bytes.
Unrecognized allocations of size 0x38 have 3255995 instances taking
0xade38e8(182,335,720) bytes.
Unrecognized allocations of size 0x48 have 2096221 instances taking
0x8fefa28(150,927,912) bytes.
Unrecognized allocations of size 0x78 have 1792042 instances taking
0xcd153b0(215,045,040) bytes.
Unrecognized allocations of size 0x88 have 1393038 instances taking
0xb4ad370(189,453,168) bytes.
Unrecognized allocations of size 0x58 have 657193 instances
taking
0x3727618(57,832,984) bytes.
Unrecognized allocations of size 0xa8 have 360742 instances
taking
0x39cc0f0(60,604,656) bytes.
Unrecognized allocations of size 0x18 have 270834 instances
taking
0x632eb0(6,500,016) bytes.
Unrecognized allocations of size 0x98 have 60372 instances taking
0x8c05e0(9,176,544) bytes.
Unrecognized allocations of size 0xb8 have 37559 instances taking
0x697388(6,910,856) bytes.
Unrecognized allocations of size 0xd8 have 19033 instances taking
0x3ebb18(4,111,128) bytes.
Unrecognized allocations of size 0x128 have 19010 instances
taking
0x55dc50(5,626,960) bytes.
Unrecognized allocations of size 0xe8 have 9526 instances taking
0x21b8f0(2,210,032) bytes.
Unrecognized allocations of size 0x118 have 9513 instances taking
0x28a4d8(2,663,640) bytes.
Unrecognized allocations of size 0x1f8 have 9499 instances taking
0x490d28(4,787,496) bytes.
Unrecognized allocations of size 0x408 have 292 instances taking
0x49920(301,344) bytes.
Unrecognized allocations of size 0x2008 have 194 instances taking
0x184610(1,590,800) bytes.
Unrecognized allocations of size 0x1008 have 155 instances taking
0x9b4d8(636,120) bytes.
Unrecognized allocations of size 0x4008 have 137 instances taking
0x224448(2,245,704) bytes.
Unrecognized allocations of size 0x8008 have 133 instances taking
0x428428(4,359,208) bytes.
Unrecognized allocations of size 0x10008 have 131 instances
taking
0x830418(8,586,264) bytes.
Unrecognized allocations of size 0x40008 have 131 instances
taking
0x20c0418(34,341,912) bytes.
Unrecognized allocations of size 0x100008 have 130 instances taking
0x8200410(136,315,920) bytes.
Unrecognized allocations of size 0x20008 have 129 instances
taking
0x1020408(16,909,320) bytes.
Unrecognized allocations of size 0x80008 have 129 instances
taking
0x4080408(67,634,184) bytes.
Unrecognized allocations of size 0x200008 have 129 instances taking
0x10200408(270,533,640) bytes.
Unrecognized allocations of size 0x400008 have 128 instances taking
0x20000400(536,871,936) bytes.
Unrecognized allocations of size 0x418 have 53 instances taking
0xd8f8(55,544) bytes.
[...]
Pattern %VectorBody has 57401 instances taking 0x42eaaf0(70,167,280) bytes.
[...]
Pattern %MapOrSetNode has 51362 instances taking 0x3f2f40(4,140,864) bytes.
let me know if it would be interesting for you to have this (huge) core
file with memory dump available somewhere.
I have executed a few minutes of queries on a single backend setup. The
RssAnon was growing during this time by about 10 MB in size.
Full query logs are here:
https://downloads.osm-tools.org/postgresql-2021-04-03_183913.csv.gz
I have the following extensions installed. hstore is heavily used and a
little bit postgis.
gis=# SELECT extname FROM pg_extension;
extname
-----------
plpgsql
adminpack
postgis
hstore
(4 rows)
as per other suggestions I enabled auto_explain as well.
I hope you can get a hint what is allocating the memory. I could offer
to share some docker images to reproduce, but It would require a few
commands to set it up and a few hundred megabytes of download.
Stephan
