Re: pjsip Digest, Vol 173, Issue 1 memory leak with media port connected to a sound device

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Andreas,
    since the process should be always active with the board, I have not arranged a graceful shutdown, but when I exit from the process all the memory is released.

What I do is to create a media port to send a tone and receive its echo, to create a sound port and to connect them. I have verified - until the call at the free() function - that the pools are released. I have also created a second pool factory which I destroy after the media port and the sound device are released. I have the suspect that the stack of the thread involved with the sound port is not released. However after the and of this thread pj_thread_join() is called.

I implemented a workaround, performing this function in a separate thread out of the pjsip framework. In this way the memory does not grow.



Il 08/02/2022 09:30, pjsip-request@xxxxxxxxxxxxxxx ha scritto:
Send pjsip mailing list submissions to
	pjsip@xxxxxxxxxxxxxxx

To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
	pjsip-request@xxxxxxxxxxxxxxx

You can reach the person managing the list at
	pjsip-owner@xxxxxxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of pjsip digest..."

Today's Topics:

    1. Re: pjsip Digest, Vol 172, Issue 3 (MÅrîØ Raƒƒin)
    2. Re: pjsip Digest, Vol 172, Issue 3 (Andreas Wehrmann)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Feb 2022 17:27:49 +0100
From: MÅrîØ Raƒƒin <raffin@xxxxxxxxxxxxxx>
Subject:  Re: pjsip Digest, Vol 172, Issue 3
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <217a3d74-9328-51af-9175-73b5e3e9ac7f@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8; format=flowed

Thank you for your suggestion Christian,

     I tried to call malloc_trim() each time I close the audio device
after releasing the cache pools.

It has a beneficial effect on the short term, now I see sometimes that
the RSS occupation decreases, but on the long term the used still memory
grows up.

Since I have not malloc() and free() in this part of code which manages
the audio, and the pools are released I don't think the issue should
caused by a malloc() without its free().

What I noticed from the debugger is that the memory increases when I
enter on the thread function which manages the audio capture and playback.
I have some suspect about the thread stack.
Howaver this thread is created with the pj_thread_create() and when the
audio is closed the pj_thread_join() and the pj_thread_destroy() are
always called.

mario

Il 05/02/2022 09:30, pjsip-request@xxxxxxxxxxxxxxx ha scritto:
Send pjsip mailing list submissions to
	pjsip@xxxxxxxxxxxxxxx

To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
	pjsip-request@xxxxxxxxxxxxxxx

You can reach the person managing the list at
	pjsip-owner@xxxxxxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of pjsip digest..."

Today's Topics:

     1. Re: pjsip Digest, Vol 172, Issue 2, Memory Leak from pjmedia_snd_port_create2
        (MÅrîØ Raƒƒin)
     2. Re: pjsip Digest, Vol 172, Issue 2, Memory Leak from pjmedia_snd_port_create2
        (Christian Hoff)


----------------------------------------------------------------------

Message: 1
Date: Fri, 4 Feb 2022 18:19:28 +0100
From: MÅrîØ Raƒƒin <raffin@xxxxxxxxxxxxxx>
Subject:  Re: pjsip Digest, Vol 172, Issue 2, Memory Leak from
	pjmedia_snd_port_create2
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <7cec20f2-1784-b91b-5c63-e4dea53f6b7d@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8; format=flowed

Dear Members,
       I further investigated the memory leak I am experimentig.

The steps to get this leak are the following:

1. create a cache pool from the factory using "pj_pool_create"
2. create a media port to generate a 40 ms sine wave samples e record
from microphone using "pjmedia_port_info_init"
3. create a sound port "pjmedia_snd_port_create2"
4. connect the the media port to the sound device "pjmedia_snd_port_connect"
5. after the 40ms tone is output and the echo has been received
disconnect the port "pjmedia_snd_port_disconnect"
6. release the pool (pj_pool_release)

Calling this each minute the heap size increases until I get in one day,
an out of memory (at the beninning I exploit only 3.3% of the available
memory).
In this code all the memory is allocated int the pool using
pj_pool_alloc. I also tried to remove all the calls to the ALSA driver
with any improvement.

What I observed is that if I only allocate/deallocate the media port the
leak is not present. It start when I call "pjmedia_snd_port_create2"
even if I don't connect the port to the sound device.

Moreover dumping the cache pools I find their size is stable and reasonable.
Nevertheless, pmap shows an increasing size of the heap used by the
process.
Andreas Wehrmann suggested a possible memory fragmentation issue but I
am not sure it could cause such a dramatic memory waste and I can not
explain why pjmedia_snd_port_create2 create this issue while the
allocation of the media port, which is more demanding, does not produce
the issue.

The alsa audio device starts a thread which captures the audio frames
and sends frames to the audio dev.  I have some suspect on the stack
allocation of this thread but it should use the memory pools (I use
"pj_thread_create") so everithing should be released with the pool.

I have run out the ideas on the possible cause of this leak, every hint
will be extremely appreciated.

mario


Il 29/01/2022 09:30, pjsip-request@xxxxxxxxxxxxxxx ha scritto:
Send pjsip mailing list submissions to
	pjsip@xxxxxxxxxxxxxxx

To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
	pjsip-request@xxxxxxxxxxxxxxx

You can reach the person managing the list at
	pjsip-owner@xxxxxxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of pjsip digest..."

Today's Topics:

      1. Memory Leak with alsa driver (MÅrîØ Raƒƒin)
      2. Re: Memory Leak with alsa driver (Andreas Wehrmann)


----------------------------------------------------------------------

Message: 1
Date: Fri, 28 Jan 2022 18:24:00 +0100
From: MÅrîØ Raƒƒin <raffin@xxxxxxxxxxxxxx>
Subject:  Memory Leak with alsa driver
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <7244f6b2-38e6-390c-06dc-35c2e205ecf7@xxxxxxxxxxxxxx>
Content-Type: multipart/alternative;
	boundary="------------Id3V26CN7btFojAhitK5fSdz"

Dear All,
        I am debugging an application which exploit the pjsip stack.
I noticed a memory leak when using the alsa driver since each time I open and close the audio, pmap shows and increment on the user memory (4-12 KB each time).

Apparently this increment is in the heap region:

20608:   /home/root/easysip
Address   Kbytes     RSS   Dirty Mode  Mapping
00010000    1324    1324      28 r-x-- /home/root/easysip
0016a000       4       4       4 r---- /home/root/easysip
0016b000      20      20      16 rw--- /home/root/easysip
00170000     524     464     464 rw---   [ anon ]   <--------increment on this area
b09da000       4       0       0 -----   [ anon ]
b09db000    8192       8       8 rw---   [ anon ]
b39d8000       4       0       0 -----   [ anon ]
b39d9000    8192      16      16 rw---   [ anon ]
b41d9000       4       0       0 -----   [ anon ]
b41da000    8192      16      16 rw---   [ anon ]
b49da000       4       0       0 -----   [ anon ]
b49db000    8192      12      12 rw---   [ anon ]
....
b549b000     332     332       0 r-x-- /usr/lib/pulseaudio/libpulsecommon-12.2.so
b54ee000      64       0       0 ----- /usr/lib/pulseaudio/libpulsecommon-12.2.so
b54fe000       4       4       4 r---- /usr/lib/pulseaudio/libpulsecommon-12.2.so
b54ff000       4       4       4 rw--- /usr/lib/pulseaudio/libpulsecommon-12.2.so
b5500000     952     836     836 rw---   [ anon ] <--------then increment on this area
b55ee000      72       0       0 -----   [ anon ]
...

I have spent 10 day looking for a trouble in the code which manages the ALSA but everything looks fine.
In the code there aren't malloc or calloc, all the memory is managed by the pj_pool and the pools used during the audio stream are released.

I noticed that the first 5/6 times I open ond close the audio, the memory does not grow, then it starts to increase. This behaviour make me think to a memory pool which first uses its allocated space and then start allocating more space.

However, the pj_pool_factory_dump() shows that the pools size is stable and there are not pools growing. I use a single pool factory

The only functions which allocate ram outside the pj_pools are snd_pcm_open() which is followed by snd_pcm_close() and
snd_pcm_(hw|sw)_params_malloc() which is always followed by the snd_pcm_(hw|sw)_params_free()

The memory area where the pools are allocated is the same I see growing with pmap.
I wonder wether there is some wrong use of the memory pools which could cause this kind of problems (for instance, what happen if I create two time a pool with the same name? however I didn't found this in my code ).

Is it possible that when the memory of the pool is released, for some reason the system retain it?

Every idea that could help me to debug this issue will be very appreciated.

Thank you.
Best regards.
Mario


------------------------------

Message: 2
Date: Fri, 4 Feb 2022 19:32:51 +0100
From: Christian Hoff <christian_hoff@xxxxxxx>
Subject:  Re: pjsip Digest, Vol 172, Issue 2, Memory Leak from
	pjmedia_snd_port_create2
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <3735ba1d-dfe1-d09c-c3f6-e18926069c3f@xxxxxxx>
Content-Type: text/plain; charset=utf-8; format=flowed

Hello Mario,

at least you should be able to easily verify if memory fragmentation is
the root cause or not: just call malloc_trim() - either from your
program or manually from GDB - after you see the leak. "malloc_trim()"
walks down the whole heap to find unused memory chunks and so should be
able to return the memory back to the OS even if you necounter memory
fragmentation. If the RSS memory size of your program goes back to
normal after calling malloc_trim(), you are dealing with memory
fragmentation. Otherwise it's likely a real leak.


Best regards,

      Christian

On 2/4/22 6:19 PM, MÅrîØ Raƒƒin wrote:
Dear Members,
      I further investigated the memory leak I am experimentig.

The steps to get this leak are the following:

1. create a cache pool from the factory using "pj_pool_create"
2. create a media port to generate a 40 ms sine wave samples e record
from microphone using "pjmedia_port_info_init"
3. create a sound port "pjmedia_snd_port_create2"
4. connect the the media port to the sound device
"pjmedia_snd_port_connect"
5. after the 40ms tone is output and the echo has been received
disconnect the port "pjmedia_snd_port_disconnect"
6. release the pool (pj_pool_release)

Calling this each minute the heap size increases until I get in one
day, an out of memory (at the beninning I exploit only 3.3% of the
available memory).
In this code all the memory is allocated int the pool using
pj_pool_alloc. I also tried to remove all the calls to the ALSA driver
with any improvement.

What I observed is that if I only allocate/deallocate the media port
the leak is not present. It start when I call
"pjmedia_snd_port_create2" even if I don't connect the port to the
sound device.

Moreover dumping the cache pools I find their size is stable and
reasonable.
Nevertheless, pmap shows an increasing size of the heap used by the
process.
Andreas Wehrmann suggested a possible memory fragmentation issue but I
am not sure it could cause such a dramatic memory waste and I can not
explain why pjmedia_snd_port_create2 create this issue while the
allocation of the media port, which is more demanding, does not
produce the issue.

The alsa audio device starts a thread which captures the audio frames
and sends frames to the audio dev.  I have some suspect on the stack
allocation of this thread but it should use the memory pools (I use
"pj_thread_create") so everithing should be released with the pool.

I have run out the ideas on the possible cause of this leak, every
hint will be extremely appreciated.

mario


Il 29/01/2022 09:30, pjsip-request@xxxxxxxxxxxxxxx ha scritto:
Send pjsip mailing list submissions to
      pjsip@xxxxxxxxxxxxxxx

To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
      pjsip-request@xxxxxxxxxxxxxxx

You can reach the person managing the list at
      pjsip-owner@xxxxxxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of pjsip digest..."

Today's Topics:

      1. Memory Leak with alsa driver (MÅrîØ Raƒƒin)
      2. Re: Memory Leak with alsa driver (Andreas Wehrmann)


----------------------------------------------------------------------

Message: 1
Date: Fri, 28 Jan 2022 18:24:00 +0100
From: MÅrîØ Raƒƒin <raffin@xxxxxxxxxxxxxx>
Subject:  Memory Leak with alsa driver
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <7244f6b2-38e6-390c-06dc-35c2e205ecf7@xxxxxxxxxxxxxx>
Content-Type: multipart/alternative;
      boundary="------------Id3V26CN7btFojAhitK5fSdz"

Dear All,
        I am debugging an application which exploit the pjsip stack.
I noticed a memory leak when using the alsa driver since each time I
open and close the audio, pmap shows and increment on the user memory
(4-12 KB each time).

Apparently this increment is in the heap region:

20608:   /home/root/easysip
Address   Kbytes     RSS   Dirty Mode  Mapping
00010000    1324    1324      28 r-x-- /home/root/easysip
0016a000       4       4       4 r---- /home/root/easysip
0016b000      20      20      16 rw--- /home/root/easysip
00170000     524     464     464 rw---   [ anon ] <--------increment
on this area
b09da000       4       0       0 -----   [ anon ]
b09db000    8192       8       8 rw---   [ anon ]
b39d8000       4       0       0 -----   [ anon ]
b39d9000    8192      16      16 rw---   [ anon ]
b41d9000       4       0       0 -----   [ anon ]
b41da000    8192      16      16 rw---   [ anon ]
b49da000       4       0       0 -----   [ anon ]
b49db000    8192      12      12 rw---   [ anon ]
....
b549b000     332     332       0 r-x--
/usr/lib/pulseaudio/libpulsecommon-12.2.so
b54ee000      64       0       0 -----
/usr/lib/pulseaudio/libpulsecommon-12.2.so
b54fe000       4       4       4 r----
/usr/lib/pulseaudio/libpulsecommon-12.2.so
b54ff000       4       4       4 rw---
/usr/lib/pulseaudio/libpulsecommon-12.2.so
b5500000     952     836     836 rw---   [ anon ] <--------then
increment on this area
b55ee000      72       0       0 -----   [ anon ]
...

I have spent 10 day looking for a trouble in the code which manages
the ALSA but everything looks fine.
In the code there aren't malloc or calloc, all the memory is managed
by the pj_pool and the pools used during the audio stream are released.

I noticed that the first 5/6 times I open ond close the audio, the
memory does not grow, then it starts to increase. This behaviour make
me think to a memory pool which first uses its allocated space and
then start allocating more space.

However, the pj_pool_factory_dump() shows that the pools size is
stable and there are not pools growing. I use a single pool factory

The only functions which allocate ram outside the pj_pools are
snd_pcm_open() which is followed by snd_pcm_close() and
snd_pcm_(hw|sw)_params_malloc() which is always followed by the
snd_pcm_(hw|sw)_params_free()

The memory area where the pools are allocated is the same I see
growing with pmap.
I wonder wether there is some wrong use of the memory pools which
could cause this kind of problems (for instance, what happen if I
create two time a pool with the same name? however I didn't found
this in my code ).

Is it possible that when the memory of the pool is released, for some
reason the system retain it?

Every idea that could help me to debug this issue will be very
appreciated.

Thank you.
Best regards.
Mario


_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list -- pjsip@xxxxxxxxxxxxxxx
To unsubscribe send an email to pjsip-leave@xxxxxxxxxxxxxxx
------------------------------

Subject: Digest Footer

_______________________________________________
pjsip mailing list -- pjsip@xxxxxxxxxxxxxxx
To unsubscribe send an email to pjsip-leave@xxxxxxxxxxxxxxx


------------------------------

End of pjsip Digest, Vol 172, Issue 3
*************************************
------------------------------

Message: 2
Date: Mon, 7 Feb 2022 19:31:56 +0100
From: Andreas Wehrmann <a.wehrmann@xxxxxxxxxx>
Subject:  Re: pjsip Digest, Vol 172, Issue 3
To: pjsip@xxxxxxxxxxxxxxx
Message-ID: <db0c900a-6516-5340-adeb-2800822416e6@xxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 07.02.22 17:27, MÅrîØ Raƒƒin wrote:
Thank you for your suggestion Christian,

    I tried to call malloc_trim() each time I close the audio device
after releasing the cache pools.

It has a beneficial effect on the short term, now I see sometimes that
the RSS occupation decreases, but on the long term the used still
memory grows up.

Since I have not malloc() and free() in this part of code which
manages the audio, and the pools are released I don't think the issue
should caused by a malloc() without its free().

What I noticed from the debugger is that the memory increases when I
enter on the thread function which manages the audio capture and
playback.
I have some suspect about the thread stack.
Howaver this thread is created with the pj_thread_create() and when
the audio is closed the pj_thread_join() and the pj_thread_destroy()
are always called.

mario

Hey,
when you step through the shutdown functions of the library, will the
memory come back once the pools are finally released?

Regards,
Andreas

------------------------------

Subject: Digest Footer

_______________________________________________
pjsip mailing list -- pjsip@xxxxxxxxxxxxxxx
To unsubscribe send an email to pjsip-leave@xxxxxxxxxxxxxxx


------------------------------

End of pjsip Digest, Vol 173, Issue 1
*************************************
_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list -- pjsip@xxxxxxxxxxxxxxx
To unsubscribe send an email to pjsip-leave@xxxxxxxxxxxxxxx




[Index of Archives]     [Asterisk Users]     [Asterisk App Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [Linux API]
  Powered by Linux