Hey folks,
I noticed some behaviour which I was not expecting at all and after hours of debugging I am getting somewhere, but still any help understanding what is happening would be appreciated.
I was running a PostgreSQL server within a docker container, and after starting to generate some moderate load (I/O heavy), I noticed the memory usage was going up (reaching the container limit) and down, which reminded me of a Heap GC event, but it was not.
The container limit was 16GB of RAM, and as soon as this limit was reached, there was no restart or OOM errors, rather than a huge drop in memory (image `ContainerRelativeAbsolute`).
After digging into this issue I found that this is file cache (image `Below CLI`), and by default this measurement is included to docker stats CLI, and is the default way of measuring in some observability tools as far as I can tell.
I have two questions:
1. Should the file cache be indeed part of the reported memory that is allocated to the container? The issue I see is that if we have an alerting system, for example, on 90% of memory used in a container, this definitely triggers many false positives. If we ignore the file cache, could this lead to any reliability issues?
2. What is happening on the OS level when suddenly 15GB of file cache is getting erased (image `ContainerRelativeAbsolute`)? I would expect for incremental deletes rather than so many GB of cache being evicted.
I noticed some behaviour which I was not expecting at all and after hours of debugging I am getting somewhere, but still any help understanding what is happening would be appreciated.
I was running a PostgreSQL server within a docker container, and after starting to generate some moderate load (I/O heavy), I noticed the memory usage was going up (reaching the container limit) and down, which reminded me of a Heap GC event, but it was not.
The container limit was 16GB of RAM, and as soon as this limit was reached, there was no restart or OOM errors, rather than a huge drop in memory (image `ContainerRelativeAbsolute`).
After digging into this issue I found that this is file cache (image `Below CLI`), and by default this measurement is included to docker stats CLI, and is the default way of measuring in some observability tools as far as I can tell.
I have two questions:
1. Should the file cache be indeed part of the reported memory that is allocated to the container? The issue I see is that if we have an alerting system, for example, on 90% of memory used in a container, this definitely triggers many false positives. If we ignore the file cache, could this lead to any reliability issues?
2. What is happening on the OS level when suddenly 15GB of file cache is getting erased (image `ContainerRelativeAbsolute`)? I would expect for incremental deletes rather than so many GB of cache being evicted.
Cheers,
Costa
Attachment:
BelowCLI.png
Description: PNG image
Attachment:
ContainerRelativeAbsolute.png
Description: PNG image
