Re: Repeated lvcreate&lvremove quickly increases page cache size

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dne 02. 05. 24 v 7:47 Ryotaro Banno (伴野 良太郎) napsal(a):
Hi all,

When I run repeatedly `lvm lvcreate` and `lvm lvremove`, I have noticed that
the page cache size increases gradually. For example, when I used the following
commands:

```
while :; do
   sudo lvm lvcreate -n 7ee0d142-bffe-459f-87cb-56a355d24d2c -L 104857600b -W y -y ubuntu-vg;
   sudo lvm lvremove -f ubuntu-vg/7ee0d142-bffe-459f-87cb-56a355d24d2c;
   sleep 0.1;
done
```

free(1)'s buff/cache increased by about 50KiB/s on average.

This phenomenon would not be a problem if I ran LVM on the host.  In fact, I
run the same commands in a Docker container with relatively small RAM limited
by cgroup.  Then, the container is killed by OOM some time after startup,
presumably due to the increased page cache. Does anyone know how to work around
this problem?  This behavior can be reproduced as follows:

```
# Assume a VG named "ubuntu-vg" is already set up on the host.
docker run --privileged --pid=host --memory=15m -it ubuntu:22.04 bash -c '
while :; do
   /usr/bin/nsenter -m -u -i -n -p -t 1 /sbin/lvm lvcreate -n 7ee0d142-bffe-459f-87cb-56a355d24d2c -L 104857600b -W y -y ubuntu-vg
   /usr/bin/nsenter -m -u -i -n -p -t 1 /sbin/lvm lvremove -f ubuntu-vg/7ee0d142-bffe-459f-87cb-56a355d24d2c
   sleep 0.1
done'

# After a while, lvcreate gets OOM killed like:
#   bash: line 6: 4099016 Killed                  /usr/bin/nsenter -m -u -i -n -p -t 1 /sbin/lvm lvremove -f ubuntu-vg/7ee0d142-bffe-459f-87cb-56a355d24d2c
```

The same thing happens on a Kubernetes container.  For some unknown reason, the
increase in the page cache size is faster on Kubernetes than on others (host
and docker); it only increases, never decreases.  This can be reproduced using
Minikube with KVM2 as follows:

```
# Set up Minikube and LVM
minikube start --driver=kvm2
minikube ssh -- sudo truncate --size=10G backing_store
minikube ssh -- sudo losetup -f backing_store
minikube ssh -- sudo vgcreate vg1 $(minikube ssh -n minikube -- sudo losetup -j backing_store | cut -d':' -f1)

# Create a pod that repeatedly run lvcreate&lvremove
cat <<EOS | minikube kubectl -- apply -f -
apiVersion: v1
kind: Pod
metadata:
   name: test-pod
spec:
   hostPID: true
   containers:
   - command:
     - bash
     - -cex
     - |
       # Run lvcreate&lvremove repeatedly
       while :; do
           /usr/bin/nsenter -m -u -i -n -p -t 1 /sbin/lvm lvcreate -n 7ee0d142-bffe-459f-87cb-56a355d24d2c -L 104857600b -W y -y vg1
           /usr/bin/nsenter -m -u -i -n -p -t 1 /sbin/lvm lvremove -f vg1/7ee0d142-bffe-459f-87cb-56a355d24d2c
           sleep 0.1
       done
     image: ubuntu:22.04
     name: lvm
     securityContext:
       privileged: true

     # Limit memory usage upto 15Mi
     resources:
       limits:
         memory: 15Mi
EOS

# Watch the pod status. It will get OOM killed continuously.
minikube kubectl -- get pod test-pod -w

# Check the memory usage statistics exposed by cgroup. The field `total_cache` should be huge.
minikube ssh -- cat /sys/fs/cgroup/memory/kubepods/burstable/pod$(minikube kubectl -- get pod test-pod -o jsonpath='{.metadata.uid}')/memory.stat

# Clean up
minikube delete --all --purge
```

Does anyone know how to work around this problem?

My environment is as follows:

- OS: Ubuntu 22.04.4 LTS
   - `uname -a`: Linux _ 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- Docker version: 26.0.1
- minikube version: v1.32.0 (commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d)


Hello

Well you could eventually try to use kernel with 'memory leak' enabled detector to see whether there is some kernel leak.

There can't be leak in lvm2 code - since the binary 'exits' - and kernel is responsible for any leaks here.

However I'd first start reproducing your issue with  newer kernel 6.8, 6.9-rcX
As there is nothing more boring than chasing some old possibly already fixed kernel bugs.

You should also probably flush all your page cache between your memory size sampling with: 'echo 3 >/proc/sys/vm/drop_caches'

And final note: lvm2 is NOT designed and should NOT be used within Docker (or any other container) software - it's tightly integrated with your kernel API - while containers are more or less 'virtual machines' for user-space apps. So problems caused by misbehaving udev (which doesn't exists/runs) in your Docker are quite normal and it requires some level of expertise to know how to deal with those issues...



Regards

Zdenek





[Index of Archives]     [Gluster Users]     [Kernel Development]     [Linux Clusters]     [Device Mapper]     [Security]     [Bugtraq]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]

  Powered by Linux