Stuart D. Gathman schrieb:
On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:
Out of memory: kill process 2220 (getty) score 24 or a child
Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...
Rebooting in 20 seconds..
Again, replying to myself, as no one on this list seem to have a clue
about problems with LVM snapshots... I added RAM, and was able to boot
again.
I don't have any answers, but I am very impressed by this kind of testing
and grateful for the contribution. My thoughts on evil experiments
focused on timing and component failure. Obviously, resource limits
are extremely important as well.
I suppose ideal behaviour would be to refuse to create a snapshot when
memory is "low". But defining "low" can be tricky.
Not really.
We can still have plenty memory when we want to create a snapshot - but
memory can shrink later, as the snapshot fills.
As I understand, and as experimentation shown, memory usage increases as
the snapshot gets filled, with 3 MB of RAM per 1 GB of snapshot filled
(more or less).
As the memory used for a snapshot is not swappable, from a certain
point, it is possible to realize "I'm in trouble, but I can't do anything".
Let's watch at the process once again:
1. Make a snapshot of some large volume (it's easier than making
multiple small snapshots). We still have plenty of RAM, so there is no
sense in refusing to create a snapshot.
2. Start filling the snapshot by either writing to the original, or to
the snapshot. Memory usage increases.
3. As the snapshot fills, and you have less and less memory, from some
point, you will no longer be able to remove the snapshot.
4. Now, you can only watch as the snapshot grows, your memory shrinks,
the processes are getting killed, and eventually, the kernel panics. If
you only have remote access, you're screwed.
Perhaps, in 1., instead of refusing to make a snapshot, make a printk,
so that the admin knows what may happen? Is the memory used for a
snapshot possible to calculate? Is it always ~3 MB RAM per 1 GB changes,
or is it sometimes 1 MB, and sometimes 4 MB, depending on which data
changes? If it remains more or less the same, add up all space reserved
for snapshots, if it's more than we have, print a message.
BTW, wasn't it like that in some kernels before 2.6.22? With 2.6.18, I
was getting a "cannot allocate memory" when I tried to create several
snapshots. I guess it was a bug, though.
Ideally would be to make the memory needed for snapshots swappable, but
I'm not sure the kernel would really like it.
--
Tomasz Chmielewski
http://wpkg.org
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
