02.09.2015 15:36, Bob Peterson wrote:
----- Original Message -----
Hi,
I've got weird state on GFS2 (activated only on one node from the very
beginning, but with dlm locking), when I'm unable to write with 'No
space left on device' error, but df -m reports:
/dev/mapper/vg_shared-lv_shared 570875 569622 1254 100% /storage/staging
Umount/mount doesn't help, umount/fsck/rmmod/mount also does nothing.
That is centos6 with 2.6.32-504.30.3.el6.x86_64 kernel.
What could be the reason for such desync?
Is there a way to fix that?
Any help is appreciated,
thank you,
Vladislav
Hi Vladislav,
Thank you Bob for a rapid answer!
It sounds like maybe your system statfs file has gotten out of sync with
the actual free space. We've seen this before, and have bugzilla records
open to fix it. https://bugzilla.redhat.com/show_bug.cgi?id=1191219
That is not public unfortunately. Anyways, I'm willing to help to fix
that as the project I'm currently work on is relying on GFS2 to be fast,
stable and free from such defects.
Ordinarily that should not prevent blocks from being allocated, because
unlinked dinodes should be automatically reclaimed as needed.
It could be a fragmentation issue: Maybe there's enough free space, but
the free space is too fragmented to allow for a required block allocation.
So it is difficult to say what exactly is going on. If you want to send
me your file system metadata, I'd be happy to examine it and let you
know what I find. This can be saved with: gfs2_edit savemeta <device> <output file>
The resulting files are often too big to email, so you may need to put
it on an ftp server or something instead.
Yes, that is a filesystem used only for tests so I can provide its
metadata. I'll sent a link to you when it is ready.
Should fs be unmounted btw?
Also, bear in mind that GFS2 has a severe performance penalty when your
file system is nearly full. The less free space available, the more time
it takes to find free space. So you'll probably get much better performance
if you make the file system bigger (lvextend the volume then gfs2_grow).
That is actually a desired behavior, more, I use controlled IO
throttling when it becomes full. That is temporary tier-0 area, all
files go to another storage asynchronously.
Thank you,
Vladislav
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
