----- Original Message ----- > Hi, > > I've got weird state on GFS2 (activated only on one node from the very > beginning, but with dlm locking), when I'm unable to write with 'No > space left on device' error, but df -m reports: > /dev/mapper/vg_shared-lv_shared 570875 569622 1254 100% /storage/staging > > Umount/mount doesn't help, umount/fsck/rmmod/mount also does nothing. > > That is centos6 with 2.6.32-504.30.3.el6.x86_64 kernel. > > What could be the reason for such desync? > Is there a way to fix that? > > Any help is appreciated, > > thank you, > Vladislav Hi Vladislav, It sounds like maybe your system statfs file has gotten out of sync with the actual free space. We've seen this before, and have bugzilla records open to fix it. https://bugzilla.redhat.com/show_bug.cgi?id=1191219 Ordinarily that should not prevent blocks from being allocated, because unlinked dinodes should be automatically reclaimed as needed. It could be a fragmentation issue: Maybe there's enough free space, but the free space is too fragmented to allow for a required block allocation. So it is difficult to say what exactly is going on. If you want to send me your file system metadata, I'd be happy to examine it and let you know what I find. This can be saved with: gfs2_edit savemeta <device> <output file> The resulting files are often too big to email, so you may need to put it on an ftp server or something instead. Also, bear in mind that GFS2 has a severe performance penalty when your file system is nearly full. The less free space available, the more time it takes to find free space. So you'll probably get much better performance if you make the file system bigger (lvextend the volume then gfs2_grow). Regards, Bob Peterson Red Hat File Systems -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
