I just wanted to confirm that I saw just this exact same issue today on a stock RHEL 4 Update 5. Running gfs_tool reclaim made my file system "work" again.
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
ccs-1.0.10-0
ccs-devel-1.0.10-0
cman-1.0.17-0
cman-devel-1.0.17-0
cman-kernel-xenU-2.6.9-50.2
cman-kernheaders-2.6.9-50.2
GFS-6.1.14-0
GFS-kernel-xenU-2.6.9-72.2
magma-1.0.7-1
magma-plugins-1.0.12-0
# uname -rm
2.6.9-55.ELxenU x86_64
--
Dave Costakos
mailto:david.costakos@xxxxxxxxx
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
ccs-1.0.10-0
ccs-devel-1.0.10-0
cman-1.0.17-0
cman-devel-1.0.17-0
cman-kernel-xenU-2.6.9-50.2
cman-kernheaders-2.6.9-50.2
GFS-6.1.14-0
GFS-kernel-xenU-2.6.9-72.2
magma-1.0.7-1
magma-plugins-1.0.12-0
# uname -rm
2.6.9-55.ELxenU x86_64
2008/7/15 Reiner Rottmann <rottmann@xxxxxxx>
Hello everyone,
I've experienced strange behavior on a 20 GB GFS formatted volume (although
same behaviour applies to smaller and larger sizes) when reaching the max
available disk space by writing lots of 256 byte files in a nested directory
structure (~15k files in one dir).
The expected behaviour would be that all free data blocks are transformed to
inodes and metadata as required but although there are still plenty
datablocks free, new 256 byte files cannot be created due to "No space left
on device".
After that, when creating sequential files via touch, it is expected that they
are created till all data blocks are transformed in inodes representing the
files. When all data blocks are used, "No space left on device" is expected.
But in this strange scenario, files are created at random!?
Also when executing gfs_tool reclaim, new files are createable again. But
gfs_tool reclaim only should increase the number of already available free
data blocks by cleaning unused metadata blocks.
In my understanding, it should not be necessary to reclaim blocks, if there
are still free data blocks left.
Has anyone an explanation for this?
Best regards,
Reiner Rottmann
--%<---------------------------------------------------------------------------
(Filesystem filled with 256 byte files.)
# for i in $(seq 1 1000); do touch waste.$i; done
touch: cannot touch `waste.3': No space left on device
touch: cannot touch `waste.6': No space left on device
touch: cannot touch `waste.12': No space left on device
touch: cannot touch `waste.13': No space left on device
touch: cannot touch `waste.15': No space left on device
touch: cannot touch `waste.16': No space left on device
touch: cannot touch `waste.20': No space left on device
touch: cannot touch `waste.25': No space left on device
touch: cannot touch `waste.28': No space left on device
touch: cannot touch `waste.29': No space left on device
touch: cannot touch `waste.32': No space left on device
touch: cannot touch `waste.37': No space left on device
touch: cannot touch `waste.38': No space left on device
touch: cannot touch `waste.39': No space left on device
touch: cannot touch `waste.48': No space left on device
touch: cannot touch `waste.55': No space left on device
touch: cannot touch `waste.56': No space left on device
touch: cannot touch `waste.59': No space left on device
touch: cannot touch `waste.60': No space left on device
touch: cannot touch `waste.63': No space left on device
^C
# for i in $(seq 1 1000); do touch waste2.$i; done
touch: cannot touch `waste2.1': No space left on device
touch: cannot touch `waste2.8': No space left on device
touch: cannot touch `waste2.10': No space left on device
touch: cannot touch `waste2.11': No space left on device
touch: cannot touch `waste2.12': No space left on device
touch: cannot touch `waste2.14': No space left on device
touch: cannot touch `waste2.17': No space left on device
touch: cannot touch `waste2.19': No space left on device
touch: cannot touch `waste2.21': No space left on device
touch: cannot touch `waste2.24': No space left on device
touch: cannot touch `waste2.28': No space left on device
touch: cannot touch `waste2.31': No space left on device
touch: cannot touch `waste2.32': No space left on device
touch: cannot touch `waste2.33': No space left on device
touch: cannot touch `waste2.40': No space left on device
touch: cannot touch `waste2.43': No space left on device
touch: cannot touch `waste2.44': No space left on device
touch: cannot touch `waste2.49': No space left on device
touch: cannot touch `waste2.54': No space left on device
touch: cannot touch `waste2.55': No space left on device
touch: cannot touch `waste2.57': No space left on device
touch: cannot touch `waste2.58': No space left on device
touch: cannot touch `waste2.61': No space left on device
^C
# gfs_tool df .
/mnt/gfstest:
SB lock proto = "lock_dlm"
SB lock table = "axqa01:gfstest"
SB ondisk format = 1309
SB multihost format = 1401
Block size = 1024
Journals = 3
Resource Groups = 78
Mounted lock proto = "lock_dlm"
Mounted lock table = "axqa01:gfstest"
Mounted host data = ""> Journal number = 0
Lock module flags =
Local flocks = FALSE
Local caching = FALSE
Oopses OK = FALSE
Type Total Used Free use%
------------------------------------------------------------------------
inodes 18343309 18343309 0 100%
metadata 1690156 1687524 2632 100%
data 43931 0 43931 0%
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
GFS-6.1.15-1
GFS-kernel-2.6.9-60.9
GFS-kernel-2.6.9-75.11
GFS-kernel-smp-2.6.9-60.9
GFS-kernel-smp-2.6.9-75.11
ccs-1.0.11-1
cman-1.0.17-0.el4_6.3
cman-kernel-smp-2.6.9-45.15
cman-kernel-smp-2.6.9-53.8
magma-1.0.8-1
magma-devel-1.0.8-1
magma-plugins-1.0.12-0
# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
# uname -a
Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
--%<---------------------------------------------------------------------------
--
Gruss / Regards,
Dipl.-Ing. (FH) Reiner Rottmann
Phone: +49-89 452 3538-12
http://www.atix.de/
http://open-sharedroot.org/
PGP Key ID: 0xCA67C5A6
PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6
**
ATIX Informationstechnologie und Consulting AG
Einsteinstr. 10
85716 Unterschleissheim
Deutschland/Germany
Phone: +49-89 452 3538-0
Fax: +49-89 990 1766-0
Registergericht: Amtsgericht Muenchen
Registernummer: HRB 168930
USt.-Id.: DE209485962
Vorstand:
Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)
Vorsitzender des Aufsichtsrats:
Dr. Martin Buss
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Dave Costakos
mailto:david.costakos@xxxxxxxxx
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
