I've set up everything and have moved close to 2Tb of data to the glusterFS volume without incident. I thought that I try a recursive long listing just to make sure everything was working properly (in anticipation of doing just that when I turn on AFR soon). Doing ls -alR on mounted GlusterFS volume. Everything looks good, but every so often I get output that looks like: -rw------- 1 zope zope 30512 Jan 27 2008 efb1f22f2312af6ed5fbef014ee3 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1ac485 0e731fed2e0eb2fdbc8d05 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1d7d22 6f5f6845d62fddcadb4ca9 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/284371 d43c24d0d3267109c674e7 If I stop the listing and check the files individually I get: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1ac4850e731fed2e0 eb2fdbc8d05 -rw------- 1 zope zope 23328 Aug 12 2008 /mnt/BACKUPS/blobdata/00/66/1ac4850e73 1fed2e0eb2fdbc8d05 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6845d62 fddcadb4ca9 -rw-rw-r-- 1 zope zope 1616 Nov 2 2007 /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6 845d62fddcadb4ca9 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/284371d43c24d0d32 67109c674e7 -rw-rw-r-- 1 zope zope 34928 Dec 28 2007 /mnt/BACKUPS/blobdata/00/66/284371d43c 24d0d3267109c674e7 [zope at zodb01 prodscripts]$ Nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last few files listed in an individual directory (that is it nevery happens in the middle of a folder). Occasionally I get and error like the following: ls: /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc36324a41227: Permission denied Stopping the listing to check this file results in: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc 36324a41227 -rw-rw-r-- 1 zope zope 512 Nov 22 2007 /mnt/BACKUPS/blobdata/00/84/7590a35e615c 736cc36324a41227 [zope at zodb01 prodscripts]$ Again, nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last file or files listed in an individual directory (that is it nevery happens in the middle of a folder). Sometimes it affects entire folders not just individual files: ls: /mnt/BACKUPS/blobdata/00/e1: Permission denied Stopping the recursive listing and just executing on this single folder works perfectly. [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/e1 total 15976 drwxrwxr-x 2 zope zope 8192 Dec 15 16:46 . drwxrwxr-x 258 zope zope 16384 Dec 15 16:46 .. -rw------- 1 zope zope 909456 Sep 15 2008 00fe85dffc1830fe10f34942c2c5 -rw------- 1 zope zope 399136 Oct 15 2008 0effcac8a7fb6819f17fcaea296f -rw------- 1 zope zope 149456 Jan 27 2008 1ebd7783357503e7b553dcfae23a -rw------- 1 zope zope 19472 Sep 29 10:27 2d3ada857f0e3692146f98cc923b -rw------- 1 zope zope 1440 Jun 21 10:18 302ad12cda83dcc7a7f241f9d7d4 -rw------- 1 zope zope 896 Mar 5 2008 3298c88694ee904cfd0ade6715fa -rw------- 1 zope zope 16 Jun 21 22:06 3ebb81573577e37fff643fbb7dcc -rw-rw-r-- 1 zope zope 132624 Dec 28 2007 415255d42503872b3a88cfe70d6c -rw------- 1 zope zope 416 Jun 26 2008 489e5ef508e56e2d76ba67c306ea -rw------- 1 zope zope 29200 Mar 17 2008 4a9f16e818309cc601bd886631cd -rw------- 1 zope zope 554272 Jan 26 2008 4ac36056b24a9b46e659abc44f49 -rw-rw-r-- 1 zope zope 468160 Jan 8 2008 5204e6d98d70e74240ec91263a46 -rw------- 1 zope zope 401616 Jan 26 2008 5255ecacc6709b64b7633cfb9199 -rw------- 1 zope zope 22096 Aug 4 02:49 590c4138783609d0d11b5d7d04a7 -rw------- 1 zope zope 2362752 Nov 8 19:48 631542f36a07601d678975374428 -rw------- 1 zope zope 1248 Feb 8 2009 6335e89fef0bdb300491a246e489 -rw------- 1 zope zope 875024 Jun 29 2008 6439dde68e0d91909518dce73e65 -rw------- 1 zope zope 1248 Feb 20 2008 6818b0f5686f2f4f7d892f42c5bf -rw------- 1 zope zope 27184 Jul 2 2008 693c281e4f7ae9ced89d0c326411 -rw------- 1 zope zope 1057392 Jan 26 2008 6a0ac5e30a06f3ee7723e4a1eb13 -rw-rw-r-- 1 zope zope 2871776 Dec 28 2007 6d932ed95810a51d7a5dd3852ad9 -rw-rw-r-- 1 zope zope 2996896 Nov 2 2007 73000cfd294042e5fcdc322e1f3d -rw------- 1 zope zope 3392 Oct 28 20:23 77638a97c972a1c1a395324d3d68 -rw-rw-r-- 1 zope zope 21584 Dec 29 2007 78fdd154c8c45aa793c2a4fdfac5 -rw------- 1 zope zope 21008 Jul 26 19:29 79ec9ef2f3937effc6af0f25d5c8 -rw-rw-r-- 1 zope zope 19872 Jan 8 2008 8044fb8c6b68f4484716c9757bfa -rw-rw-r-- 1 zope zope 21168 Jan 10 2008 83074f956ab297c62b3c1a41574c -rw------- 1 zope zope 784 Oct 16 08:56 8c72c89d9d2d86efc8fe1ed71cfe -rw-rw-r-- 1 zope zope 144 Dec 29 2007 99c5f01f3ed0dbdafed07ae38205 -rw------- 1 zope zope 16 Feb 12 2008 a2a02f4aa6ffc7f36103576c36e2 -rw------- 1 zope zope 26640 Sep 12 14:03 a2e55ba90942c56738e5c704c545 -rw------- 1 zope zope 95760 Jul 2 2008 a3c63b279c7edc4b4c797b8f332e -rw-rw-r-- 1 zope zope 77712 Jan 8 2008 ab93c709881a668b6f9f8a92c964 -rw-rw-r-- 1 zope zope 16544 Jan 8 2008 aed5db4a22f81825165b86a99aa1 -rw------- 1 zope zope 5040 Aug 20 2008 c9e589908be1c448737a5fca842c -rw------- 1 zope zope 3184 Jun 30 2008 cd45ed25b3ccea39d0091bbad4d3 -rw-rw-r-- 1 zope zope 534976 Dec 29 2007 cd4a58908bb8b8a8c84c004e258d -rw-rw-r-- 1 zope zope 144 Jan 10 2008 d84cfb4389497fd679bbb7ce4077 -rw------- 1 zope zope 4800 May 30 2008 ed0c4a25941795c2d216355af1d4 -rw------- 1 zope zope 1362768 Oct 7 06:12 edec7b082feab6b5ab923094954c -rw------- 1 zope zope 37392 Aug 4 17:48 ef358bc52a503dc9302526af3ce1 -rw------- 1 zope zope 1616 Jun 26 2008 f221db1d18f01ebe93caf29fe730 -rw------- 1 zope zope 400 Jan 20 2008 f4c43212930051c45da5163fba15 -rw------- 1 zope zope 10064 Aug 24 2008 f612a2284de15ad71b166132bb8b -rw-rw-r-- 1 zope zope 416704 Dec 28 2007 fbfca80d116b0d092aacd6712260 -rw------- 1 zope zope 640 Nov 26 16:13 fec8f41e8306758f238983948c64 -rw------- 1 zope zope 30224 Mar 16 2008 ffbda6d769a1bb8e73a8e966914f Ideas? Larry
