Hi folks
We saw this at two different customer sites, one running 3.0.5 and one
running 3.1.1. This is a distribute layout, no replication. Pretty
intensive loads on the siCluster.
Every now and then there was a hiccup:
258% ls -l /x/y/.z/t.xml
-rw-r--r-- 1 a b 574 Dec 9 07:10
/x/y/.z/t.xml
259% cat !$
cat /x/y/.z/t.xml
cat: /x/y/.z/t.xml: Input/output error
260% df -h !$
df -h /x/y/.z/t.xml
Filesystem Size Used Avail Use% Mounted on
a.b.c.d:/nfs 15T 100G 15T 1% /x
The file was really there. And it was really inaccessible. Remounts
sometimes helped. Logs contained access errors. File system on backend
was fine, no issues. Network was fine, no issues.
Way back in 2.0.x days, there were caching bugs. stat-prefetch and
others were problematic. I remembered this. So we tried this at the
3.0.5 customer site (set timeout to zero and remove stat-prefetch)
volume quickread
type performance/quick-read
option cache-timeout 0
option max-file-size 64kB
subvolumes iocache
end-volume
#volume statprefetch
# type performance/stat-prefetch
# subvolumes quickread
#end-volume
and it seemed to fix it (changed cache-timeout to 0 and commented out
stat-prefetch). Still testing, but it seems to have made that bug less
problematic.
On 3.1.1, there are no config files. And there is no comprehensive list
of volume/translator switches, nor any real clue as to what the
translator layers are in use.
So we experimented until we found this:
[root at manager ~]# gluster volume set nfs performance.cache-refresh-timeout 0
Set volume successful
[root at manager ~]# gluster volume set nfs performance.stat-prefetch 0
Set volume successful
These seem to have worked. And we are testing to see if they fix the
problem.
Has someone somewhere compiled a complete list of the settable config
elements and their meanings? I don't know if the settings simply set
key-value pairs, or if they actually impact things. A listing of these
would be nice, though I found what we found by some creative guessing
based upon older configurations.
Regards,
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
