Today (2009-08-12) I'm using:
kernel-2.6.18-128.2.1.el5
autofs-5.0.1-0.rc2.102.el5_3.1
Look my last test:
--------------------------------------------------------------
[root@KSTATION areas]# time ls testdown
ls: testdown: No such file or directory
real 3m9.025s
user 0m0.000s
sys 0m0.002s
Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
mounting root /misc/areas, mountpoint testdown, what
1.2.3.4:/areas/testdown, fstype nfs4, options
acl,sec=krb5p,proto=tcp,retry=0
Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
acl,sec=krb5p,proto=tcp,retry=0 using module nfs4
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
root=/misc/areas name=testdown what=1.2.3.4:/areas/testdown,
fstype=nfs4, options=acl,sec=krb5p,proto=tcp,retry=0
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
nfs options="acl,sec=krb5p,proto=tcp,retry=0", nosymlink=0, ro=0
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
calling mkdir_path /misc/areas/testdown
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
calling mount -t nfs4 -s -o acl,sec=krb5p,proto=tcp,retry=0
1.2.3.4:/areas/testdown /misc/areas/testdown
Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
3078093712 path /misc
Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
submounts remaining in /misc
Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
3078093712 path /misc stat 3
Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
exp 3078093712 finished, switching from 2 to 1
Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
= 2 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
3078093712 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
submounts remaining in /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
3078093712 path /misc stat 3
Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
exp 3078093712 finished, switching from 2 to 1
Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
= 2 path /misc
Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
server '1.2.3.4' failed: timed out (giving up).
Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token = 17
Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/testdown
Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
--------------------------------------------------------------
2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
> Carlos André wrote:
>> Hi Ian,
>> I'm getting crazy trying put "retry=" to work on mount... this option
>> just DONT WORK if use proto=tcp and/OR kerberos (sec=krb5/krb5i/krb5p)
>> like you can see on my previous emails...
>
> Right, my mistake for not looking closely enough at post.
>
> Maybe this is related to the same sort of problem we had with mount in
> the past, before the options parsing went into the kernel, where other
> services, like portmapper (or rpcbind), were being done with different
> timeout parameters before the RPC calls for mounting. That's just an
> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>
> But what version of autofs and kernel did you say you were using?
>
>>
>> I appreciate any help.
>>
>> Carlos.
>>
>>
>> 2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
>>> Chuck Lever wrote:
>>>> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>>>>> This long timeout is good if workstation need mount a critical
>>>>> directory using /etc/fstab on boot (for example)..
>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>> down for any reason, and will again hangs if user try access directory
>>>>> pointing to a NFS down server...
>>>> "retry=0" means the mount command will fail as soon as the first
>>>> mount(2) system call fails. When you set SYN retries to 1, this means
>>>> after 9 seconds, the connect fails, and that causes the mount(2) system
>>>> call to fail.
>>>>
>>>> Recent conversations with Ian suggested that a long timeout was desired
>>>> for automounter as well as other cases. Ian, is there something else we
>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>> mount points handled via automounter? How should mount.nfs wait so we
>>>> don't make other use cases worse? (Looks like most of the history is
>>>> intact below).
>>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>>> mount.nfs in particular). This has always been a difficult situation for
>>> the automounter because interactive mount invocations should wait. But I
>>> believe automount mounts should always time out quickly, but that leads
>>> to its own set of problems, especially when home directories are concerned.
>>>
>>> I think adding "retry=0" is the right thing to do myself but I'm not
>>> certain that will work as we expect. I'll have to do some experimentation.
>>>
>>>> How long do you think is appropriate for the automounter to wait if the
>>>> server is down, in your case, Carlos?
>>>>
>>>>> Am losing something or there have was something weirdo...!?
>>>>> ------------------------------------------------
>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries [DEFAULT]
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=tcp,retry=1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 3m9.000s
>>>>> user 0m0.002s
>>>>> sys 0m0.001s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp,retry=1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 3m9.000s
>>>>> user 0m0.000s
>>>>> sys 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=tcp,retry=0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 3m9.001s
>>>>> user 0m0.000s
>>>>> sys 0m0.003s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp,retry=0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 3m9.001s
>>>>> user 0m0.002s
>>>>> sys 0m0.001s
>>>>>
>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]
>>>>>
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=tcp,retry=1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 1m3.002s
>>>>> user 0m0.000s
>>>>> sys 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp,retry=1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 2m6.000s
>>>>> user 0m0.000s
>>>>> sys 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=tcp,retry=0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 0m9.003s
>>>>> user 0m0.001s
>>>>> sys 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp,retry=0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real 2m6.001s
>>>>> user 0m0.001s
>>>>> sys 0m0.002s
>>>>> [root@KSTATION ~]#
>>>>> ------------------------------------------------
>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>> using retry=0 without kerberos I got only 9s...
>>>>>
>>>>> *sigh*
>>>>>
>>>>>
>>>>>
>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>> Right. Normally the RPC client calls the kernel's socket connect
>>>>>> function,
>>>>>> which does 6 SYN retries. That one call usually takes longer than
>>>>>> the RPC
>>>>>> client's connect timeout, so it only makes one connect call, and then
>>>>>> fails.
>>>>>>
>>>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>>>> client
>>>>>> to retry the connect call until its connect timeout expires. Each
>>>>>> connect
>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>
>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>> sec=krb5p,proto=tcp
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real 3m9.000s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.002s
>>>>>>>
>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>> sec=krb5p,proto=tcp ("retry=1" = no change)
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real 2m6.004s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.004s
>>>>>>>
>>>>>>> (3,6,3,6... secs interval)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/8/10 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>
>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>>> 2049...
>>>>>>>> I tried use "retry=1" option on mount without any change... I dont
>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>
>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>>>> Bruce, no... you're right. I'm describing a situation where my
>>>>>>>>>> server
>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>>>>> and 9 seconds...
>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>> give up
>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>>> exponential retries, or something like that). For stock CentOS
>>>>>>>>> 5.3, I
>>>>>>>>> think
>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>> kernel
>>>>>>>>> just
>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>>>>>>> request.
>>>>>>>>>
>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>> components
>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>
>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@xxxxxxxxxxxx>:
>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@xxxxxxxxx>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/7/29 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>> command
>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>>>>>>> proto=udp)
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real 3m9.001s) until show error
>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>> I thought he was describing a situation where the server the server
>>>>>>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>>>>>>> the
>>>>>>>>>>> mount fail faster. But I may be misunderstanding.
>>>>>>>>>>>
>>>>>>>>>>> --b.
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-nfs" in
>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>> --
>>>>>>>>> Chuck Lever
>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
