Carlos André wrote:
> Today (2009-08-12) I'm using:
> kernel-2.6.18-128.2.1.el5
> autofs-5.0.1-0.rc2.102.el5_3.1
Thanks,
My mistake, the wait time I was referring to is used for umounts during
expires and is present in rev rc2.102.
It shouldn't be hard to add this for mount as well.
Would you like me to put something together?
Probably would be good to test something out to see if we can make a
difference with the killing mount after some configured timeout but, if
we make progress, probably the best way to deal with it is for you to
log a bug against rhel-5 so I can get it committed to the rhel package.
The possible issue is that I'm not sure if the RPC subsystem in the
above rhel kernel will respond well to process death with potential
outstanding requests. But we'll see.
>
>
> Look my last test:
> --------------------------------------------------------------
> [root@KSTATION areas]# time ls testdown
> ls: testdown: No such file or directory
>
> real 3m9.025s
> user 0m0.000s
> sys 0m0.002s
>
>
>
>
> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
> mounting root /misc/areas, mountpoint testdown, what
> 1.2.3.4:/areas/testdown, fstype nfs4, options
> acl,sec=krb5p,proto=tcp,retry=0
> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
> acl,sec=krb5p,proto=tcp,retry=0 using module nfs4
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> root=/misc/areas name=testdown what=1.2.3.4:/areas/testdown,
> fstype=nfs4, options=acl,sec=krb5p,proto=tcp,retry=0
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> nfs options="acl,sec=krb5p,proto=tcp,retry=0", nosymlink=0, ro=0
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> calling mkdir_path /misc/areas/testdown
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> calling mount -t nfs4 -s -o acl,sec=krb5p,proto=tcp,retry=0
> 1.2.3.4:/areas/testdown /misc/areas/testdown
> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
> 3078093712 path /misc
> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
> submounts remaining in /misc
> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
> 3078093712 path /misc stat 3
> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
> exp 3078093712 finished, switching from 2 to 1
> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
> = 2 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
> 3078093712 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
> submounts remaining in /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
> 3078093712 path /misc stat 3
> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
> exp 3078093712 finished, switching from 2 to 1
> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
> = 2 path /misc
> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
> server '1.2.3.4' failed: timed out (giving up).
> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token = 17
> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/testdown
> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
> --------------------------------------------------------------
>
> 2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
>> Carlos André wrote:
>>> Hi Ian,
>>> I'm getting crazy trying put "retry=" to work on mount... this option
>>> just DONT WORK if use proto=tcp and/OR kerberos (sec=krb5/krb5i/krb5p)
>>> like you can see on my previous emails...
>> Right, my mistake for not looking closely enough at post.
>>
>> Maybe this is related to the same sort of problem we had with mount in
>> the past, before the options parsing went into the kernel, where other
>> services, like portmapper (or rpcbind), were being done with different
>> timeout parameters before the RPC calls for mounting. That's just an
>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>
>> But what version of autofs and kernel did you say you were using?
>>
>>> I appreciate any help.
>>>
>>> Carlos.
>>>
>>>
>>> 2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
>>>> Chuck Lever wrote:
>>>>> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>>>>>> This long timeout is good if workstation need mount a critical
>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>>> down for any reason, and will again hangs if user try access directory
>>>>>> pointing to a NFS down server...
>>>>> "retry=0" means the mount command will fail as soon as the first
>>>>> mount(2) system call fails. When you set SYN retries to 1, this means
>>>>> after 9 seconds, the connect fails, and that causes the mount(2) system
>>>>> call to fail.
>>>>>
>>>>> Recent conversations with Ian suggested that a long timeout was desired
>>>>> for automounter as well as other cases. Ian, is there something else we
>>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>>> mount points handled via automounter? How should mount.nfs wait so we
>>>>> don't make other use cases worse? (Looks like most of the history is
>>>>> intact below).
>>>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>>>> mount.nfs in particular). This has always been a difficult situation for
>>>> the automounter because interactive mount invocations should wait. But I
>>>> believe automount mounts should always time out quickly, but that leads
>>>> to its own set of problems, especially when home directories are concerned.
>>>>
>>>> I think adding "retry=0" is the right thing to do myself but I'm not
>>>> certain that will work as we expect. I'll have to do some experimentation.
>>>>
>>>>> How long do you think is appropriate for the automounter to wait if the
>>>>> server is down, in your case, Carlos?
>>>>>
>>>>>> Am losing something or there have was something weirdo...!?
>>>>>> ------------------------------------------------
>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries [DEFAULT]
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=tcp,retry=1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 3m9.000s
>>>>>> user 0m0.002s
>>>>>> sys 0m0.001s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 3m9.000s
>>>>>> user 0m0.000s
>>>>>> sys 0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=tcp,retry=0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 3m9.001s
>>>>>> user 0m0.000s
>>>>>> sys 0m0.003s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 3m9.001s
>>>>>> user 0m0.002s
>>>>>> sys 0m0.001s
>>>>>>
>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]
>>>>>>
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=tcp,retry=1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 1m3.002s
>>>>>> user 0m0.000s
>>>>>> sys 0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 2m6.000s
>>>>>> user 0m0.000s
>>>>>> sys 0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=tcp,retry=0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 0m9.003s
>>>>>> user 0m0.001s
>>>>>> sys 0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real 2m6.001s
>>>>>> user 0m0.001s
>>>>>> sys 0m0.002s
>>>>>> [root@KSTATION ~]#
>>>>>> ------------------------------------------------
>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>>> using retry=0 without kerberos I got only 9s...
>>>>>>
>>>>>> *sigh*
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>> Right. Normally the RPC client calls the kernel's socket connect
>>>>>>> function,
>>>>>>> which does 6 SYN retries. That one call usually takes longer than
>>>>>>> the RPC
>>>>>>> client's connect timeout, so it only makes one connect call, and then
>>>>>>> fails.
>>>>>>>
>>>>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>>>>> client
>>>>>>> to retry the connect call until its connect timeout expires. Each
>>>>>>> connect
>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>
>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>> sec=krb5p,proto=tcp
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>
>>>>>>>> real 3m9.000s
>>>>>>>> user 0m0.000s
>>>>>>>> sys 0m0.002s
>>>>>>>>
>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>> sec=krb5p,proto=tcp ("retry=1" = no change)
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>
>>>>>>>> real 2m6.004s
>>>>>>>> user 0m0.000s
>>>>>>>> sys 0m0.004s
>>>>>>>>
>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/10 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>
>>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>>>> 2049...
>>>>>>>>> I tried use "retry=1" option on mount without any change... I dont
>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>
>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>>>>> Bruce, no... you're right. I'm describing a situation where my
>>>>>>>>>>> server
>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>>>>>> and 9 seconds...
>>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>>> give up
>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>>>> exponential retries, or something like that). For stock CentOS
>>>>>>>>>> 5.3, I
>>>>>>>>>> think
>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>>> kernel
>>>>>>>>>> just
>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>>> components
>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>
>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@xxxxxxxxxxxx>:
>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@xxxxxxxxx>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/7/29 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>>>>>>>> proto=udp)
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real 3m9.001s) until show error
>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>> I thought he was describing a situation where the server the server
>>>>>>>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>>>>>>>> the
>>>>>>>>>>>> mount fail faster. But I may be misunderstanding.
>>>>>>>>>>>>
>>>>>>>>>>>> --b.
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>
>>>>>
>>>>>
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
