Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Carlos André wrote:
> Hi Ian,
> 
> Thanks for patch and sorry for delay (i'm expecting receive u reply on
> bug track, not here) :)
> 
> But, this patch doesnt worked to me like expected...  :(
> 
> 
> Firstly I've changed "#MOUNT_WAIT=-1" to "MOUNT_WAIT=10"
> and later changed "10" to "2" with same results...
> (always restarting service, of course :)
> 
> Then, tried remove "sec=krb5p", and later removed "nfs4" but i got
> same results again.
> 
> Or i'm doing something wrong?
> 
> 
> [root@KSTATION areas]# automount -V
> 
> Linux automount version 5.0.1-0.rc2.131.bz517349.1
> [...]
> 
> [root@KSTATION areas]# time ls -la testdown
> ls: testedown: No such file or directory
> 
> real    3m9.006s
> user    0m0.002s
> sys     0m0.000s

OK, that isn't behaving the way I expect, I'll have a look.

> 
> 
> LOGGING:
> -----------------------------------------
> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
> calling mount -t nfs4 -s -o rw,acl,sec=krb5p 1.2.3.4:/areas/testdown
> /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token = 91
> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/areas/testdown
> -----------------------------------------
> 
> 
> 
> 
> 
> 2009/8/17 Ian Kent <ikent@xxxxxxxxxx>:
>> On Thu, 2009-08-13 at 12:18 -0300, Carlos André wrote:
>>> Filled bug report:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=517349
>> Hi Carlos,
>>
>> I have a patched source rpm to add a mount wait parameter to autofs
>> located at:
>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>
>> Could you build it and see if it works.
>> I haven't tested it at all but it is fairly straight forward.
>> It is still unclear if this is the right way to do this and what the
>> consequences are in sending a term signal to mount. This mount request
>> will likely be followed by other requests for the same mount causing an
>> accumulation of mount(8) processes waiting for RPC timeouts before they
>> can answer the TERM signal.
>>
>> Anyway, for information the patch included in the source rpm above is:
>>
>> autofs-5.0.4 - add mount wait parameter
>>
>> From: Ian Kent <raven@xxxxxxxxxx>
>>
>> Often delays when trying to mount from a server that is not reponding
>> for some reason are undesirable. To try and prevent these delays we
>> provide a configuration setting to limit the time that we wait for
>> our spawned mount(8) process to complete before sending it a SIGTERM
>> signal. This patch adds a configuration parameter to allow us to
>> request we limit the time we wait for mount(8) to complete before
>> send it a TERM signal.
>> ---
>>
>>  daemon/spawn.c                 |    3 ++-
>>  include/defaults.h             |    2 ++
>>  lib/defaults.c                 |   13 +++++++++++++
>>  man/auto.master.5.in           |    7 +++++++
>>  redhat/autofs.sysconfig.in     |    9 +++++++++
>>  samples/autofs.conf.default.in |    9 +++++++++
>>  6 files changed, 42 insertions(+), 1 deletion(-)
>>
>>
>> --- autofs-5.0.1.orig/daemon/spawn.c
>> +++ autofs-5.0.1/daemon/spawn.c
>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>        unsigned int options;
>>        unsigned int retries = MTAB_LOCK_RETRIES;
>>        int update_mtab = 1, ret, printed = 0;
>> +       unsigned int wait = defaults_get_mount_wait();
>>        char buf[PATH_MAX];
>>
>>        /* If we use mount locking we can't validate the location */
>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>        va_end(arg);
>>
>>        while (retries--) {
>> -               ret = do_spawn(logopt, -1, options, prog, (const char **) argv);
>> +               ret = do_spawn(logopt, wait, options, prog, (const char **) argv);
>>                if (ret & MTAB_NOTUPDATED) {
>>                        struct timespec tm = {3, 0};
>>
>> --- autofs-5.0.1.orig/include/defaults.h
>> +++ autofs-5.0.1/include/defaults.h
>> @@ -24,6 +24,7 @@
>>
>>  #define DEFAULT_TIMEOUT                        600
>>  #define DEFAULT_NEGATIVE_TIMEOUT       60
>> +#define DEFAULT_MOUNT_WAIT             -1
>>  #define DEFAULT_UMOUNT_WAIT            12
>>  #define DEFAULT_BROWSE_MODE            1
>>  #define DEFAULT_LOGGING                        0
>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>  struct ldap_searchdn *defaults_get_searchdns(void);
>>  void defaults_free_searchdns(struct ldap_searchdn *);
>>  unsigned int defaults_get_append_options(void);
>> +unsigned int defaults_get_mount_wait(void);
>>  unsigned int defaults_get_umount_wait(void);
>>  const char *defaults_get_auth_conf_file(void);
>>  unsigned int defaults_get_map_hash_table_size(void);
>> --- autofs-5.0.1.orig/lib/defaults.c
>> +++ autofs-5.0.1/lib/defaults.c
>> @@ -45,6 +45,7 @@
>>  #define ENV_NAME_VALUE_ATTR            "VALUE_ATTRIBUTE"
>>
>>  #define ENV_APPEND_OPTIONS             "APPEND_OPTIONS"
>> +#define ENV_MOUNT_WAIT                 "MOUNT_WAIT"
>>  #define ENV_UMOUNT_WAIT                        "UMOUNT_WAIT"
>>  #define ENV_AUTH_CONF_FILE             "AUTH_CONF_FILE"
>>
>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>                    check_set_config_value(key, ENV_NAME_ENTRY_ATTR, value, to_syslog) ||
>>                    check_set_config_value(key, ENV_NAME_VALUE_ATTR, value, to_syslog) ||
>>                    check_set_config_value(key, ENV_APPEND_OPTIONS, value, to_syslog) ||
>> +                   check_set_config_value(key, ENV_MOUNT_WAIT, value, to_syslog) ||
>>                    check_set_config_value(key, ENV_UMOUNT_WAIT, value, to_syslog) ||
>>                    check_set_config_value(key, ENV_AUTH_CONF_FILE, value, to_syslog) ||
>>                    check_set_config_value(key, ENV_MAP_HASH_TABLE_SIZE, value, to_syslog))
>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>        return res;
>>  }
>>
>> +unsigned int defaults_get_mount_wait(void)
>> +{
>> +       long wait;
>> +
>> +       wait = get_env_number(ENV_MOUNT_WAIT);
>> +       if (wait < 0)
>> +               wait = DEFAULT_MOUNT_WAIT;
>> +
>> +       return (unsigned int) wait;
>> +}
>> +
>>  unsigned int defaults_get_umount_wait(void)
>>  {
>>        long wait;
>> --- autofs-5.0.1.orig/man/auto.master.5.in
>> +++ autofs-5.0.1/man/auto.master.5.in
>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>  60). If the equivalent command line option is given it will override this
>>  setting.
>>  .TP
>> +.B MOUNT_WAIT
>> +Set the default time to wait for a response from a spawned mount(8)
>> +before sending it a SIGTERM. Note that we still need to wait for the
>> +RPC layer to timeout before the sub-process exits so this isn't ideal
>> +but it is the best we can do. The default is to wait until mount(8)
>> +returns without intervention.
>> +.TP
>>  .B UMOUNT_WAIT
>>  Set the default time to wait for a response from a spawned umount(8)
>>  before sending it a SIGTERM. Note that we still need to wait for the
>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>> @@ -14,6 +14,15 @@ TIMEOUT=300
>>  #
>>  #NEGATIVE_TIMEOUT=60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=12
>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>> @@ -14,6 +14,15 @@ TIMEOUT=300
>>  #
>>  #NEGATIVE_TIMEOUT=60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=12
>>
>>
>>> Thanks!
>>>
>>> 2009/8/13 Carlos André <candrecn@xxxxxxxxx>:
>>>> 2009/8/13 Ian Kent <ikent@xxxxxxxxxx>:
>>>>> Carlos André wrote:
>>>>>> Today (2009-08-12) I'm using:
>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>> Thanks,
>>>>>
>>>>> My mistake, the wait time I was referring to is used for umounts during
>>>>> expires and is present in rev rc2.102.
>>>>>
>>>>> It shouldn't be hard to add this for mount as well.
>>>>> Would you like me to put something together?
>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>>>>
>>>>> Probably would be good to test something out to see if we can make a
>>>>> difference with the killing mount after some configured timeout but, if
>>>>> we make progress, probably the best way to deal with it is for you to
>>>>> log a bug against rhel-5 so I can get it committed to the rhel package.
>>>>> The possible issue is that I'm not sure if the RPC subsystem in the
>>>>> above rhel kernel will respond well to process death with potential
>>>>> outstanding requests. But we'll see.
>>>> Ok, on my way :)
>>>>
>>>> Thanks a lot!
>>>>
>>>>>>
>>>>>> Look my last test:
>>>>>> --------------------------------------------------------------
>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>> ls: testdown: No such file or directory
>>>>>>
>>>>>> real    3m9.025s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>> acl,sec=krb5p,proto=tcp,retry=0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>> acl,sec=krb5p,proto=tcp,retry=0 using module nfs4
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> root=/misc/areas name=testdown what=1.2.3.4:/areas/testdown,
>>>>>> fstype=nfs4, options=acl,sec=krb5p,proto=tcp,retry=0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> nfs options="acl,sec=krb5p,proto=tcp,retry=0", nosymlink=0, ro=0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> calling mount -t nfs4 -s -o acl,sec=krb5p,proto=tcp,retry=0
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
>>>>>> = 2 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
>>>>>> = 2 path /misc
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token = 17
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/testdown
>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>> 2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
>>>>>>> Carlos André wrote:
>>>>>>>> Hi Ian,
>>>>>>>> I'm getting crazy trying put "retry=" to work on mount... this option
>>>>>>>> just DONT WORK if use proto=tcp and/OR kerberos (sec=krb5/krb5i/krb5p)
>>>>>>>> like you can see on my previous emails...
>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>
>>>>>>> Maybe this is related to the same sort of problem we had with mount in
>>>>>>> the past, before the options parsing went into the kernel, where other
>>>>>>> services, like portmapper (or rpcbind), were being done with different
>>>>>>> timeout parameters before the RPC calls for mounting. That's just an
>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>
>>>>>>> But what version of autofs and kernel did you say you were using?
>>>>>>>
>>>>>>>> I appreciate any help.
>>>>>>>>
>>>>>>>> Carlos.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/12 Ian Kent <ikent@xxxxxxxxxx>:
>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>>>>>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>>>>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>>>>>>>> down for any reason, and will again hangs if user try access directory
>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>> "retry=0" means the mount command will fail as soon as the first
>>>>>>>>>> mount(2) system call fails.  When you set SYN retries to 1, this means
>>>>>>>>>> after 9 seconds, the connect fails, and that causes the mount(2) system
>>>>>>>>>> call to fail.
>>>>>>>>>>
>>>>>>>>>> Recent conversations with Ian suggested that a long timeout was desired
>>>>>>>>>> for automounter as well as other cases.  Ian, is there something else we
>>>>>>>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>>>>>>>> mount points handled via automounter?  How should mount.nfs wait so we
>>>>>>>>>> don't make other use cases worse?  (Looks like most of the history is
>>>>>>>>>> intact below).
>>>>>>>>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>>>>>>>>> mount.nfs in particular). This has always been a difficult situation for
>>>>>>>>> the automounter because interactive mount invocations should wait. But I
>>>>>>>>> believe automount mounts should always time out quickly, but that leads
>>>>>>>>> to its own set of problems, especially when home directories are concerned.
>>>>>>>>>
>>>>>>>>> I think adding "retry=0" is the right thing to do myself but I'm not
>>>>>>>>> certain that will work as we expect. I'll have to do some experimentation.
>>>>>>>>>
>>>>>>>>>> How long do you think is appropriate for the automounter to wait if the
>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>
>>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEFAULT]
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=tcp,retry=1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=tcp,retry=0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.003s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=tcp,retry=1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    1m3.002s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=tcp,retry=0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    0m9.003s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.001s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>>>>>>>> using retry=0 without kerberos I got only 9s...
>>>>>>>>>>>
>>>>>>>>>>> *sigh*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's socket connect
>>>>>>>>>>>> function,
>>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes longer than
>>>>>>>>>>>> the RPC
>>>>>>>>>>>> client's connect timeout, so it only makes one connect call, and then
>>>>>>>>>>>> fails.
>>>>>>>>>>>>
>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>>>>>>>>>> client
>>>>>>>>>>>> to retry the connect call until its connect timeout expires.  Each
>>>>>>>>>>>> connect
>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>>>>> sec=krb5p,proto=tcp
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>>>>> sec=krb5p,proto=tcp  ("retry=1" = no change)
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>>
>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/10 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>> I tried use "retry=1" option on mount without any change... I dont
>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@xxxxxxxxxx>:
>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation where my
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>>>>>>>>> exponential retries, or something like that).  For stock CentOS
>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@xxxxxxxxxxxx>:
>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@xxxxxxxxx>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos André <candrecn@xxxxxxxxx>:
>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>>>>>>>>>>>>> proto=udp)
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error
>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>>>>> I thought he was describing a situation where the server the server
>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux