Re: nfs client performance while server is down

Chuck Lever <chuck.lever@xxxxxxxxxx> · Mon, 25 Jan 2010 16:26:09 -0500

On Jan 25, 2010, at 4:18 PM, Whoop Whouzer wrote:
Any idea how I could do that?

The strace(1) man page says "-f" follows children.  In any event, you  
can strace the running children processes using "strace -p <pid>".

On Mon, Jan 25, 2010 at 10:01 PM, Chuck Lever  
<chuck.lever@xxxxxxxxxx> wrote:
On Jan 25, 2010, at 2:38 PM, Whoop Whouzer wrote:

Running  "strace nautilus" gives me allot of output. When I run it
while the server is down it completes the trace without a hiccup, it
returns and than nautilus is launched and hangs.
There are differences between the traces (with server up and server
down). I can't really see where the problem lies in there.

I would expect that the command-line nautilus forks when it starts  
up.  If
it has some option you can specify to prevent that, it might allow  
a deeper
look.  You would need to tell strace to look at the children, too.

On Mon, Jan 25, 2010 at 8:08 PM, Chuck Lever  
<chuck.lever@xxxxxxxxxx>
wrote:

On Jan 25, 2010, at 2:02 PM, Whoop Whouzer wrote:

Ok, I did that, after shutting down the server and enabling debug
trace I tried to open the home folder of the current account  
(totally
unrelated to the nfsshare), it wouldn't open at all, I got no  
nautilus
at all. During the time my cursor was in busy mode I got the  
following
messages in kern.log (for ubuntu 10.04 client):
Jan 25 19:30:13 whoop-desktop kernel: [  160.719262] NFS call   
fsstat
Jan 25 19:30:37 whoop-desktop kernel: [  184.458611] NFS:
permission(0:16/74386), mask=0x10, res=0
Jan 25 19:30:37 whoop-desktop kernel: [  184.458647] NFS call   
access
Jan 25 19:30:43 whoop-desktop kernel: [  190.721086] nfs: server
192.168.1.130 not responding, timed out
Jan 25 19:30:43 whoop-desktop kernel: [  190.721113] NFS reply  
statfs:
-5
Jan 25 19:30:43 whoop-desktop kernel: [  190.721116] nfs_statfs:
statfs error = 5
These series of traces are repeating over and over again at a set
interval (there is no flooding of the logs), even if I do nothing.
It's even worse than I thought because when I tried to shutdown,  
the
machine wouldn't shutdown because it claimed
the "File manager" was still running (although it was not  
visible on
screen); so I had to kill that before I could shutdown (properly).

In Fedora 12 I had a similar user experience (nautilus did show up
without showing any contents and it was hanging). I had enabled
tracing and it seems to be logged to /var/log/messages. I got this
output in fedora:
Jan 25 20:48:38 localhost kernel: NFS reply statfs: -5
Jan 25 20:48:38 localhost kernel: nfs_statfs: statfs error = 5
Jan 25 20:48:38 localhost kernel: NFS call  fsstat
Jan 25 20:49:14 localhost kernel: nfs: server 192.168.1.130 not
responding, timed out
Jan 25 20:49:14 localhost kernel: NFS reply getattr: -5
Jan 25 20:49:14 localhost kernel: nfs_revalidate_inode:  
(0:14/74386)
getattr failed, error=-5
Jan 25 20:49:25 localhost kernel: NFS: revalidating (0:14/74386)
Jan 25 20:49:25 localhost kernel: NFS call  getattr
Jan 25 20:50:14 localhost kernel: nfs: server 192.168.1.130 not
responding, timed out
Jan 25 20:50:14 localhost kernel: NFS reply access: -5
Jan 25 20:50:14 localhost kernel: NFS: permission(0:14/74386),  
mask=0x1,
res=-5
Jan 25 20:50:14 localhost kernel: NFS call  access
Jan 25 20:51:14 localhost kernel: nfs: server 192.168.1.130 not
responding, timed out
Jan 25 20:51:14 localhost kernel: NFS reply statfs: -5
Jan 25 20:51:14 localhost kernel: nfs_statfs: statfs error = 5
Jan 25 20:51:14 localhost kernel: NFS call  fsstat
Most of the trace is repeating in set intervals as well, there  
is no
flooding of the logs...
Fedora would not shutdown normally either

This verifies that your client is attempting to access the NFS  
server,
but
doesn't tell us which file it's attempting to access.   
Essentially the
EIO
means "failed to connect".

Maybe try an strace of the nautilus process next?

On Mon, Jan 25, 2010 at 5:48 PM, Chuck Lever <chuck.lever@xxxxxxxxxx 
>
wrote:

On Jan 24, 2010, at 7:09 PM, Whoop Whouzer wrote:

I did some network traces and there is nothing strange  
happening as
far as I can tell. I shut down the server (some network traffic
occurred as is to be expected). It got quiet again, I launched
nautilus, it got stuck without displaying anything and there  
was no
real network activity except 3 broadcasts using the ARP protocol
asking where the server was (could be just coincidence).

That sounds like the client does want to reconnect with the  
server.

You could try enabling debug tracing on your client (sudo  
rpcdebug -m
nfs
-s
all) after shutting down your server, then try to start  
nautilus.  The
kernel log would then contain NFS-related messages that might  
indicate
where
to look next.

Closing
nautilus and launching it again will let it hang again but I  
see no
additional network traffic. After a while nautilus will  
display the
contents of the folder without any network traffic.

On Sun, Jan 24, 2010 at 10:34 PM, Muntz, Daniel <Dan.Muntz@xxxxxxxxxx 
>
wrote:

Perhaps something in your $PATH is in the NFS mount?  Do a  
network
trace
and maybe you can see if, in fact, there are actually NFS  
operations
being
attempted that you weren't expecting.  Then try to figure out  
why.

 -Dan

-----Original Message-----
From: Whoop Whouzer [mailto:tiredandnumb@xxxxxxxxx]
Sent: Saturday, January 23, 2010 8:28 AM
To: Peter Chacko
Cc: linux-nfs@xxxxxxxxxxxxxxx
Subject: Re: nfs client performance while server is down

I don't remember all the different set-ups I tried it on,  
but I just
confirmed this with the following combinations:

ubuntu server 10.04 (alpha 2) --> ubuntu desktop 9.10, ubuntu
desktop
10.04 (alpha 2), fedora 12
ubuntu server 9.10 --> ubuntu desktop 9.10, ubuntu desktop  
10.04
(alpha 2), fedora 12

I'll be happy to test it on another client machine (distro)  
even
another server (although it would require a little more time)

Here are some examples on the bugreports I noticed and how  
they do
not
seem to get solved:
https://bugzilla.redhat.com/show_bug.cgi?id=175283
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/ 
164120

regards,
Whoop

On Sat, Jan 23, 2010 at 4:57 PM, Peter Chacko
<peterchacko35@xxxxxxxxx> wrote:

Which client OS you observed this behavior ?  This has  
nothing to
do
NFS design, and its purely stateless...Its upto the client OS
implementation about aspects like how to deal with local

IO, when NFS

share gets  disconnected..

May be a VFS bug on the local OS you found this problem ..

thanks

On Sat, Jan 23, 2010 at 9:15 PM, Whoop Whouzer

<tiredandnumb@xxxxxxxxx> wrote:

Howdy,

I was wondering why nfs is designed in such a way that the

performance

of an nfs client machine gets very bad when the nfs server

is offline?

This is even the case with a soft mount (either via mount

or fstab).

Just about every application that requires disk access (not
talking
about nfs share acces) gets really slow to unresponsive.

For instance

nautilus becomes unresponsive when displaying the contents  
of any
folder on the local disk,
playing movie files (stored on local disk) let totem or

vlc get stuck

on set intervals, even the terminal becomes unresponsive  
at times.

I could understand that these problems would occur while

accessing the

nfs share directoiourry while the server is offline, but

why for totally

unrelated directories?

I have experienced this behaviour on various distro's, and

also found

various bug reports on this issue, they don't seem to get  
solved
as
this is viewed as nfs design.
I see this as a flaw because clients are totally dependent  
on the
server. This would be less of a deal if the entire home  
directory
would be stored on nfs (although I even think some sort of
synchronisation technology could and should be implemented  
in this
case). It is a bit odd that (technically) one machine  
serving some
"useless" files to a non-trivial directory on client

machines can take

down these client machines.

For me the preferred functionality would be:
*If an nfs server gets offline the client's nfs share  
becomes
unaccessible, but local directories and applications (that  
only
require local disk access) stay responsive.
*If an nfs server gets online (after being offline while the
client
has not been restarted) the nfs share becomes reconnected.

regards,
Whoop
--
To unsubscribe from this list: send the line "unsubscribe

linux-nfs" in

the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe
linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe  
linux-nfs"
in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

<stracesdiff.log>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html