Re: Failure to reconnect after cluster failvoer

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2/22/19 11:25 PM, Tom Talpey wrote:
-----Original Message-----
From: Ross Lagerwall <ross.lagerwall@xxxxxxxxxx>
Sent: Friday, February 22, 2019 9:17 AM
To: Tom Talpey <ttalpey@xxxxxxxxxxxxx>; Steve French
<smfrench@xxxxxxxxx>
Cc: CIFS <linux-cifs@xxxxxxxxxxxxxxx>
Subject: Re: Failure to reconnect after cluster failvoer

On 2/21/19 5:59 PM, Tom Talpey wrote:
The reconnect is apparently using a dotted-quad as the servername, and you
can see the auth is forced to NTLM as a consequence. Is that the way you
initially mounted the share (i.e. mount 10.71.217.50:/smbshare /mnt)?

-----Original Message-----
From: linux-cifs-owner@xxxxxxxxxxxxxxx <linux-cifs-owner@xxxxxxxxxxxxxxx>
On Behalf Of Steve French
Sent: Thursday, February 21, 2019 9:07 AM
To: Ross Lagerwall <ross.lagerwall@xxxxxxxxxx>
Cc: CIFS <linux-cifs@xxxxxxxxxxxxxxx>
Subject: Re: Failure to reconnect after cluster failvoer

Couple quick thoughts.

Does this work on current kernels (5.0 for example).

Was thinking about patches that might affect this like:
- "cifs: connect to servername instead of IP for IPC$ share"
- "smb3: on reconnect set PreviousSessionId field"
- Paulo's patches (has cifs-utils coreq) to reconnect to new IP
address if hostname's IP address changed and his add support for
failover
- Paulo's patch to remove trailing slashes from server UNC name

I've reproduced this with 5.0-rc7 and the latest cifs-utils from git.
The share was mounted as follows (yes, by IP):

mount.cifs -o
vers=3.0,cache=loose,actimeo=0,username=x,domain=y,password=z
'//10.71.217.31/smbshare' /mnt

Here is the tcpdump when it fails to reconnect properly:
...

The initial connection is at timestamp 0s, reconnection at 13s,
STATUS_NETWORK_NAME_DELETED at 60s.

For comparison, here is a tcpdump using the "fix" from my previous mail:
...

The initial connection is at timestamp 0s, reconnection at 34s,
successful read request at 215s.

Note that the tree connect for IPC$ only happens _after_ the tree
connect for the share succeeds.

Thanks for the full traces, they clarify the situation. But, I don’t see any
meaningful difference in the client behavior. The ordering of the two
treeconnects is the same between the two - initially, "IPC$" then
"smbshare", and on reconnect, the other way around. So, I'm unclear
whether your patch did anything.

There is definitely a difference. Before the patch, on reconnect the client:

* Connects to "smbshare" which fails
* Then connects to "IPC$" which succeeds
* Then tries again to connect to smbshare which fails repeatedly

After the patch, on reconnect the client:

* Connects to "smbshare" which fails
* Then tries again to connect to "smbshare" which succeeds after several retries
* Then tries to connect to "IPC$" which succeeds

This subtle reordering somehow makes it work. It may indeed be a server bug rather than a client bug. I was hoping someone could shed some light on this.


The STATUS_NETWORK_NAME_DELETED is a consequence of the failed
re-establishment of the tree connect, and is not itself the problem. The
server is simply timing out the treeid, since the client did not successfully
reclaim it. The repeated STATUS_BAD_NETWORK_NAME is the issue.

Are you sure the clustered server is recovering properly when you are
forcing the failover? For example, if it's a two-node cluster, maybe node A
can take over node B, but node B has issues taking over node A. Is there
anything relevant in the server logs?


It's a two node cluster. The behaviour happens reliably when failing over either way. After failover, the server state is consistent. E.g. after a failover from node A to node B, node B shows itself as the primary server and the node A is marked as down. I couldn't find anything interesting in the server logs.

Thanks,
--
Ross Lagerwall



[Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux