Re: GlobalProtect connection loss

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2020-04-18 at 20:14, Daniel Lenski wrote:

> This is a useful log. Key interesting parts below.

<large snip>

> 5. This suggests a server misconfiguration. The server has sent new
> ESP keys and started using them, but then it gets confused and
> invalidates the ESP connection after the original interval elapses.
> That's annoying, and we've seen it before… but what OpenConnect
> should do to workaround it here is that it should give up on ESP and
> attempt to fallback to HTTPS. Which it appears to do:
> 
> [2020-04-14 00:20:43] Connecting to HTTPS tunnel endpoint ...
> [2020-04-14 00:20:43] SSL negotiation with foo.bar.baz
> [2020-04-14 00:20:43] Connected to HTTPS on foo.bar.baz with ciphersuite
> (TLS1.2)-(RSA)-(AES-256-GCM)
> [2020-04-14 00:20:43] > GET
> /ssl-tunnel-connect.sslvpn?authcookie=AUTHCOOKIE&user=foobar_user HTTP/1.1
> [2020-04-14 00:20:43] >
> 
> … but what's completely puzzling here is that we don't see either of
> the following messages prior to this attempted failover to HTTPS:
>     ESP detected dead peer
>     Failed to connect ESP tunnel; using HTTPS instead.

That's because of a quirk of the way I invoked OpenConnect and did the
redirection; those messages actually get printed to the terminal where
the program was invoked from, and don't make it into the file. This is
what I was referring to with

>> In hindsight I realize there was one thing I could have done
>> differently on this session invocation to be more sure I didn't
>> miss any stderr, etc., messages - but I didn't do it, so I'm not
>> entirely positive there's nothing missing. I hope all the important
>> stuff is there.

in the original attempt at sending this.

The relevant messages from the terminal from one such session, starting
from that point and continuing until exit, are:

>> [2020-04-15 14:25:56] ESP detected dead peer
>> [2020-04-15 14:26:01] Failed to connect ESP tunnel; using HTTPS instead.
>> [2020-04-15 14:26:01] Got inappropriate HTTP GET-tunnel response: HTTP/1.1 502 Bad Gateway
>> [2020-04-15 14:26:01] Invalid user name
>> [2020-04-15 14:26:01] Logout failed.
>> RTNETLINK answers: No such process
>> RTNETLINK answers: No such process
>> [2020-04-15 14:26:01] Unknown error; exiting.

but unfortunately I don't know how these interleave with the messages
which *did* get logged.

If that's critical information, I can adjust the way I do invocation and
log initiation for one session, although it will be a bit of a hassle.

> There is no codepath in OpenConnect v8.08 where you can go from a
> working GP ESP connection to HTTPS fallback *without* printing the
> second message (see
> https://gitlab.com/openconnect/openconnect/blob/v8.08/gpst.c#L1102-1116).
> IIRC correctly, I had all of this tricky ESP-to-HTTPS handoff "tap
> dance" nailed down prior to v8.0, but in case it hasn't changed
> recently.
> 
> 6. Finally, one more strange thing in your log. Something (????)
> causes it to attempt a logout immediately after trying to fallback to
> HTTPS, but the logout *fails* (POST
> https://foo.bar.baz/ssl-vpn/logout.esp -> "Invalid user name").

Yeah, this is the thing that seemed weirdest to me. I certainly did not
do anything which should have led this logout attempt to happen - as I
noted in my original mail, I was asleep at the time.

I just checked one of the other logs I've been grabbing of these
sessions in the meantime, and it does end with a similar logout interaction.

If I had the foundation for doing so, this is the part that I'd be
focusing on in trying to investigate this behavior.

> As I have in the past, I just tried to torture-test the rekey and
> ESP-HTTPS fallback by building v8.08 with an artificially-short
> hardcodedrekey interval (2 minutes), and then blocking UDP packets
> with iptables for long enough that OpenConnect thinks the ESP
> connection is dead, and falls back to ESP. As expected, I get both of
> the aforementioned messages ("ESP detected dead peer" and "Failed to
> connect ESP tunnel; using HTTPS instead.") prior to fallback from
> ESP to HTTPS, and then the HTTPS connection works fine. If I then
> unblock UDP, it will detect this on the next rekey and resume using
> ESP.
> 
> Basically… I can't reproduce your issue, and I'm having trouble
> understanding how it's even possible to get the sequence of log
> output thatyou show. You're 100% sure that this output comes from
> invoking OpenConnectv8.08 from the command line?

See above.

What I'm doing here is invoking a shell script (as ordinary user) from
the command line, which runs

su - -c "~/bin/vpn-bar 2>&1 >>
/home/ORIGINALUSER/tmp/vpn-bar-highverbosity.log"

and that command line in turn calls a shell script which sits under
/root/bin/, gets run as root, and contains the actual invocation of
openconnect - including
  echo password | openconnect --passwd-on-stdin [other-options]
and a custom routing script which adjusts the routing table so that only
traffic which *needs* the VPN goes over it (that latter being why I need
to run this as root, rather than pre-creating a tun* device node and
telling openconnect to use it).

(Yes, I know putting the password in the script and passing it in that
way is insecure; I'd be happy for a better way of doing it, but I'm not
currently prepared to accept one that requires me to manually enter the
password on every connection attempt, on top of entering the root
password - especially not when I'm connecting three times a workday.)

Apparently, something about redirecting stderr into stdout and doing
output-append from the outer invocation shell this way skips redirecting
some of the console output. I haven't managed to figure out why yet, but
it doesn't materially affect my use of the program.

Does that help clear up any of the confusion?

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
openconnect-devel mailing list
openconnect-devel@xxxxxxxxxxxxxxxxxxx
http://lists.infradead.org/mailman/listinfo/openconnect-devel

[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux