Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

[John Sweet-Escott <eype69@xxxxxxxxx>]

Hi All

We are trying to run Squid 4.8, compiled with OpenSSL 1.1.1 (see [1]) on Ubuntu 18.04 as a transparent proxy for the purpose of egress filtering of HTTPS traffic using SNI (see config in [2]). It it works correctly when contacting some addresses (e.g. https://www.ubuntu.com) but not others (e.g. https://www.google.com). When we contact https://www.google.com using TLS1.2 we get the error in the logs:

2019/09/15 10:33:09 kid1| ERROR: negotiating TLS on FD 19: error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)

and the page returned to the client contains ERR_SECURE_CONNECT_FAIL. When TLS1.3 is used, the connections are made correctly, however my application is constrained to java8 & tomcat8 which does not support TLS1.3.

Connections are made using curl or openssl s_client. For example:

openssl s_client -tls1_2  -CAfile squid.crt -connect www.google.com:443 -tlsextdebug
[237/1854]CONNECTED(00000005)
TLS server extension "renegotiation info" (id=65281), len=1
0000 - 00                                                .
TLS server extension "EC point formats" (id=11), len=4
0000 - 03 00 01 02                                       ....
TLS server extension "session ticket" (id=35), len=0
TLS server extension "extended master secret" (id=23), len=0
depth=1 C = GB, ST = London, L = squid, O = squid, CN = squid
verify return:1
depth=0 CN = www.google.com
verify return:1
---
Certificate chain
 0 s:CN = www.google.com
   i:C = GB, ST = London, L = squid, O = squid, CN = squid
 1 s:C = GB, ST = London, L = squid, O = squid, CN = squid
   i:C = GB, ST = London, L = squid, O = squid, CN = squid
 2 s:C = GB, ST = London, L = squid, O = squid, CN = squid
   i:C = GB, ST = London, L = squid, O = squid, CN = squid

etc

Attached are pcap files showing first a bad connection to google and then a working connection to ubuntu. Looking at the pcap files the difference in the google and ubuntu server hello lies in the extensions and the cypher:

Google:

Handshake Protocol: Server Hello
    Handshake Type: Server Hello (2)
    Length: 59
    Version: TLS 1.2 (0x0303)
    Random: 5d7e05552e1fdea260f67e0bdf413f6a9837fbaffdebeb35…
    Session ID Length: 0
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
    Compression Method: null (0)
    Extensions Length: 19
    Extension: extended_master_secret (len=0)
    Extension: renegotiation_info (len=1)
    Extension: ec_point_formats (len=2)

    Extension: session_ticket (len=0)

Ubuntu:

Handshake Protocol: Server Hello
    Handshake Type: Server Hello (2)
    Length: 61
    Version: TLS 1.2 (0x0303)
    Random: 7ec2c3a2554bac610e0290ac1f160c3ed185bdd1159e377c…
    Session ID Length: 0
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
    Compression Method: null (0)
    Extensions Length: 21
    Extension: server_name (len=0)
    Extension: renegotiation_info (len=1)
    Extension: ec_point_formats (len=4)
    Extension: session_ticket (len=0)

Differences are that Google supplies extended_master_secretand Ubuntu supplies server_name extensions. The cyphers chosen by the server is also different. Enabling debug using squid -k debug indicates that this is the likely problematic area:

2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(712) checkForMissingCertificates: SSL server sent 2 certificates
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(26) AsyncCall: The AsyncCall Security::PeerConnector::negotiate constructed, this=0x560d937c94b0 [call515701]2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(93) ScheduleCall: PeerConnector.cc(391) will call Security::PeerConnector::negotiate() [call515701]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncJob.cc(154) callEnd: Ssl::PeekingPeerConnector status out: [ FD 19 job24663]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCallQueue.cc(57) fireNext: leaving Security::PeerConnector::negotiate()
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCallQueue.cc(55) fireNext: entering Security::PeerConnector::negotiate()
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(38) make: make call Security::PeerConnector::negotiate [call515701]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncJob.cc(123) callStart: Ssl::PeekingPeerConnector status in: [ FD 19 job24663]
2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(188) negotiate: SSL_connect session=0x560d93835950
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(356) giveBuffered: 5<=5 bytes to OpenSSL
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(356) giveBuffered: 63<=63 bytes to OpenSSL
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x1001 TWCH (SSLv3/TLS write client hello)
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(471) write: postpone writing 7 bytes to SSL FD 19
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x4008 SSLERR (error)
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x1002 SSLERR (error)
2019/09/15 11:21:02.486 kid1| 83,5| NegotiationHistory.cc(83) retrieveNegotiatedInfo: SSL connection info on FD 19 SSL version NONE/0.0 negotiated cipher
2019/09/15 11:21:02.486 kid1| ERROR: negotiating TLS on FD 19: error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)
2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(570) callBack: TLS setup ended for local=10.20.251.235:37882 remote=216.58.210.196:443 FD 19 flags=1

I am not sure how to resolve this issue. I am guessing that there is something in the response from Google that is causing OpenSSL to reject the connections. From what I can see, there is no evidence of a downgrade being attempted, which is what the inappropriate fallback error message might indicate.

Any advice/guidance greatfully recieved.

John

[1] OpenSSL and Squid versions

openssl version
OpenSSL 1.1.1  11 Sep 2018

squid -v
Squid Cache: Version 4.8
Service Name: squid
Ubuntu linux
This binary uses OpenSSL 1.1.1  11 Sep 2018. For legal restrictions on distribution see https://www.openssl.org/source/license.html
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--$
ysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silen$
-rules' 'BUILDCXXFLAGS=-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -$
format -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -latomic' 'BUILDCXX=x86_64-linux-gn$
-g++' '--with-build-environment=default' '--enable-build-info=Ubuntu linux' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '-$
mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' $
--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-ssl' '--enable-ssl-crtd' '--with-openssl' '--enable-follow-x-forwarded-for' '--enab$
e-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntl
m=fake,SMB_LM' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-security-
cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '
--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=
65536' '--with-large-files' '--with-default-user=proxy' '--with-gnutls' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=
-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -Wformat -Werror=format-s
ecurity -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -latomic' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXX=x86_64-linux-gnu-
g++' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -Wformat -

Werror=format-security'

[2] Configuration squid.conf

visible_hostname squid
max_filedesc 4096

http_port 3128
#Handling HTTP requests
http_port 3129 intercept
include /etc/squid/http_sites.conf
http_access allow allowed_http_sites

#Handling HTTPS requests
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept
acl SSL_port port 443http_access allow SSL_port
# Allows access to www.google.com and www.ubuntu.com
include /etc/squid/https_sites.conf

# TLS peek-and-splice configuration
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump peek step2 allowed_https_sites
ssl_bump splice step3 allowed_https_sitesssl_bump terminate step2 all
http_access deny all

Attachment: badgoodonly.pcap
Description: Binary data

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users