Hi Alex,
Thank you once again!
See replies inline below.
On 28/12/22 22:57, Alex Rousskov wrote:
Hi Amish,
TLS options are used on _both_ sides, in various cases, but there
are still too many unknowns, and I cannot quickly answer all of your
questions at once. Sorry. I can only guide you one step at a time.
* If you are sure that SSL_CTX_set_options() is not called when
talking to the origin server in your setup, then there is no need to
hard-code those options. Instead, we need to figure out why Squid does
not call SSL_CTX_set_options(). The best next step is to configure
your Squid with the following ssl_bump rules and check whether
SSL_CTX_set_options() is called for the Squid-server connection in
that case. Sharing access.log records to double check that the origin
server was contacted may be a good idea.
ssl_bump stare all
ssl_bump bump all
I have concluded that updateContextOptions() is not being called for
squid to origin server as cache.log does not show any calls.
Here is how I concluded it.
$ cat /etc/squid/squid.conf
debug_options 83,6
ssl_bump stare all
ssl_bump bump all
tls_outgoing_options \
cafile=/etc/ssl/cert.pem \
cipher=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
\
options=0x40000
http_port 3128 intercept
http_port 8080 ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4
https_port 8081 intercept ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4
Note that I have kept different options. 0x40000 for
tcp_outgoing_options and 0x4 for http(s)_port.
$ systemctl reload squid
$ curl --no-progress-meter -kx 127.0.0.1:8080 'https://www.jio.com/?foo=bar'
$ tail -f /var/log/squid/cache.log |grep -i 'openssl\|parsed'
2022/12/29 06:13:04.182 kid1| 83,5| PeerOptions.cc(447) parseOptions:
INFO: TLS parsedOptions(1)=4
2022/12/29 06:13:04.182 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for context=0x559210db6b20,
parsedOptions=4
2022/12/29 06:13:04.182 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for context=0x559210db6b20,
getOptions=1179652
parsedOptions is 4 which squid picked up from http_port line.
$ tail -f /var/log/squid/access.log
1672274753.787 1290 127.0.0.1 NONE_NONE/200 0 CONNECT www.jio.com:443
local HIER_NONE/- -
1672274753.789 0 127.0.0.1 NONE_NONE/503 4343 GET
https://www.jio.com/?foo=bar - HIER_NONE/- text/html
SSL Bumping is working fine as squid is able to find full HTTPS URL
including CGI parameters.
But just to double check that SSL bumping is working fine, we try
accessing Google:
$ curl --no-progress-meter -kx 127.0.0.1:8080 'https://google.com/?foo=bar'
$ tail -f /var/log/squid/access.log
1672274850.924 1120 127.0.0.1 NONE_NONE/200 0 CONNECT google.com:443
local HIER_DIRECT/2404:6800:4009:81d::200e -
1672274851.000 75 127.0.0.1 TCP_MISS/301 1041 GET
https://google.com/?foo=bar local HIER_DIRECT/2404:6800:4009:81d::200e
text/html
Google.com is accessed successfully and access.log also shows the same
with full HTTPS URL.
Now I reversed the options. Setting the options as 0x4 for
tcp_outgoing_options and 0x40000 for http(s)_port.
$ cat /etc/squid/squid.conf
tls_outgoing_options \
...
options=0x4
...
http_port 8080 ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x40000
https_port 8081 intercept ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x40000
$ systemctl reload squid
$ curl --no-progress-meter -kx 127.0.0.1:8080 https://www.jio.com/?foo=bar'
$ tail -f /var/log/squid/cache.log |grep -i 'openssl\|parsed'
2022/12/29 06:29:30.106 kid1| 83,5| PeerOptions.cc(447) parseOptions:
INFO: TLS parsedOptions(1)=262144
2022/12/29 06:29:30.106 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for context=0x559211c674e0,
parsedOptions=262144
2022/12/29 06:29:30.106 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for context=0x559211c674e0,
getOptions=1441792
Note: 262144 is 0x40000 = SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION.
And 1441792 is 0x160000 (i.e. 0x100000 = SSL_OP_ENABLE_MIDDLEBOX_COMPAT
| 0x40000 | 0x20000 = SSL_OP_NO_COMPRESSION)
$ tail -f /var/log/squid/access.log
1672275570.422 325 127.0.0.1 NONE_NONE/200 0 CONNECT www.jio.com:443
local HIER_NONE/- -
1672275570.423 0 127.0.0.1 NONE_NONE/503 4343 GET
https://www.jio.com/?foo=bar - HIER_NONE/- text/html
So from above two squid.conf settings we can conclude that
updateContextOptions() and hence SSL_CTX_set_options() is getting called
only for client to squid side of the connection and not squid to origin
side of the connection.
Now we need to find, where the server side connection happens OR why
updateContextOptions() is not called for squid to origin server connections.
Any idea?
Thanks and regards
Amish
* If you are not sure whether SSL_CTX_set_options() is called when
talking to the origin server in your setup, then hard-coding those
options may still be the best next step. If it fixes the problem, then
we will be able to resolve a few variables without the test in the
first bullet.
HTH,
Alex.
On 12/28/22 11:49, Amish wrote:
Hi Alex,
On 28/12/22 21:31, Alex Rousskov wrote:
Hi Amish,
Squid parsing code is tricky. tls_outgoing_options parsing code
is triply so. Even its authors misinterpret it!
I assume you have removed multiple tls_outgoing_options directives
from your configuration before testing. If you have not, please
merge those directives into one and retest. You should still see
multiple parsing paths, in part, due to (unfortunate)
Security::PeerOptions implementation and, in part, due to Squid
parsing default options before Squid parses your actual
configuration files.
Yes I had combined all tls_outgoing_options in to a single directive
(but in multiple lines ending with a backslash \). It can be seen in
the grep command in my previous email.
But wait, I have found something.
And I have a doubt that parsedOptions work only for client to squid
side of the connection and does not work for squid to server side of
the connection.
What I did is changed my "http(s)_port" directives to include
options=0x4. These directives control options for the client to squid
side of TLS connection.
$ grep 'ssl-bump' /etc/squid/squid.conf
http_port 8080 ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4
https_port 8081 intercept ssl-bump generate-host-certificates=on
tls-cert=/etc/squid/ssl_cert/squid.pem
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4
Now lets reproduce the issue:
$ curl --no-progress-meter -kx 127.0.0.1:8080 https://www.jio.com
|grep 'TLS\|SSL'
<pre>[No Error] (TLS code:
SQUID_TLS_ERR_CONNECT+TLS_LIB_ERR=A000152+TLS_IO_ERR=1)</pre>
<p>Failed to establish a secure connection: error:0A000152:SSL
routines::unsafe legacy renegotiation disabled</p>
Lets see what cache.log has to say:
$ tail -f /var/log/squid/cache.log |grep -i 'openssl\|parsed'
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(447) parseOptions:
INFO: TLS parsedOptions(1)=4
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for context=0x559075043e30,
parsedOptions=4
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for context=0x559075043e30,
getOptions=1179652
Bingo!! As we can see, parsedOptions is now set to 4!
And it is also confirmed by ssl_get_options() - 1179652 = 0x120004
(4 means the option is accepted by OpenSSL too)
But this also means that updateContextOptions() is called ONLY FOR
client to squid side (curl to squid) and it is not called for squid
to server (jio.com in this case) side.
The reason being that we do not see updateContextOptions() being
called twice for a request. But only once. And that is why request
still fails with the negotiation error.
So where exactly is the call for squid to server side being made?
If merging directives does not deliver custom options to
SSL_CTX_set_options(), then let's attack this from the other end:
Supply the right options to each SSL_CTX_set_options() call:
const Security::ParsedOptions forcedParsedOptions = 0x4 | 0x40000;
SSL_CTX_set_options(ctx.get(), forcedParsedOptions);
Does the above temporary hack fix the problem in your test?
I will try this tomorrow (its late night here).
But I think I will have to set this somewhere else and NOT in
PeerOptions.cc. Because above code appears to be for client to squid
side.
And I need to add forcedParsedOptions to the code which connects
squid to server. But where?
Thank you,
Amish.
Thank you,
Alex.
On 12/28/22 02:32, Amish wrote:
Hi Alex,
Thanks again for your reply.
To find answers to your questions, I added few debugs() lines to
PeerOptions.cc.
The diff file (patch) is attached.
It prints parsedOptions and options retrieved from SSL context and
session objects at several stages.
Here is tls_outgoing_options setting:
$ grep tls_outgoing_options /etc/squid/squid.conf
tls_outgoing_options \
cafile=/etc/ssl/cert.pem \
cipher=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
\
options=0x4
Here is what squid logs on reload i.e. on parsing the squid.conf
$ systemctl reload squid
$ tail -f /var/log/squid/cache.log |grep -i 'openssl\|parsed'
2022/12/28 12:19:30.596 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:30.598 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:30.723 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=4
2022/12/28 12:19:30.729 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:30.729 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:32.147 kid1| 83,5| PeerOptions.cc(447)
parseOptions: INFO: TLS parsedOptions(1)=0
2022/12/28 12:19:32.147 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:33.524 kid1| 83,5| PeerOptions.cc(447)
parseOptions: INFO: TLS parsedOptions(1)=0
2022/12/28 12:19:33.532 kid1| 83,5| PeerOptions.cc(547)
parseOptions: INFO: TLS parsedOptions(3)=0
2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(447)
parseOptions: INFO: TLS parsedOptions(1)=0
2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for
context=0x562a5387fe30, parsedOptions=0
2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for
context=0x562a5387fe30, getOptions=1179648
2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(447)
parseOptions: INFO: TLS parsedOptions(1)=0
2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for
context=0x562a53e6e740, parsedOptions=0
2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for
context=0x562a53e6e740, getOptions=1179648
It seems that squid parses the options multiple times and only once
it gets the value as 4. Rest are parsed as 0.
The value of 1179648 (0x120000) corresponds to
SSL_OP_NO_COMPRESSION (0x20000) and SSL_OP_ENABLE_MIDDLEBOX_COMPAT.
(0x100000)
Now lets reproduce the issue:
$ curl --no-progress-meter -kx 127.0.0.1:8080 https://www.jio.com
|grep 'TLS\|SSL'
<pre>[No Error] (TLS code:
SQUID_TLS_ERR_CONNECT+TLS_LIB_ERR=A000152+TLS_IO_ERR=1)</pre>
<p>Failed to establish a secure connection: error:0A000152:SSL
routines::unsafe legacy renegotiation disabled</p>
So, as we can see, we are still not able to access the site.
Lets see what cache.log has to say.
$ tail -f /var/log/squid/cache.log |grep -i 'openssl\|parsed'
2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(447)
parseOptions: INFO: TLS parsedOptions(1)=0
2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(643)
updateContextOptions: set OpenSSL options for
context=0x562a53eb14e0, parsedOptions=0
2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(645)
updateContextOptions: get OpenSSL options for
context=0x562a53eb14e0, getOptions=1179648
Strangely parsedOptions was zero and not 4!
Now we can answer your questions as below.
On 27/12/22 21:52, Alex Rousskov wrote:
On 12/27/22 10:42, Amish wrote:
On 26/12/22 21:31, Alex Rousskov wrote:
tls_outgoing_options options=0x4,0x40000
With numeric hex values, I do not see the ERROR on stderr.
But it still does not seem to be working as expected. Squid still
does not open the page and gives same legacy negotiation error.
There are still many unknowns (from my point of view), including:
1. Does OpenSSL accept the above options? You ask that question
below.
Google search shows some projects using OpenSSL v3 where there is
mention to use above option when a similar error occurred to them.
But in our case, its clear that squid does not pass value 4 to SSL
context, hence we do not know yet if OpenSSL accepts above options.
2. Does Squid indeed stare at the server, as expected?
3. Does Squid apply the accepted options when staring at the server?
A comment for parseOptions() in PeerOptions.cc states this:
/**
* Pre-parse TLS options= parameter to be applied when the TLS
objects created.
* Options must not used in the case of peek or stare bump mode.
*/
void Security::PeerOptions::parseOptions()
So it appears that TLS options is NOT used for peek as well as
stare. But why? I am not sure.
How do I make squid use it for 'stare' atleast?
4. Why does TLS negotiation fail despite those options applied,
especially since it succeeds with using openssl s_client
It possibly fails because options are not applied by squid.
So, where do I check next on why parsedOptions is still set to 0
and why its not used for 'stare'?
Thanks and regards,
Amish.
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
