Thank you Amos for the review.
Following your advice... trying to use the pure transparent proxy (Tproxy) but getting different behaviours with google domains in google chrome and after adding peek there aren't still no names in https addresses.
First create ssl certs directory, just to check later.
/usr/lib64/squid/ssl_crtd -c -s /var/log/squid/lib/ssl_db
remove from squid.conf:
#tcp_outgoing_address
#always_direct allow ssl-bump_port
#ssl_bump none all
add to squid.conf:
ssl_bump splice localnet
ssl_bump peek all
ssl_bump splice all
I removed IPv6 restriction in kernel and in new squid compile without --disable-ipv6 option. And I added some similiar IPv6 rules to Tproxy boot script.
I continue to not generate-host-certificates(=off), I can check it in ssl_db directory. But, just for testing, if I change generate-host-certificates to on then there are certificates changes with my self CA Autority notice in Google Chrome related to google sites, some other sites tested get the certificate in right way. In IE and Firefox there are no certificates issues.
A workaround for this issue in Google Chrome with websites in google domains was create a specific acl for IP from google domains and splice first like did with localnet:
acl google dst "/etc/squid/google.txt"
...
ssl_bump splice google
...
After those changes and setup ssl-bump with peek for most of sites (e.g. facebook) but still no names in logs, just the IP in https navigation.
access.log:
1439907457.958 49 192.168.0.102 TCP_MISS/200 910 POST http://ocsp.digicert.com/ - ORIGINAL_DST/93.184.220.29 application/ocsp-response
1439907466.798 59053 192.168.0.102 TCP_TUNNEL/200 3944 CONNECT 212.113.184.216:443 - ORIGINAL_DST/212.113.184.216 -
1439907472.813 58798 192.168.0.102 TCP_TUNNEL/200 8320 CONNECT 212.113.185.35:443 - ORIGINAL_DST/212.113.185.35 -
1439907472.817 59014 192.168.0.102 TCP_TUNNEL/200 5234 CONNECT 212.113.184.221:443 - ORIGINAL_DST/212.113.184.221 -
1439907490.935 65674 192.168.0.102 TCP_TUNNEL/200 4207 CONNECT 54.171.32.174:443 - ORIGINAL_DST/54.171.32.174 -
1439907527.998 115712 192.168.0.102 TCP_TUNNEL/200 5485 CONNECT 54.192.61.197:443 - ORIGINAL_DST/54.192.61.197 -
1439907545.804 122741 192.168.0.102 TCP_TUNNEL/200 5817 CONNECT 132.245.50.66:443 - ORIGINAL_DST/132.245.50.66 -
cache.log:
---------
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=499906
Content-Type: application/ocsp-response
Date: Tue, 18 Aug 2015 14:17:31 GMT
ETag: "55d2d1da-1d7"
Expires: Tue, 25 Aug 2015 02:17:31 GMT
Last-Modified: Tue, 18 Aug 2015 06:34:02 GMT
Server: ECS (mad/42F2)
X-Cache: HIT
Content-Length: 471
X-Cache: MISS from squidhead2.skywalker.local
Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)
Connection: keep-alive
----------
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15
2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:38.153 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.6:443 remote=192.168.0.102 FD 40 flags=17
2015/08/18 15:17:38.153 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=31.13.90.6:443 remote=192.168.0.102 FD 40 flags=17
2015/08/18 15:17:38.153 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT 31.13.90.6:443 HTTP/1.1
Host: 31.13.90.6:443
----------
2015/08/18 15:17:38.153 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:38.153 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:38.153 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:38.153 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '31.13.90.6:443'
2015/08/18 15:17:38.153 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED
2015/08/18 15:17:38.153 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED
2015/08/18 15:17:38.153 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.6:443 flags=25
2015/08/18 15:17:38.153 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0
2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15
2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:38.156 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.6:443 remote=192.168.0.102 FD 42 flags=17
2015/08/18 15:17:38.156 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=31.13.90.6:443 remote=192.168.0.102 FD 42 flags=17
2015/08/18 15:17:38.156 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT 31.13.90.6:443 HTTP/1.1
Host: 31.13.90.6:443
----------
2015/08/18 15:17:38.157 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:38.157 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:38.157 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:38.157 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '31.13.90.6:443'
2015/08/18 15:17:38.157 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED
2015/08/18 15:17:38.157 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED
2015/08/18 15:17:38.157 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.6:443 flags=25
2015/08/18 15:17:38.157 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0
2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15
2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:45.351 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=212.113.185.24:443 remote=192.168.0.102 FD 46 flags=17
2015/08/18 15:17:45.351 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=212.113.185.24:443 remote=192.168.0.102 FD 46 flags=17
2015/08/18 15:17:45.351 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT 212.113.185.24:443 HTTP/1.1
Host: 212.113.185.24:443
----------
2015/08/18 15:17:45.351 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 212.113.185.24:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:45.351 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:45.351 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT 212.113.185.24:443 is ALLOWED; last ACL checked: ssl-bump_port
2015/08/18 15:17:45.351 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '212.113.185.24:443'
2015/08/18 15:17:45.351 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED
2015/08/18 15:17:45.351 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED
2015/08/18 15:17:45.351 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=212.113.185.24:443 flags=25
2015/08/18 15:17:45.351 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0
Can I ask what I missed?
Thank you for your time.
On Tue, Aug 18, 2015 at 6:30 AM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
On 18/08/2015 12:25 a.m., Pedro Correia Sardinha wrote:
> Hello,
>
> I'm trying to setup a Squid server simple as possible just to review the
> web use in office using the last stable version 3.5.7.
>
And you chose TPROXY with ssl-bump'ing. The two most complex features to
setup. lol.
> I setup the bridge with 2 NIC, br0 with IP 192.168.0.5 and I had disable
> IPv6 on boot in my Slackware Current (Fri Aug 14 2015) server.
Sigh. Ever heard of IPv6-over-IPv4, 6-in-4, 6to4, etc. ?
All protocols designed to "fix" connectivity going through machines
setup like yours.
And why bother disabling a (BCP 177) mandatory part of the kernel?
The correct way to handle unwanted traffic is to firewall it. Not to
play around with kernel internals.
<snip>
> My squid.conf:
> tcp_outgoing_address 85.138.204.43
This is irrelevant with TPROXY. The client IP address is used instead.
For the regular forward-proxy traffic on port 3128 the machines default
IP will be used.
> dns_v4_first on
> pinger_enable off
> http_port 3128
> http_port 3129 tproxy
> https_port 3130 ssl-bump tproxy generate-host-certificates=off
> cert=/etc/squid/ssl/squid.pem cafile=/etc/squid/ssl/squid.pem
> always_direct allow ssl-bump_port
> ssl_bump none all
You have configued Squid not to even look at the TLS details.
> dns_nameservers 8.8.8.8 8.8.4.4
> access_log daemon:/var/log/squid/access.log squid
> cache deny all
> pid_filename /var/run/squid/squid.pid
> coredump_dir /var/log/squid/cache/squid
> visible_hostname myservername.domain.local
>
> In general the configuration (squid.conf) it's working but has some
> incomplete behaviors as shows in log files.
>
> access.log (I know this is Facebook but there are no dns resolusion in
> https, just IP):
> 1439811492.625 2377 192.168.0.102 TCP_TUNNEL/200 3574 CONNECT
> 31.13.90.2:443 - ORIGINAL_DST/31.13.90.2 -
What sort of resolution were you expecting?
* The above log line is recording the TCP connection. TCP packets do not
have any "domain name" fields that need resolving to IP addresses.
* you also configured Squid not to look at the TLS details where it
might have found an SNI entry with server domain name.
The result is that Squid is working purely with TPROXY IP addresses and
setting up a TCP tunnel to relay the traffic through.
>
> cache.log:
> HTTP/1.1 200 OK
> Accept-Ranges: bytes
> Cache-Control: max-age=504747
> Content-Type: application/ocsp-response
> Date: Mon, 17 Aug 2015 11:38:03 GMT
> ETag: "55d15943-1d7"
> Expires: Sun, 23 Aug 2015 23:38:03 GMT
> Last-Modified: Mon, 17 Aug 2015 03:47:15 GMT
> Server: ECS (mad/439C)
> X-Cache: HIT
> Content-Length: 471
> X-Cache: MISS from squidhead2.skywalker.local
> Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)
> Connection: keep-alive
> ----------
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> StoreEntry::checkCachable: NO: not cachable
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> StoreEntry::checkCachable: NO: not cachable
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> StoreEntry::checkCachable: NO: not cachable
Above is all the end of some transaction that was started earlier. No
useful details in the provided log snippet about it.
This is where a transaction actually starts:
> 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(222) doAccept: New connection
> on FD 12
> 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(297) acceptNext: connection on
> local=0.0.0.0:3130 remote=[::] FD 12 flags=25
> 2015/08/17 12:38:10.248 kid1| client_side.cc(3890)
Notice how the connection local IP:port details change from port 3130 to
port 443. Thats TPROXY working.
> httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.2:443
> remote=192.168.0.102 FD 50 flags=17
This is the "ssl_bump none" action working (by not doing anything TLS
related) exactly as you configured.
Squid is now processing an internally generated CONNECT request
representing the intercepted TPROXY connection in a way that can be
logged and/or relayed to other proxies if it needs to.
> 2015/08/17 12:38:10.248 kid1| client_side.cc(2337) parseHttpRequest: HTTP
> Client local=31.13.90.2:443 remote=192.168.0.102 FD 50 flags=17
> 2015/08/17 12:38:10.248 kid1| client_side.cc(2338) parseHttpRequest: HTTP
> Client REQUEST:
> ---------
> CONNECT 31.13.90.2:443 HTTP/1.1
> Host: 31.13.90.2:443
> ---------
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)
> clientAccessCheckDone: The request CONNECT 31.13.90.2:443 is ALLOWED; last
> ACL checked: localnet
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(717)
> clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)
> clientAccessCheckDone: The request CONNECT 31.13.90.2:443 is ALLOWED; last
> ACL checked: localnet
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(280) peerSelectDnsPaths: Found
> sources for '31.13.90.2:443'
The CONNECT request uses a raw-IP address provided by TPROXY. There is
no name to resolve.
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(281) peerSelectDnsPaths:
> always_direct = ALLOWED
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(282) peerSelectDnsPaths:
> never_direct = DENIED
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(288) peerSelectDnsPaths:
> ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.2:443 flags=25
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(295) peerSelectDnsPaths:
> timedout = 0
... stuff happens for 2 seconds...
> 2015/08/17 12:38:12.621 kid1| client_side.cc(815) swanSong: local=
> 31.13.90.2:443 remote=192.168.0.102 flags=17
> 2015/08/17 12:38:12.625 kid1| client_side.cc(815) swanSong: local=
> 31.13.90.2:443 remote=192.168.0.102 flags=17
Then the connection closes.
Looks perfectly normal and expected behaviour to me considering what you
configured.
>
>
> The logs with http (port 80) has the name resolution of navigation.
>
> I disabled pinger because give some error:
<snip>
> 2015/08/17 12:49:55| FATAL: pinger: Unable to open any ICMP sockets.
Okay. Not a big problem. The pinger helper needs its suid bit set, which
is not working on all system installations yet. Disabling it is fine.
> Do I have to setup local DNS server? the internal DNS of squid can't handle
> https in Tproxy?
> What's missing to have name resolution in https traffic as its showed in
> http traffic?
Whats missing is the SSL-bumping part. HTTPS works differently to HTTP.
The URL domain name and all the rest of the HTTP message is encrypted
*in full*. There is simply no client HTTP message involved if you don't
decrypt.
As I mentioned above what you are seeing in the log is a Squid-generated
CONNECT message. Its the HTTP representation of the intercepted TCP SYN
packet and contains purely raw-IP:port details.
TLS does have an SNI record which is sent by browsers un-encrypted that
can be used as domain for some things. BUT, that requires "ssl_bump
peek" action at minimum, and has no guarantee of actually being present.
SNI is also still a new feature, and is not used for these fake CONNECT
requests anyway (since they represent the TCP SYN). So you wont see any
log difference in 3.5 even if you do let your Squid use it for ACLs.
Amos
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
