________________________________ From: Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> > 1. Your "works" and "does not work" setups currently differ in at least > three variables: user agent name, slash after the user agent name, and > acl negation in http_access. Find out which single variable is > responsible for the breakage by eliminating all other differences. > > 2. Post two ALL,2 cache.logs, each containing a single transaction, one > for the "works" case and one for the "does not work" case polished as > discussed in #1. I can't really do anything about #1 except maybe leave out the forward slash. That's because my 2 examples are trying to achieve the opposite. Let me just rephrase everything so it's crystal clear. My goal is to deny all client traffic from browsers that DO NOT have a specific user-agent string. So this is a negated statement. One of the things I can't do in Squid is define an ACL with a negated lookahead such as (?!useragentname). So I set up two examples. Common to both: acl allowed_useragent browser MyAllowedUAstring acl denied_useragent browser MyDeniedUAstring # example 1: http_access deny denied_useragent http_reply_access deny denied_useragent deny_info http://proxy-server1/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_useragent denied_useragent I then run this from my test client: # curl --insecure --user-agent MyAllowedUAstring https://www.gentoo.org -> works as expected (I see the web site). I guess you don't need to see cache.log here. Now I run this: # curl --insecure --user-agent MyDeniedUAstring https://www.gentoo.org -> works as expected (I'm denied access and I see Squid's error page). I guess there's no need for the full log here either. It boils down to this anyway: 2017/11/17 13:24:26.937 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyDeniedUAstring)' found in 'MyDeniedUAstring' 2017/11/17 13:24:26.937 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET https://www.gentoo.org/ is DENIED; last ACL checked: denied_useragent I'm done with example 1. That's because I cannot make a consistent list of all user agents I want to actively block. Instead, I want to "deny everyone except one or two". Also, since negative lookaheads are not supported in regular expressions, I change my example 1 to: # example 2: http_access deny !allowed_useragent http_reply_access deny !allowed_useragent deny_info http://proxy-server1/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_useragent allowed_useragent Then I run this from the client: # curl --insecure --user-agent MyAllowedUAstring https://www.gentoo.org -> I was expecting to be allowed access since Squid denies "everything that's not" MyAllowedUAstring. Well, at least I should have passed the "deny" line in example 2. However, I'm being blocked right there. This is the full log: 2017/11/17 13:30:42.216 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 88 2017/11/17 13:30:42.216 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3229 remote=[::] FD 88 flags=25 2017/11/17 13:30:42.216 kid1| 33,2| client_side.cc(3943) httpsSslBumpAccessCheckDone: sslBump needed for local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 method 4 2017/11/17 13:30:42.216 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:42.216 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- CONNECT 89.16.167.134:443 HTTP/1.1 Host: 89.16.167.134:443 ---------- 2017/11/17 13:30:42.216 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 89.16.167.134:443 is DENIED; last ACL checked: allowed_useragent 2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:42.226 kid1| 83,2| client_side.cc(3843) clientNegotiateSSL: clientNegotiateSSL: New session 0x125e030 on FD 8 (10.215.144.48:65262) 2017/11/17 13:30:42.226 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:42.226 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- GET / HTTP/1.1 Host: www.gentoo.org User-Agent: MyAllowedUAstring Accept: */* ---------- 2017/11/17 13:30:42.227 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring' 2017/11/17 13:30:42.227 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is ALLOWED, because it matched denied_mimetypes_rep 2017/11/17 13:30:42.227 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:42.227 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY: --------- HTTP/1.1 307 Temporary Redirect Server: squid Mime-Version: 1.0 Date: Fri, 17 Nov 2017 12:30:42 GMT Content-Type: text/html;charset=utf-8 Content-Length: 0 Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=89.16.167.134&i=10.215.144.48&M=CONNECT&o=&R=/&T=Fri,%2017%20Nov%202017%2012%3A30%3A42%20GMT&U=https%3A%2F%2F89.16.167.134%2F*&u=89.16.167.134%3A443&w=IT%40mydomain.org&x=&acl=denied_useragent X-Squid-Error: 403 Access Denied X-Cache: MISS from proxy-server1 X-Cache-Lookup: NONE from proxy-server1:3227 Connection: close Note that I have these defaults in my squid conf file: acl CONNECT method CONNECT http_access deny CONNECT !SSL_ports Let's try another one: # curl --insecure --user-agent MyDeniedUAstring https://www.gentoo.org -> This is as expected, I guess. Full log: 2017/11/17 13:30:10.365 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 88 2017/11/17 13:30:10.365 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3229 remote=[::] FD 88 flags=25 2017/11/17 13:30:10.365 kid1| 33,2| client_side.cc(3943) httpsSslBumpAccessCheckDone: sslBump needed for local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 method 4 2017/11/17 13:30:10.365 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:10.365 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- CONNECT 89.16.167.134:443 HTTP/1.1 Host: 89.16.167.134:443 ---------- 2017/11/17 13:30:10.365 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 89.16.167.134:443 is DENIED; last ACL checked: allowed_useragent 2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.385 kid1| 83,2| client_side.cc(3843) clientNegotiateSSL: clientNegotiateSSL: New session 0xdbdc70 on FD 8 (10.215.144.48:65237) 2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- GET / HTTP/1.1 Host: www.gentoo.org User-Agent: MyDeniedUAstring Accept: */* ---------- 2017/11/17 13:30:10.386 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is DENIED, because it matched allowed_useragent 2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 13:30:10.386 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is ALLOWED, because it matched allowed_useragent 2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY: --------- HTTP/1.1 302 Found Server: squid Mime-Version: 1.0 Date: Fri, 17 Nov 2017 12:30:10 GMT Content-Type: text/html;charset=utf-8 Content-Length: 0 Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=www.gentoo.org&i=10.215.144.48&M=GET&o=&R=/&T=Fri,%2017%20Nov%202017%2012%3A30%3A10%20GMT&U=https%3A%2F%2Fwww.gentoo.org%2F&u=https%3A%2F%2Fwww.gentoo.org%2F&w=IT%40mydomain.org&x=&acl=denied_useragent X-Squid-Error: 403 Access Denied X-Cache: MISS from proxy-server1 X-Cache-Lookup: NONE from proxy-server1:3227 Connection: close Now for plain HTTP with example 2. # curl --user-agent MyDeniedUAstring http://www.fltk.org/index.php -> As expected. It blocks access. Full log: 2017/11/17 15:56:52.648 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 85 2017/11/17 15:56:52.648 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3228 remote=[::] FD 85 flags=25 2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17 2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- GET /index.php HTTP/1.1 Host: www.fltk.org User-Agent: MyDeniedUAstring Accept: */* ---------- 2017/11/17 15:56:52.648 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is DENIED; last ACL checked: allowed_useragent 2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable 2017/11/17 15:56:52.648 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET http://www.fltk.org/index.php is ALLOWED, because it matched allowed_useragent 2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17 2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY: --------- HTTP/1.1 302 Found Server: squid Mime-Version: 1.0 Date: Fri, 17 Nov 2017 14:56:52 GMT Content-Type: text/html;charset=utf-8 Content-Length: 0 Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=www.fltk.org&i=10.215.144.48&M=GET&o=&R=/index.php&T=Fri,%2017%20Nov%202017%2014%3A56%3A52%20GMT&U=http%3A%2F%2Fwww.fltk.org%2Findex.php&u=http%3A%2F%2Fwww.fltk.org%2Findex.php&w=IT%40mydomain.org&x=&acl=denied_useragent X-Squid-Error: 403 Access Denied X-Cache: MISS from proxy-server1 X-Cache-Lookup: NONE from proxy-server1:3227 Connection: keep-alive However, now comes the interesting part. # curl --user-agent MyAllowedUAstring http://www.fltk.org/index.php -> works as expected (I see the web site). Full log: 2017/11/17 15:55:23.550 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 85 2017/11/17 15:55:23.550 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3228 remote=[::] FD 85 flags=25 2017/11/17 15:55:23.551 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17 2017/11/17 15:55:23.551 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST: --------- GET /index.php HTTP/1.1 Host: www.fltk.org User-Agent: MyAllowedUAstring Accept: */* ---------- 2017/11/17 15:55:23.551 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring' 2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(805) aclMatchExternal: bllookup("http www.fltk.org 80 /index.php") = lookup needed 2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(808) aclMatchExternal: "http www.fltk.org 80 /index.php": queueing a call. 2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(1444) Start: fg lookup in 'bllookup' for 'http www.fltk.org 80 /index.php' 2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(811) aclMatchExternal: "http www.fltk.org 80 /index.php": return -1. 2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(1372) externalAclHandleReply: reply={result=OK, notes={message: www.fltk.org site not found in blacklist; }} 2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(1288) external_acl_cache_add: external_acl_cache_add: Adding 'http www.fltk.org 80 /index.php' = ALLOWED 2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(841) aclMatchExternal: bllookup = ALLOWED 2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is ALLOWED; last ACL checked: bl_lookup 2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(721) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW 2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is ALLOWED; last ACL checked: bl_lookup 2017/11/17 15:55:23.554 kid1| 88,2| client_side_reply.cc(593) cacheHit: clientProcessHit: Vary detected! 2017/11/17 15:55:23.554 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17, url=http://www.fltk.org/index.php 2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://www.fltk.org/index.php' 2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED 2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED 2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=10.215.144.48 remote=66.39.46.122:80 flags=25 2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths: timedout = 0 2017/11/17 15:55:23.708 kid1| 11,2| http.cc(2229) sendRequest: HTTP Server local=10.215.144.48:35373 remote=66.39.46.122:80 FD 13 flags=25 2017/11/17 15:55:23.708 kid1| 11,2| http.cc(2230) sendRequest: HTTP Server REQUEST: --------- GET /index.php HTTP/1.1 User-Agent: MyAllowedUAstring Accept: */* Host: www.fltk.org Cache-Control: max-age=259200 Connection: keep-alive ---------- 2017/11/17 15:55:23.884 kid1| ctx: enter level 0: 'http://www.fltk.org/index.php' 2017/11/17 15:55:23.884 kid1| 11,2| http.cc(719) processReplyHeader: HTTP Server local=10.215.144.48:35373 remote=66.39.46.122:80 FD 13 flags=25 2017/11/17 15:55:23.884 kid1| 11,2| http.cc(720) processReplyHeader: HTTP Server REPLY: --------- HTTP/1.1 200 OK Date: Fri, 17 Nov 2017 14:55:23 GMT Server: Apache/2.4.29 Cache-Control: no-cache Vary: Accept-Encoding Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html 3a02 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";> <html> <head> <title>Fast Light Toolkit - Fast Light Toolkit (FLTK)</title> <meta http-equiv='Pragma' content='no-cache'> <meta http-equiv='Content-Type' content='text/html; charset=utf-8'> <link rel='stylesheet' type='text/css' href='fltk.css'> <link rel='alternate' title='FLTK RSS' type='application/rss+xml' href='index.rss'> <link rel='shortcut icon' href='favicon.ico' type='image/x-icon'> <meta name='keywords' content='gui toolkit,c++,linux,unix,macos x,x11,windows'> </head> <body> <table width='100%' border='0' cellspacing='0' cellpadding='0' summary='Page'> <tr class='header'><td valign='top' width='15' rowspan='2'><a href='index.php'><img src='images/top-left.gif' width='15' height='70' border='0' alt=''></a></td><td valign='top' width='224' rowspan='2'><a href='index.php'><img src='images/top-middle.gif' width='224' height='70' border='0' alt=''></a></td><td width='100%' height='40'><h1>Fast Light Toolkit</h1> </td><td align='right' nowrap> <table cellpadding=0 cellspacing=0 border=0><tr><td valign=top nowrap> <a href=fltk-rss.xml><img src=images/rss-fee ---------- 2017/11/17 15:55:23.885 kid1| ctx: exit level 0 2017/11/17 15:55:23.885 kid1| 23,2| url.cc(407) urlParse: urlParse: URI has whitespace: {icap://127.0.0.1:1344/clamav ICAP/1.0 } 2017/11/17 15:55:24.038 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring' 2017/11/17 15:55:24.038 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET http://www.fltk.org/index.php is ALLOWED, because it matched denied_mimetypes_rep 2017/11/17 15:55:24.038 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17 2017/11/17 15:55:24.038 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY: --------- HTTP/1.1 200 OK Date: Fri, 17 Nov 2017 14:55:23 GMT Server: Apache/2.4.29 Cache-Control: no-cache Vary: Accept-Encoding Content-Type: text/html Via: ICAP/1.0 proxy-server1.hospitalmanacor.org (C-ICAP/0.5.2 SquidClamav/Antivirus service ) X-Cache: MISS from proxy-server1 X-Cache-Lookup: MISS from proxy-server1:3227 Transfer-Encoding: chunked Connection: keep-alive How can I modify my example 2 settings so this access control works the same way with both http and https in an ssl-bumped environment. Thanks, Vieri _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
