On 9/30/17, Rafael Akchurin <rafael.akchurin@xxxxxxxxxxxx> wrote: > Hello Jeff, > > Do not forget Google and YouTube are now using brotli encoding extensively, > not only gzip. > > Best regards, > Rafael Akchurin > >> Op 30 sep. 2017 om 23:49 heeft Jeffrey Merkey <jeffmerkey@xxxxxxxxx> het >> volgende geschreven: >> >>> On 9/30/17, Eliezer Croitoru <eliezer@xxxxxxxxxxxx> wrote: >>> Hey Jeffrey, >>> >>> What happens when you disable the next icap service this way: >>> icap_service service_avi_resp respmod_precache >>> icap://127.0.0.1:1344/cherokee bypass=0 >>> adaptation_access service_avi_resp deny all >>> >>> Is it still the same? >>> What I suspect is that the requests are defined to accept gzip >>> compressed >>> objects and the icap service is not "gnuzip" them which results in what >>> you >>> see. >>> >>> To make sure that squid is not at fault here try to disable both icap >>> services and then add then one at a time and see which of this triangle >>> is >>> giving you trouble. >>> I enhanced an ICAP library which is written in GoLang at: >>> https://github.com/elico/icap >>> >>> And I have couple examples on how to work with http requests and >>> responses >>> at: >>> https://github.com/andybalholm/redwood/ >>> https://github.com/andybalholm/redwood/search?utf8=%E2%9C%93&q=gzip&type= >>> >>> Let me know if you need help finding out the issue. >>> >>> All The Bests, >>> Eliezer >>> >>> ---- >>> Eliezer Croitoru >>> Linux System Administrator >>> Mobile: +972-5-28704261 >>> Email: eliezer@xxxxxxxxxxxx >>> >>> >>> >>> -----Original Message----- >>> From: squid-users [mailto:squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx] On >>> Behalf Of Jeffrey Merkey >>> Sent: Saturday, September 30, 2017 23:28 >>> To: squid-users <squid-users@xxxxxxxxxxxxxxxxxxxxx> >>> Subject: SSL Bump Failures with Google and Wikipedia >>> >>> Hello All, >>> >>> I have been working with the squid server and icap and I have been >>> running into problems with content cached from google and wikipedia. >>> Some sites using https, such as Centos.org work perfectly with ssl >>> bumping and I get the decrypted content as html and it's readable. >>> Other sites, such as google and wikipedia return what looks like >>> encrypted traffic, or perhaps mime encoded data, I am not sure which. >>> >>> Are there cases where squid will default to direct mode and not >>> decrypt the traffic? I am using the latest squid server 3.5.27. I >>> really would like to get this working with google and wikipedia. I >>> reviewed the page source code from the browser viewer and it looks >>> nothing like the data I am getting via the icap server. >>> >>> Any assistance would be greatly appreciated. >>> >>> The config I am using is: >>> >>> # >>> # Recommended minimum configuration: >>> # >>> >>> # Example rule allowing access from your local networks. >>> # Adapt to list your (internal) IP networks from where browsing >>> # should be allowed >>> >>> acl localnet src 127.0.0.1 >>> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network >>> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network >>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network >>> acl localnet src fc00::/7 # RFC 4193 local private network range >>> acl localnet src fe80::/10 # RFC 4291 link-local (directly >>> plugged) machines >>> >>> acl SSL_ports port 443 >>> acl Safe_ports port 80 # http >>> acl Safe_ports port 21 # ftp >>> acl Safe_ports port 443 # https >>> acl Safe_ports port 70 # gopher >>> acl Safe_ports port 210 # wais >>> acl Safe_ports port 1025-65535 # unregistered ports >>> acl Safe_ports port 280 # http-mgmt >>> acl Safe_ports port 488 # gss-http >>> acl Safe_ports port 591 # filemaker >>> acl Safe_ports port 777 # multiling http >>> acl CONNECT method CONNECT >>> >>> # >>> # Recommended minimum Access Permission configuration: >>> # >>> # Deny requests to certain unsafe ports >>> http_access deny !Safe_ports >>> >>> # Deny CONNECT to other than secure SSL ports >>> http_access deny CONNECT !SSL_ports >>> >>> # Only allow cachemgr access from localhost >>> http_access allow localhost manager >>> http_access deny manager >>> >>> # We strongly recommend the following be uncommented to protect innocent >>> # web applications running on the proxy server who think the only >>> # one who can access services on "localhost" is a local user >>> #http_access deny to_localhost >>> >>> # >>> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS >>> # >>> >>> # Example rule allowing access from your local networks. >>> # Adapt localnet in the ACL section to list your (internal) IP networks >>> # from where browsing should be allowed >>> http_access allow localnet >>> http_access allow localhost >>> >>> # And finally deny all other access to this proxy >>> http_access deny all >>> >>> # Squid normally listens to port 3128 >>> #http_port 3128 >>> >>> # Uncomment and adjust the following to add a disk cache directory. >>> #cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256 >>> >>> # Leave coredumps in the first cache dir >>> coredump_dir /usr/local/squid/var/cache/squid >>> >>> # >>> # Add any of your own refresh_pattern entries above these. >>> # >>> refresh_pattern ^ftp: 1440 20% 10080 >>> refresh_pattern ^gopher: 1440 0% 1440 >>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >>> refresh_pattern . 0 20% 4320 >>> >>> http_port 3128 ssl-bump generate-host-certificates=on >>> dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem >>> http_port 3129 >>> >>> # SSL Bump Config >>> always_direct allow all >>> ssl_bump server-first all >>> sslproxy_cert_error deny all >>> sslproxy_flags DONT_VERIFY_PEER >>> sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s /var/lib/ssl_db >>> -M 4MB sslcrtd_children 8 startup=1 idle=1 >>> >>> # For squid 3.5.x >>> #sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s /var/lib/ssl_db -M >>> 4MB >>> >>> # For squid 4.x >>> # sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s >>> /var/lib/ssl_db -M 4MB >>> >>> icap_enable on >>> icap_send_client_ip on >>> icap_send_client_username on >>> icap_client_username_header X-Authenticated-User >>> icap_preview_enable on >>> icap_preview_size 1024 >>> icap_service service_avi_req reqmod_precache >>> icap://127.0.0.1:1344/request >>> bypass=1 >>> adaptation_access service_avi_req allow all >>> >>> icap_service service_avi_resp respmod_precache >>> icap://127.0.0.1:1344/cherokee bypass=0 >>> adaptation_access service_avi_resp allow all >>> >>> Jeff >>> _______________________________________________ >>> squid-users mailing list >>> squid-users@xxxxxxxxxxxxxxxxxxxxx >>> http://lists.squid-cache.org/listinfo/squid-users >>> >>> >> >> Eliezer, >> >> Well, you certainly hit the nail on the head. I added the following >> code to check the content being sent to the icap server from squid, >> and here is what I found when I check the headers being sent from the >> remote web server: >> >> Code to check for content type and encoding received by the icap >> server added to c-icap: >> >> hdrs = ci_http_response_headers(req); >> content_type = ci_headers_value(hdrs, "Content-Type"); >> if (content_type) >> ci_debug_printf(1,"srv_cherokee: content-type: %s\n", >> content_type); >> >> content_encoding = ci_headers_value(hdrs, "Content-Encoding"); >> if (content_encoding) >> ci_debug_printf(1,"srv_cherokee: content-encoding: %s\n", >> content_encoding); >> >> And the output from scanned pages sent over from squid: >> >> srv_cherokee: init request 0x7f3dbc008eb0 >> pool hits:1 allocations: 1 >> Allocating from objects pool object 5 >> pool hits:1 allocations: 1 >> Geting buffer from pool 4096:1 >> Requested service: cherokee >> Read preview data if there are and process request >> srv_cherokee: content-type: text/html; charset=utf-8 >> srv_cherokee: content-encoding: gzip <-- As you stated, I am >> getting gzipped data >> srv_cherokee: we expect to read :-1 body data >> Allow 204... >> Preview handler return allow 204 response >> srv_cherokee: release request 0x7f3dbc008eb0 >> Store buffer to long pool 4096:1 >> Storing to objects pool object 5 >> Log request to access log file /var/log/i-cap_access.log >> >> >> Wikipedia at https://en.wikipedia.org/wiki/HTTP_compression describes >> the process as: >> >> " ... >> Compression scheme negotiation[edit] >> In most cases, excluding the SDCH, the negotiation is done in two >> steps, described in >> RFC 2616: >> >> 1. The web client advertises which compression schemes it supports >> by including a list >> of tokens in the HTTP request. For Content-Encoding, the list in a >> field called Accept - >> Encoding; for Transfer-Encoding, the field is called TE. >> >> GET /encrypted-area HTTP/1.1 >> Host: www.example.com >> Accept-Encoding: gzip, deflate >> >> 2. If the server supports one or more compression schemes, the >> outgoing data may be >> compressed by one or more methods supported by both parties. If >> this is the case, the >> server will add a Content-Encoding or Transfer-Encoding field in >> the HTTP response with >> the used schemes, separated by commas. >> >> HTTP/1.1 200 OK >> Date: mon, 26 June 2016 22:38:34 GMT >> Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux) >> Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT >> Accept-Ranges: bytes >> Content-Length: 438 >> Connection: close >> Content-Type: text/html; charset=UTF-8 >> Content-Encoding: gzip >> >> The web server is by no means obligated to use any compression method – >> this >> depends on the internal settings of the web server and also may >> depend on the internal >> architecture of the website in question. >> >> In case of SDCH a dictionary negotiation is also required, which >> may involve additional >> steps, like downloading a proper dictionary from . >> .." >> >> >> So, it looks like it is a feature of the browser. So, is it possible >> to have squid gunzip the data or configure the browser not to send the >> header to remove "Accept-Encoding: gzip, deflate" from the request >> sent to the remote server telling it to gzip the data? >> >> Thanks >> >> Jeff >> _______________________________________________ >> squid-users mailing list >> squid-users@xxxxxxxxxxxxxxxxxxxxx >> http://lists.squid-cache.org/listinfo/squid-users > Resetting the browser settings to disable gzip worked. I can see the pages now. The best way to implement this is to intercept the accept-encoding header and either remove it or modify it from icap before being sent to the remote server. Jeff _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
