Re: Transparent Squid Proxy Server

Nathan Hoad <nathan@xxxxxxxxxxxxxxxx> · Sun, 12 Jul 2015 11:02:35 +1000

I'm using 3.5 with transparent server first bumping in ~100 deployments so far, it works just fine, excluding with SNI and everything.
On 12/07/2015 10:58 am, "Yuri Voinov" <yvoinov@xxxxxxxxx> wrote:

    Man,

    3.5.x don't work with server-first. It must be for backward
    compatibility - but don't be.

    Also, AFAIK, 3.5.x series don't work with transparent NAT
    interception in bump mode. Fake certs are generated, but with IP
    against hostnames (in all my test installations).

    So, if you strictly need working bump with transparent interception,
    rollback to 3.4.

    WBR, Yuri.

    24.06.15 12:04, Reet Vyas пишет:

        Hi

         Below is my squid file , I have configured squid 3.5.3 with
        ssl, but I cant filter https traffic and also in access log I
        cant see https in access logs.

        #

          # Recommended minimum configuration:

          #

          # Example rule allowing access from your local networks.

          # Adapt to list your (internal) IP networks from where
          browsing

          # should be allowed

          acl localnet src 116.72.152.37 192.168.0.0/24 # Sesuaikan
          dengan ip client/local

          acl SSL_ports port 443

          acl Safe_ports port 80  # http

          acl Safe_ports port 21  # ftp

          acl Safe_ports port 443  # https

          acl Safe_ports port 70  # gopher

          acl Safe_ports port 210  # wais

          acl Safe_ports port 1025-65535 # unregistered ports

          acl Safe_ports port 280  # http-mgmt

          acl Safe_ports port 488  # gss-http

          acl Safe_ports port 591  # filemaker

          acl Safe_ports port 777  # multiling http

          # storeid *test*

          acl urlrewrite dstdomain .fbcdn.net .akamaihd.net

          acl speedtest url_regex -i speedtest\/.*\.(jpg|txt)\?.*

          acl reverbnation url_regex -i
          reverbnation.*audio_player.*ec_stream_song.*$

          acl utmgif url_regex -i utm.gif.*

          acl playstoreandroid url_regex -i
          c.android.clients.google.com.market.GetBinary.GetBinary.*

          acl idyoutube url_regex -i
          youtube.*(ptracking|stream_204|player_204).*(v\=|docid\=|video_id\=).*$

          acl videoyoutube url_regex -i
          (youtube|googlevideo).*videoplayback\?

          acl videoyoutube url_regex -i
          (youtube|googlevideo).*videoplayback\?

          acl CONNECT method CONNECT

          acl getmethod method GET

          acl loop_302 http_status 302

          acl step1 at_step SslBump1

          acl youtube dstdomain .youtube.com

          acl blocksites dstdomain "/etc/squid/restricted-sites.squid"

          # TAG: QUERY

          #
-----------------------------------------------------------------------------

          acl QUERY urlpath_regex -i
(hackshield|blank.html|infinity.js|hshield.da|renew_session_token.php|recaptcha.js|dat.asp|notice.swf|patchlist.txt|hackshield|captcha|reset.css|update.ver|notice.html|updates.txt|gamenotice|images.kom|patchinfo.xml|noupdate.ui|\.Xtp|\.htc|\.txt)

          acl QUERY urlpath_regex -i
(patch.conf|uiimageset.xml.iop|gashaponwnd.xml.iop|loading.swf|download.swf|version.list|version.ini|launch.jnlp|server_patch.cfg.iop|core.swf|Loading.swf|resouececheck.sq|mainloading.swf|config.xml|gemmaze.swf|xml.png|size.xml|resourcesbar.swf|version.xml|version.list|delete.ini)

          acl QUERY urlpath_regex -i
          \.(jsp|asp|aspx|cfg|iop|zip|php|xml|html)(\?|$)

          cache deny QUERY

          cache deny youtube

          #

          acl dontstore url_regex
^http:\/\/(([\d\w-]*(\.[^\.\-]*?\..*?))(\/\mosalsal\/[\d]{4}\/.*\/)(.*\.flv))\?start.*

          acl dontstore url_regex redbot\.org \.php

          acl dontstore url_regex -i ^http:\/\/.*gemscool\.com\/.*

          acl dontstore url_regex \.(aspx|php)\?

          acl dontstore url_regex
          goldprice\.org\/NewCharts\/gold\/images\/.*\.png

          acl dontstore url_regex
          google\.co(m|\.[a-z]{2})\/complete\/search\?

          acl dontstore url_regex
redirector\.([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/(get_video\?|videodownload\?|videoplayback.*id|get_video_info\?|ptracking\?|player_204\?|stream_204\?).*

          acl store_yt_id url_regex -i
youtube.*(ptracking|stream_204|playback|player_204|watchtime|set_awesome|s\?|ads).*(video_id|docid|\&v|content_v)\=([^\&\s]*).*$

          acl store_id_list_yt url_regex -i
          (youtube|googlevideo).*videoplayback.*$

          acl store_id_list_yt url_regex
^https?\:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/(get_video\?|videodownload\?|videoplayback.*id).*

          acl store-id_list urlpath_regex -i dl\.sourceforge\.net

          acl store-id_list urlpath_regex -i \.ytimg\.com

          acl store-id_list urlpath_regex -i \.(akamaihd|fbcdn)\.net

          acl store_id_list urlpath_regex -i
          [a-zA-Z]{2}[0-9]*\.4shared\.com\/download\/

          acl store_id_list_url url_regex
          ^http:\/\/[0-9]\.bp\.blogspot\.com.*\.(jpeg|jpg|png|gif|ico)

          acl store_id_list_url url_regex
          ^http[s]?:\/\/.*\.twimg\.com\/(.*)\.(gif|jpeg|jpg|png|js|css)

          acl store_id_list_url url_regex
          ^http[s]?:\/\/(media|static)\.licdn\.com\/.*\.(png|jpg|gif|woff)

          acl store_id_list_url url_regex
          ^https:\/\/fb(static|cdn)\-.*\-a.akamaihd.net\/(.*)\.(gif|jpeg|jpg|png|js|css|mp4)

          acl store_id_list_url url_regex
          ^http:\/\/.*\.ak\.fbcdn\.net\/.*\.(gif|jpg|png|js|mp4)

          # pass requests 

          url_rewrite_program /etc/squid/phpredir.php

          url_rewrite_access allow youtube

          request_header_access Range deny store_id_list_yt

          range_offset_limit 10 KB store_id_list_yt

###############################################################################

          # Recommended minimum Access Permission configuration:

          #

          # Deny requests to certain unsafe ports

###############################################################################

          http_access deny !Safe_ports

          http_access deny CONNECT !SSL_ports

          http_access deny blocksites

          http_access allow localhost manager

          http_access deny manager

          http_access allow localnet

          http_access allow localhost

          http_access deny all

###############################################################################

          # squid ssl_bump option

###############################################################################

          always_direct allow all

          ssl_bump server-first all

          sslproxy_cert_error deny all

          sslproxy_flags DONT_VERIFY_PEER

          sslcrtd_program /usr/lib/squid/ssl_crtd -s
          /var/lib/squid/ssl_db -M 4MB

          sslcrtd_children 8 startup=1 idle=1

          #ssl_bump peek step1

          #ssl_bump bump all 

###############################################################################

          # Squid normally listens to port 3128

###############################################################################

          https_port 3130 intercept ssl-bump
          generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
          cert=/etc/squid/ssl_certs/squid.crt
          key=/etc/squid/ssl_certs/squid.key

          http_port 3129 intercept

          http_port 3128

          # TAG: Store-id Program

          #
-----------------------------------------------------------------------------

          store_id_program /usr/bin/perl /etc/squid/store-id.pl

          store_id_children 100 startup=0 idle=1 concurrency=1000

          # TAG: Store-id Access

          #
-----------------------------------------------------------------------------

          store_id_access allow urlrewrite

          store_id_access allow speedtest

          store_id_access allow reverbnation

          store_id_access allow utmgif

          store_id_access allow playstoreandroid

          store_id_access allow idyoutube

          store_id_access allow videoyoutube

          store_id_access deny dontstore

          store_id_access deny !getmethod

          store_id_access allow store_id_list_yt

          store_id_access allow store_yt_id

          store_id_access allow store-id_list

          store_id_access deny all

          store_id_bypass on

          # TAG: Youtube 302

          #
-----------------------------------------------------------------------------

          store_miss deny store_id_list_yt loop_302

          send_hit deny store_id_list_yt loop_302

###############################################################################

          ## MEMORY CACHE OPTIONS

###############################################################################

          client_dst_passthru on

          cache_mem 1024 MB

          maximum_object_size_in_memory 1024 KB

          memory_cache_shared off

          memory_cache_mode disk

          memory_replacement_policy heap GDSF

###############################################################################

          ## DISK CACHE OPTIONS

###############################################################################

          cache_replacement_policy heap LFUDA

          minimum_object_size 1 bytes

          maximum_object_size 10 GB

###############################################################################

          # Uncomment and adjust the following to add a disk cache
          directory.

###############################################################################

          cache_dir aufs /usr/local/cache_proxy 25000 16 256 # sesuaikan
          dengan drive penyimpanan cache 

          store_dir_select_algorithm round-robin

          cache_swap_low 90

          cache_swap_high 95

###############################################################################

          # Leave coredumps in the first cache dir

###############################################################################

          coredump_dir /var/spool/squid

###############################################################################

          ## LOGFILE OPTIONS

###############################################################################

          #access_log daemon:/tmp/access.log !log

          #logfile_daemon /usr/lib/squid/log_file_daemon

          cache_store_log none

          logfile_rotate 1

          mime_table /etc/squid/mime.conf

          pid_filename /var/run/squid.pid

          strip_query_terms off

          buffered_logs off

###############################################################################

          ## OPTIONS FOR TROUBLESHOOTING

###############################################################################

          #cache_log /tmp/cache.log

          cache_log /dev/null

          #debug_options ALL,1 22,3

          coredump_dir /var/spool/squid

###############################################################################

          ## OPTIONS FOR TUNING THE CACHE

###############################################################################

          max_stale 1 years

          vary_ignore_expire on

          shutdown_lifetime 10 seconds

###############################################################################

          # Add any of your own refresh_pattern entries above these.

###############################################################################

          refresh_pattern ^ftp:  1440 20% 10080

          refresh_pattern ^gopher: 1440 0% 1440

          refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

          # Youtube Video

          refresh_pattern -i
          (get_video\?|videoplayback\?|videodownload\?|\.mp4|\.webm|\.flv|((audio|video)\/(webm|mp4)))
          241920 100% 241920 override-expire ignore-reload
          ignore-private ignore-no-store ignore-must-revalidate
          reload-into-ims ignore-auth store-stale

          refresh_pattern -i
          ^https?\:\/\/.*\.googlevideo\.com\/videoplayback.*     10080
          99% 43200 override-lastmod override-expire ignore-reload
          reload-into-ims ignore-private reload-into-ims ignore-auth
          store-stale

          refresh_pattern -i
          ^https?\:\/\/.*\.googlevideo\.com\/videoplayback.*$    241920
          100% 241920 override-expire ignore-reload ignore-private
          ignore-no-store ignore-must-revalidate reload-into-ims
          ignore-auth store-stale

          refresh_pattern (akamaihd|fbcdn)\.net 14400 99% 518400 
          ignore-no-store ignore-private ignore-reload
          ignore-must-revalidate store-stale

          refresh_pattern -i squid\.internal 14400 99% 518400 
          ignore-no-store ignore-private ignore-reload
          ignore-must-revalidate store-stale

          refresh_pattern \.(jpg|png|gif|css|ico)($|\?) 14400 99%
          518400  ignore-no-store ignore-private reload-into-ims
          ignore-must-revalidate store-stale

          refresh_pattern . 0 99% 518400  ignore-no-store ignore-private
          reload-into-ims store-stale 

          # Image Youtube

          refresh_pattern -i (yimg|twimg)\.com\.*         1440 100%
          129600 override-expire ignore-reload reload-into-ims

          refresh_pattern -i (ytimg|ggpht)\.com\.*        1440 80%
          129600 override-expire override-lastmod ignore-auth
          ignore-reload reload-into-ims

          #images facebook

          refresh_pattern -i
          fbcdn.*net\/.*\.((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(\?|.*$))
          241920 99% 241920 ignore-no-store ignore-private
          override-expire override-lastmod reload-into-ims ignore-auth

          refresh_pattern -i
          pixel\.facebook\.com.*\.(jpg|png|gif|ico|css|js) 241920 80%
          241920 override-expire ignore-reload reload-into-ims
          ignore-auth

          refresh_pattern -i \.akamaihd\.net.*\.(jpg|png|gif|ico|css|js)
          241920 80% 241920 override-expire ignore-reload
          reload-into-ims ignore-auth

          refresh_pattern -i ((facebook.com)|(85.131.151.39))\.(jpg|png|gif)
          241920 99% 241920 ignore-reload override-expire
          ignore-no-store store-stale

          refresh_pattern -i
          fbcdn\.net\/.*\.((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(\?|.*$))
          241920 99% 241920 ignore-no-store ignore-private
          override-expire override-lastmod reload-into-ims ignore-auth

          refresh_pattern static\.(xx|ak)\.fbcdn\.net*\.(jpg|gif|png)
          241920 99% 241920 ignore-reload override-expire
          ignore-no-store

          refresh_pattern ^https?\:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)
          241920 99% 241920 ignore-reload override-expire
          ignore-no-store

          # Video Facebook

          refresh_pattern -i
          \.video.ak.fbcdn.net.*\.(mp4|flv|mp3|amf)                   
          10080 80% 43200 override-expire ignore-reload reload-into-ims
          ignore-private ignore-no-store ignore-must-revalidate

          refresh_pattern (audio|video)\/(webm|mp4) 129600 99% 129600
          ignore-reload override-expire override-lastmod
          ignore-must-revalidate  ignore-private ignore-no-store
          ignore-auth store-stale

          refresh_pattern -i ^http://.*squid\.internal.*  241920 100%
          241920 override-lastmod override-expire ignore-reload
          ignore-must-revalidate ignore-private ignore-no-store
          ignore-auth store-stale

          # All File

          refresh_pattern -i
          \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)
          10080 80% 10080 override-expire override-lastmod
          reload-into-ims

          refresh_pattern -i
          \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar|iop|nzp|pak|mar|msp)
          10080 80% 10080 override-expire override-lastmod
          reload-into-ims ignore-reload

          refresh_pattern -i
          \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)
          10080 80% 10080 override-expire override-lastmod
          reload-into-ims

          refresh_pattern -i
          \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob|webm)
          10080 80% 10080 override-expire override-lastmod
          reload-into-ims

          refresh_pattern -i
          \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv)
          10080 80% 10080 override-expire override-lastmod
          reload-into-ims

          refresh_pattern .  0 20% 4320

###############################################################################

          ## ADMINISTRATIVE PARAMETERS

###############################################################################

          cache_mgr reetika@xxxxxxxxxxxxx

          cache_effective_user proxy

          cache_effective_group proxy

          visible_hostname foxysquid.foxymoron.tv

          unique_hostname foxysquid.foxymoron.tv

###############################################################################

          ## PERSISTENT CONNECTION HANDLING

###############################################################################

          detect_broken_pconn on

          client_persistent_connections off

          server_persistent_connections on

###############################################################################

          ## ERROR PAGE OPTIONS

###############################################################################

          error_directory /usr/share/squid/errors/en

          error_log_languages off

###############################################################################

          ## DNS OPTIONS

###############################################################################

          check_hostnames off

          hosts_file /etc/hosts

          connect_retries 2

          ipcache_low 90

          ipcache_high 95

          ipcache_size 84024                        # 2x Besar RAM 

          fqdncache_size 64024                        # real RAM
          Hardware

          pipeline_prefetch 100

###############################################################################

          ## MISCELLANEOUS

###############################################################################

          memory_pools off

          reload_into_ims on

          uri_whitespace strip

          max_filedescriptors 65536

        IPtable rules :

          ................................................

        My IPtables Rules

          Chain PREROUTING (policy ACCEPT 27405 packets, 1872K bytes)

           pkts bytes target     prot opt in     out    
          source               destination         

          76873 4457K DNAT       tcp  --  eth1   *       0.0.0.0/0           
          0.0.0.0/0           
          tcp dpt:80 to:192.168.0.200:3129

             26  1184 REDIRECT   tcp  --  eth0   *       0.0.0.0/0           
          0.0.0.0/0           
          tcp dpt:80 redir ports 3129

              0     0 DNAT       tcp  --  eth0   *       0.0.0.0/0           
          0.0.0.0/0           
          tcp dpt:443 to:192.168.0.200:3130

          Chain INPUT (policy ACCEPT 9321 packets, 543K bytes)

           pkts bytes target     prot opt in     out    
          source               destination         

          Chain OUTPUT (policy ACCEPT 1426 packets, 85560 bytes)

           pkts bytes target     prot opt in     out    
          source               destination         

          Chain POSTROUTING (policy ACCEPT 1426 packets, 85560 bytes)

           pkts bytes target     prot opt in     out    
          source               destination         

          81432   14M MASQUERADE  all  --  *      eth0    192.168.0.0/24      
          0.0.0.0/0

        On Fri, Jun 5, 2015 at 1:43 PM, Reet
          Vyas <reet.vyas28@xxxxxxxxx>
          wrote:

                Hi 

                Thanks for reply. I am trying to cache youtube using
                this wiki http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
                but I cant cache youtube.

              I want to cache facebook and youtube. SSl certificate
              installation that I have to do . Please suggest some
              links.

                  On Thu, Jun 4, 2015 at 6:48
                    PM, Amos Jeffries <squid3@xxxxxxxxxxxxx>
                    wrote:

                    On
                        5/06/2015 12:55 a.m., Reet Vyas wrote:

                        > Thank you everyone for helping me to setup
                        squid , Now its working but in

                        > access.logs  I only see tcp_miss if m using
                        same website. I mean squid is

                        > not caching

                      You will get MISS a fair bit more with
                      intercepted traffic than with

                      normal proxied traffic. Particularly on certain
                      major CDN who play

                      tricks with DNS.

                      The reasons and some workarounds to need to be
                      doing are explained in

                      <http://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery>

                          Amos

_______________________________________________

                          squid-users mailing list

                          squid-users@xxxxxxxxxxxxxxxxxxxxx

                          http://lists.squid-cache.org/listinfo/squid-users

      _______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________

squid-users mailing list

squid-users@xxxxxxxxxxxxxxxxxxxxx

http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users