Re: Squid for windows Very slow downloads of large files through squid with normal uploads

Yuri <yvoinov@xxxxxxxxx> · Fri, 23 Mar 2018 04:38:53 +0600



    22.03.2018 23:10, Keith Hartley пишет:

    
        I am using squid 3.5 for windows as a
          transparent proxy to provide internet access to 7 servers in a
          secure environment that otherwise does not have internet
          access. I have two squids running behind a load balancer, each
          one is running server 2016 core with 2 Xeon processors that is
          either haswell generation with 1:1 physical processor to
          virtual processor mapping or a hyper-threading Broadwell
          generation processor that is 1:1 logical processor to virtual
          processor mapping, depending on how they are provisioned when
          they get started.
         
        Doing a bandwidth test directly in the VM I
          am able to get internet throughput of 800-1200 Mbps.
         
        Doing a file copy to and from the VM I am
          able to get 1200 Mbps lan throughput.
         
        In proxied uploads I have observed speeds
          as high as 120 Mbps, which is more than enough for what I need
          and the bottleneck is likely in the backup software rather
          than squid. Uploads performance I am not worried about where
          they are at now – even if I only got 20-30 Mbps it would be
          adequate for what I need it for.
         
        Downloads however are very slow. Small
          files do not seem to be impacted. Using the test a
          thinkbroadband.com/download, files up to 20 Mb will download
          at a reasonable 20-30 Mbps, but when I get to 50, it slows
          down to about 17 Mbps, and when I download AD Connect from
          Microsoft, which is about 80 Mb, I can see it start at about
          30 Mbps, but eventually goes down to about 115 kbps and levels
          off. When I put an IP on the server I am using for testing
          that proxies through squid, I am able to download the file at
          several hundred mbps.  When I download the same file on the
          squid server – I can’t tell exactly what throughput I was
          getting, but the 80 Mb file downloaded within 5 seconds.
         
        In both squid servers, other than when the
          servers were booting, processor activity has not exceeded 9%
          in the last 7 days but usually sits below 2%. Memory usage has
          not exceeded 2 Gb, leaving 2 Gb free.
         
        I am using OpenDNS for a DNS source, and
          have tried changing DNS to level3 but it made no performance
          difference.
         
        I think that this may be squid trying to
          cache something, but had tried to turn all caching off.
         
        My cache.log doesn’t really have anything
          interesting in it that I can see. It’s the same ~30 or so log
          entries each time the service starts, and that is about it.
          Here it is:
          
         
        2018/03/22 09:47:27 kid1| Set Current
          Directory to /var/cache/squid
        2018/03/22 09:47:27 kid1| Starting Squid
          Cache version 3.5.27 for x86_64-unknown-cygwin...
        2018/03/22 09:47:27 kid1| Service Name:
          squid
        2018/03/22 09:47:27 kid1| Process ID 1164
        2018/03/22 09:47:27 kid1| Process Roles:
          worker
        2018/03/22 09:47:27 kid1| With 3200 file
          descriptors available
        2018/03/22 09:47:27 kid1| Initializing IP
          Cache...
        2018/03/22 09:47:27 kid1| parseEtcHosts:
          /etc/hosts: (2) No such file or directory
        2018/03/22 09:47:27 kid1| DNS Socket
          created at [::], FD 5
        2018/03/22 09:47:27 kid1| DNS Socket
          created at 0.0.0.0, FD 6
        2018/03/22 09:47:27 kid1| Adding nameserver
          208.67.222.222 from squid.conf
        2018/03/22 09:47:27 kid1| Adding nameserver
          208.67.220.220 from squid.conf
        2018/03/22 09:47:27 kid1| Logfile: opening
          log daemon:/var/log/squid/access.log
        2018/03/22 09:47:27 kid1| Logfile Daemon:
          opening log /var/log/squid/access.log
        2018/03/22 09:47:27 kid1| WARNING: no_suid:
          setuid(0): (22) Invalid argument
        2018/03/22 09:47:27 kid1| Store logging
          disabled
        2018/03/22 09:47:27 kid1| Swap maxSize 0 +
          262144 KB, estimated 20164 objects
        2018/03/22 09:47:27 kid1| Target number of
          buckets: 1008
        2018/03/22 09:47:27 kid1| Using 8192 Store
          buckets
        2018/03/22 09:47:27 kid1| Max Mem  size:
          262144 KB
        2018/03/22 09:47:27 kid1| Max Swap size: 0
          KB
        2018/03/22 09:47:27 kid1| Using Least Load
          store dir selection
        2018/03/22 09:47:27 kid1| Set Current
          Directory to /var/cache/squid
        2018/03/22 09:47:27 kid1| Finished loading
          MIME types and icons.
        2018/03/22 09:47:27 kid1| HTCP Disabled.
        2018/03/22 09:47:27 kid1| Squid plugin
          modules loaded: 0
        2018/03/22 09:47:27 kid1| Adaptation
          support is off.
        2018/03/22 09:47:27 kid1| Accepting HTTP
          Socket connections at local=[::]:3128 remote=[::] FD 10
          flags=9
        2018/03/22 09:47:28 kid1| storeLateRelease:
          released 0 objects
         
         
        And this is my squid.conf:
         
        #
        # Recommended minimum configuration:
        #
         
        # Example rule allowing access from your
          local networks.
        # Adapt to list your (internal) IP networks
          from where browsing
        # should be allowed
         
        #acl localnet src 10.0.0.0/8           #
          RFC1918 possible internal network
        #acl localnet src 172.16.0.0/12    #
          RFC1918 possible internal network
        #acl localnet src 192.168.0.0/16  # RFC1918
          possible internal network
        acl localnet src fc00::/7       # RFC 4193
          local private network range
        acl localnet src fe80::/10      # RFC 4291
          link-local (directly plugged) machines
        acl WSUS src 192.168.225.4/32
        acl BACKUP src 192.168.225.11/32
        acl ADFS src 192.168.224.7/32
        acl ADFS src 192.168.228.8/32
        acl DEVWEB src 192.168.226.6/32
        acl UATWEB src 192.168.226.13/32
        acl PRDWEB src 192.168.226.8/32
        acl PRDWEB src 192.168.226.9/32
         
         
        acl SSL_ports port 443
        acl Safe_ports port 80                    #
          http
        #acl Safe_ports port 21                  #
          ftp
        acl Safe_ports port 443                  #
          https
        #acl Safe_ports port 70                  #
          gopher
        #acl Safe_ports port
          210                                # wais
        #acl Safe_ports port
          1025-65535                # unregistered ports
        #acl Safe_ports port
          280                                # http-mgmt
        #acl Safe_ports port
          488                                # gss-http
        #acl Safe_ports port
          591                                # filemaker
        #acl Safe_ports port
          777                                # multiling http
        acl CONNECT method CONNECT
         
        #
        # Recommended minimum Access Permission
          configuration:
        #
         
        # Only allow cachemgr access from localhost
        #http_access allow localhost manager
        #http_access deny manager
         
        # Deny requests to certain unsafe ports
        http_access deny !Safe_ports
         
        # Deny CONNECT to other than secure SSL
          ports
        http_access deny CONNECT !SSL_ports
         
        # We strongly recommend the following be
          uncommented to protect innocent
        # web applications running on the proxy
          server who think the only
        # one who can access services on
          "localhost" is a local user
        #http_access deny to_localhost
         
        #
        # INSERT YOUR OWN RULE(S) HERE TO ALLOW
          ACCESS FROM YOUR CLIENTS
        #
         
        # Example rule allowing access from your
          local networks.
        # Adapt localnet in the ACL section to list
          your (internal) IP networks
        # from where browsing should be allowed
        http_access allow localnet
        http_access allow localhost
        http_access allow WSUS
        http_access allow ADFS
        http_access allow BACKUP
        http_access allow DEVWEB
        http_access allow UATWEB
        http_access allow PRDWEB
         
        # And finally deny all other access to this
          proxy
        http_access deny all
         
        # Squid normally listens to port 3128
        http_port 3128
         
        # Uncomment the line below to enable disk
          caching - path format is /cygdrive/<full path to cache
          folder>, i.e.
        #cache_dir aufs /cygdrive/d/squid/cache
          3000 16 256
        cache deny all
         
         
        # Leave coredumps in the first cache dir
        coredump_dir /var/cache/squid
         
        # Add any of your own refresh_pattern
          entries above these.
        refresh_pattern ^ftp:                    
          1440       20%        10080
        refresh_pattern ^gopher:           
          1440       0%          1440
        refresh_pattern -i (/cgi-bin/|\?)
          0             0%          0
        refresh_pattern
          .                             0              20%        4320
         
        dns_nameservers 208.67.222.222
          208.67.220.220
         
        max_filedescriptors 3200
         
         
        Does anyone see anything I am missing here?
      
    
    Yes. In your almost default configuration (it is complete
    squid.conf?) obvious thing is:

    
    a) You do not use on-disk cache.

    b) You use memory cache by default - i.e. 256 Mb. 

    c) You cache nothing due to deny all cache. So, it makes useless
    cache_mem default.

    d) Your configuration technically useless. I see neither proxying
    parameters, nor caching. Your squid now only additional hop for
    files. No more.

    
    So, squid nothing to do here. It simple should retransmit GET (GET?)
    request to server, and, without any caching/storing, retransmit it
    to user.

    
    Still correct?

    
    This put us directly to raw network IO. Without any buffering (which
    can be - but don't - your squid).

    
    On your place, I can start playing around with cache_mem parameter;
    of course, only after removing cache deny all.

    
    And after some experiments, may be, will make decision about drop out
    useless Squid's box.

    
    Seriously, what role of squid's here? Just setup border firewall to
    your servers to access it to Internet. It will be enough.

    
        My access.log doesn’t really have anything
          interesting in it either, it just looks like it is working
          normally. I can attach that too if anyone wants to look at it
          after I redact some of the hosts.
         
         
        Keith Hartley
        Network Engineer II
        MCSE: Productivity, MCSA: Server 2008,
            2012, Office 365 
          |
        Certified Meraki Network Associate,
            Security+
        Geocent,
              LLC
        o:
          504-405-3578
        a:
          2219 Lakeshore drive Ste 300, New Orleans, LA 70122
        w:
          www.geocent.com| e:
          khartley@xxxxxxxxxxx
         
           
        Confidentiality Notice:
      This email communication may contain
        confidential information, may be legally privileged, and is
        intended only for the use of the intended recipients(s)
        identified. Any unauthorized review, use, distribution,
        downloading, or copying of this communication is strictly
        prohibited. If you are not the intended recipient and have
        received this message in error, immediately notify the sender by
        reply email, delete the communication, and destroy all copies.
        Thank you.
      

      _______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

    
    -- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************
  

Attachment:
signature.asc

Description: OpenPGP digital signature
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users