|
Hello guys.. Thanks for the tips. I've ajusted some stuff here and noticed these repeated GETS below.. they are HITS, but why is this happening? lol I have "range_offset_limit none" for this domain (ws.microsoft.com) and: refresh_pattern -i (microsoft|windowsupdate)\.com.*\.(cab|exe|ms[i|u|f]|dat|zip|psf|appx|esd) 483840 80% 483840 override-expire ignore-reload ignore-must-revalidate ignore-private ignore-no-store store-stale These GETs have a "?" in the end, and some options which aren't logged but I tcpdumped it: P1=1456938099&P2=1&P3=1&P4=GlQQBGsBJE22%2bm1FQr3q1RnmAb8%3d Best Regards, -- Heiler Bemerguy - (91) 98151-4894 Assessor Técnico - CINBESA (91) 3184-1751 1456953828.014 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953828.748 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953829.686 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953830.314 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953830.670 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953831.468 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953832.297 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953833.310 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953833.797 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953834.638 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953835.376 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953835.766 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953836.560 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953837.372 0 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953838.138 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953838.951 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953839.810 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953840.466 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953841.607 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953842.357 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953845.467 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953846.013 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953846.951 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953847.731 0 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953848.732 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953849.825 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953850.482 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953851.263 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953852.169 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953852.950 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953853.725 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953854.482 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953855.265 3 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953856.091 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953857.154 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953857.859 1 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream 1456953858.668 0 10.101.1.50 TCP_HIT/206 402 GET http://bg.v4.a.dl.ws.microsoft.com/dl/content/d/updt/2015/07/096c4bbc-4bc2-4ba1-8fd7-2e8cf3fb1937_132a7d6799d3bd625b0e5b375aa13552593bf0ed.appxbundle? - HIER_NONE/- application/octet-stream Em 02/03/2016 01:06, Amos Jeffries
escreveu:
On 2/03/2016 10:57 a.m., Heiler Bemerguy wrote:Hey guys. For the third time, we got a sudden high bandwidth usage, almost saturating our link, and it won't stop until squid is restarted. I'm totally SURE this inbound traffic comes from squid. It's like it's download stuff itself....Yes, it probably is. Or something very close...Look that after squid was restarted near 10:45, the network usage drops immediately and won't increase as high as before anymore.. This pattern started to happen when I changed from ROCK+AUFS to ROCK+ROCK, squid 3.5.14 x64.Please upgrade to 3.5.15 asap. Or better the latest snapshot if you have trouble with the main release (a few more side effects have been fixed this week).Here's the most important conf settings.. I appreciate all comments about it. /acl windowsupdate dstdomain .ws.microsoft.com .windowsupdate.microsoft.com .update.microsoft.com .windowsupdate.com// //http_access allow windowsupdate// //range_offset_limit none windowsupdate//I suspect you are hitting a case of clients aborting downloads of Win10 files early and Squid continuing to try to complete them. The secret downloads the GWX application does of multi-GB files on a per-machine basis have been quite a problem for several people over the last few months.//cache_mem 4 GB// //maximum_object_size_in_memory 5 MB// //memory_replacement_policy heap GDSF// //cache_replacement_policy heap LFUDA// //maximum_object_size 10 GB// //cpu_affinity_map process_numbers=1,2,3,4,5,6 cores=1,2,3,4,5,6// /*/workers 2/**/ /**/cache_dir rock /cache2/rock1 90000 min-size=0 max-size=32768/**/ /**/cache_dir rock /cache/rock1 300000 min-size=32768 max-size=10737418240/*/ //store_dir_select_algorithm round-robin//Don't force-configure this when you have min/max controlling which dir are usable. Squid default should try to round-robin anyway, but it may select a better best-fit action.//read_ahead_gap 4096 KB// //client_request_buffer_max_size 2048 KB//!!! 2MB packets !?? Please have a read of <http://www.bufferbloat.net/projects/bloat/wiki/Introduction> Ths buffer only needs to store the maximum size of expected HTTP request mime headers on a single request. That is ~64KB for Squid due to hardcoded internal issues. Going far beyond that leads to trouble. Having larger buffer for multipe requests can be a small help with pipelining. BUT you have completely disabled that performance enhancing feature of HTTP in your proxy (the *_persistent_connections off settings below)//dns_v4_first on// //ipcache_size 80000// //fqdncache_size 40000// //memory_pools on// //memory_pools_limit 150 MB// //reload_into_ims on// //connect_retries 3// //cache_swap_low 98// //cache_swap_high 99// //store_avg_object_size 92 KB// //client_idle_pconn_timeout 30 seconds// //client_persistent_connections off// //server_persistent_connections off/ error.log right in this moment:Ayayeye, you got many troubles.08:55:29 kid1| local=10.1.10.9:3080 remote=10.107.0.71:54515 FD 3665 flags=1: read/write failure: (32) Broken pipe 09:00:02 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily unavailable 09:00:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily unavailableUnresolved bug in Squid.09:02:14 kid2| WARNING: Closing client connection due to lifetime timeout 09:02:14 kid2| http://prod.video.msn.com/tenant/amp/entityid/BBq7uZY?blobrefkey=103&$blob=1That would be a single HTTP request+reply transaction that took more than 24hrs (!?) to complete.09:03:34 kid1| WARNING: HTTP: Invalid Response: Bad header encountered from http://sable.madmimi.com/view?id=24371.4971993.01561ff3 e8e7c09ac362ded25f80a76b AKA sable.madmimi.com/view?id=24371.4971993.01561ff3e8e7c09ac362ded25f80a76bOkay. That server is being a bad HTTP citizen. This is just info to help with the client complaints you will probably get about the 4xx/5xx errors contacting that site through Squid. If you want to assist fixing you can report the issue to its admin.09:03:38 kid1| WARNING: Closing client connection due to lifetime timeout 09:03:38 kid1| http://download.windowsupdate.com/d/msdownload/update/software/defu/2016/02/am_delta_patch_1.213.7305.0_59c57a caccbdfa7fa9dd5574f0a7ded60de11963.exeAnother 24hr one ?09:04:15 kid1| WARNING: HTTP: Invalid Response: Bad header encountered from http://sable.madmimi.com/view?id=24371.4931972.b5133065 c861d91790f59bf39ef1abf3 AKA sable.madmimi.com/view?id=24371.4931972.b5133065c861d91790f59bf39ef1abf3 09:04:28 kid2| WARNING: Closing client connection due to lifetime timeout 09:04:28 kid2| http://www.ingressocerto.com/facet-search.json?f=/p-data-Offset:2Getting a lot of these long transactions.09:04:42 kid2| Could not parse headers from on disk objectThis innocent seeming message is related to the CVE-2016-2571 issue. It is a sign that the vulnerability has happened in some past transaction. Squid is handling this part of the fallout though, so whats happened *right now* is okay.09:05:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily unavailable 09:05:02 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily unavailable 09:05:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily unavailable 09:05:18 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see access.log for details.Should be self explanatory. Your proxy appears to be under attack. <http://wiki.squid-cache.org/SquidFaq/SquidLogs#Squid_Error_Messages> <snip many repeats of earier problems>09:25:49 kid1| urlParse: URL too large (8231 bytes) 09:27:24 kid1| urlParse: URL too large (8231 bytes) 09:27:46 kid2| urlParse: URL too large (10742 bytes)These should also be self-explanatory. They are also attack signatures for certain types of buffer-overrun attacks. Squid is coping, but you should really do something forceful to whack the source of these requests over the head. It might be related to the ALERT situation. For example a "GET http://... HTTP/1.1" where the ... is a 8-10 KB long "domain name". <snip more repeats>09:50:28 kid1| Could not parse headers from on disk object 09:50:28 kid1| varyEvaluateMatch: Oops. Not a Vary object on second attempt, 'http://pix04.revsci.net/D08734/a1/0/3/0.js?DM_LOC=%3D http%3A%2F%2Fna.com%3FdlxInitiated%3Dtrue%26nada%3D%26naid%3D2015121611542932923036123812%26namp%3D' 'accept-encoding="gzip,%20deflate,%20sdch "' 09:50:28 kid1| clientProcessHit: Vary object loop!Probably a side effect of the other nasties going on. Though some people do see this happening and it has open bug report(s), we are still trying to get to the bottom of it.09:50:46 kid1| helperHandleRead: unexpected reply on channel 0 from redirector #Hlpr301 'OK' 09:50:46 kid1| helperHandleRead: unexpected reply on channel 0 from redirector #Hlpr301 'OK'** URGENT PROBLEM: ** The redirector helper you are using is broken. It is presenting either multiple-lines for each reply, or replies without being asked about any URL. In both cases Squid will be given wrong instructions to re-write random requests to some other URL for producing the reply. This could be the root cause behind some of those weird long request timeouts or aborted transaction issues. It will *definitely* result in clients randomly being given wrong objects to their replies. So, my advice: * fix the redirector. See what other issues / side effects of that disappear. * if they remain track down what those SECURITY ALERT are about. Get that fixed if you can. I expect the high bandwith will reduce with those two above issues gone and the WU settings altered. You can also further improve things by looking into the too-long URL issues if they remain. Amos _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users |
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
