Hello We've been struggling for a few days with a permanent problem on a newly installed squid 3.1.4 and those web form-based uploads, either using ASP, javascript or any other language behind. Let me assure you guys, ALL uploads are failing, not with a few specific sites. It is just a matter of clicking an OK button to submit the file and the browser (IE or Firefox) instantly shows either its own error page (Page could not be opened) in 90% of the tries or squid's error page (Connection Reset by Peer) in the remaining 10%. By configuring a remote client to use the proxy server through an external SSH tunnel (i.e. by excluding all the local network devices), we can reduce the error ratio to around 5% of the tries. So, when the upload works, it shows this: 1281099317.664 409638 127.0.0.1 TCP_MISS/200 1840 POST http://discovirtual.terra.com.br/vd.cgi administrator DIRECT/200.154.56.65 text/html When it doesn't, it shows this: 1281102595.774 21086 127.0.0.1 TCP_MISS/000 0 POST http://discovirtual.terra.com.br/vd.cgi administrator DIRECT/200.154.56.65 - Plus, cache.log has a lot of these messages which I don't understand: 2010/08/06 08:55:53| ctx: enter level 2: 'http://p2.trrsf.com.br/image/get?o=cf&w=296&h=222&src=http://sdp.terra.com.br/Thumbox/free/cnt314458_h300_aNoChange_Prison-Break-1-Temporada-Ep-12_20108611342.jpg' 2010/08/06 10:13:35| ctx: enter level 3: 'application/vnd.google.safebrowsing-chunk' 2010/08/06 10:13:35| ctx: enter level 4: 'application/vnd.google.safebrowsing-chunk' 2010/08/06 10:23:54.948| ctx: enter level 5: 'http://discovirtual.terra.com.br/vdexecup.html' 2010/08/06 10:23:54.948| ctx: exit level 5 2010/08/06 10:23:54.949| ctx: enter level 5: 'http://discovirtual.terra.com.br/vdexecup.html' 2010/08/06 10:23:54.949| ctx: exit level 5 2010/08/06 10:23:54.951| ctx: enter level 5: 'http://discovirtual.terra.com.br/vdexecup.html' 2010/08/06 10:23:54.951| ctx: exit level 5 2010/08/06 10:23:55.810| ctx: enter level 5: 'http://uv.terra.com.br/UV?c=discovirtual&ord=907532&REF=&SCRNSZ=1440x900&BRSRSZ=924x695&TIMEZONE=Fri%20Aug%206%2010%3A23%3A41%20UTC-0300%202010' 2010/08/06 10:23:55.810| ctx: exit level 5 2010/08/06 10:23:55.812| ctx: enter level 5: 'http://uv.terra.com.br/UV?c=discovirtual&ord=907532&REF=&SCRNSZ=1440x900&BRSRSZ=924x695&TIMEZONE=Fri%20Aug%206%2010%3A23%3A41%20UTC-0300%202010' 2010/08/06 10:23:55.812| ctx: exit level 5 2010/08/06 10:23:55.814| ctx: enter level 5: 'http://uv.terra.com.br/UV?c=discovirtual&ord=907532&REF=&SCRNSZ=1440x900&BRSRSZ=924x695&TIMEZONE=Fri%20Aug%206%2010%3A23%3A41%20UTC-0300%202010' 2010/08/06 10:23:55.814| ctx: exit level 5 2010/08/06 10:23:56.623| ctx: enter level 5: 'http://br.hits.e.cl/cert/hit.dll?sitio_id=300274&path=srv/enviando_arquivo&referer=&java=true&flash=10&cert_cachebuster=1034&descr=ENVIANDO' 2010/08/06 10:23:56.623| ctx: exit level 5 2010/08/06 10:23:56.625| ctx: enter level 5: 'http://br.hits.e.cl/cert/hit.dll?sitio_id=300274&path=srv/enviando_arquivo&referer=&java=true&flash=10&cert_cachebuster=1034&descr=ENVIANDO' 2010/08/06 10:23:56.625| ctx: exit level 5 2010/08/06 10:23:56.627| ctx: enter level 5: 'http://br.hits.e.cl/cert/hit.dll?sitio_id=300274&path=srv/enviando_arquivo&referer=&java=true&flash=10&cert_cachebuster=1034&descr=ENVIANDO' 2010/08/06 10:23:56.627| ctx: exit level 5 2010/08/06 10:23:56.807| ctx: enter level 5: 'http://terra.112.2o7.net/b/ss/terrabr/1/H.20.3/s03451603567267?AQB=1&ndh=1&t=6/7/2010%2010%3A23%3A41%205%20180&ce=ISO-8859-1&ns=terra&cdp=3&pageName=discovirtual.br/vdexecup.html&g=http%3A//discovirtual.terra.com.br/vdexecup.html&cc=USD&ch=br.produtoseservicos.discovirtual%7C300274&server=discovirtual.terra.com.br&events=event1&v1=br.produtoseservicos.discovirtual%7C300274&h1=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c2=productos_y_servicios&v2=srv&h2=terra%7Cproductos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c3=productos_y_servicios%7Cdiscovirtual&v3=300274&c4=300274&v4=discovirtual.br/vdexecup.html&c5=srv&c6=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual&c7=discovirtual&v7=terra&c8=discovirtual&c9=300274%7C%7Csrv&c10=discovirtual&c11=enviando_arquivo&c12=300274%7C&c15=terra&c16=productos_y_servicios%7Cdiscovirtual%7Cterra&v19=More%20than%2030%20days&c20=http%3A//stf.terra.com.br/metrics/inc/br/201007300000b.js&v20=New&v44=D%3DpageName&v45=D%3Dg&c47=New&c48=flash%2010&c49=silverlight%203.0&c50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&v50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&s=1440x900&c=32&j=1.5&v=Y&k=Y&bw=945&bh=699&ct=lan&hp=N&pid=discovirtual.br/vd.cgi&pidt=1&oid=functiononclick%28%29%7Bjan%3Djan.open%28%27./vdexecup.html%27%2C%27Uploading%27%2C%27toolbar%3D0%2Clocation%3D0%2Cdirectories%3D0%2Cmen&oidt=2&ot=IMAGE&oi=125&AQE=1' 2010/08/06 10:23:56.807| ctx: exit level 5 2010/08/06 10:23:56.810| ctx: enter level 5: 'http://terra.112.2o7.net/b/ss/terrabr/1/H.20.3/s03451603567267?AQB=1&ndh=1&t=6/7/2010%2010%3A23%3A41%205%20180&ce=ISO-8859-1&ns=terra&cdp=3&pageName=discovirtual.br/vdexecup.html&g=http%3A//discovirtual.terra.com.br/vdexecup.html&cc=USD&ch=br.produtoseservicos.discovirtual%7C300274&server=discovirtual.terra.com.br&events=event1&v1=br.produtoseservicos.discovirtual%7C300274&h1=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c2=productos_y_servicios&v2=srv&h2=terra%7Cproductos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c3=productos_y_servicios%7Cdiscovirtual&v3=300274&c4=300274&v4=discovirtual.br/vdexecup.html&c5=srv&c6=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual&c7=discovirtual&v7=terra&c8=discovirtual&c9=300274%7C%7Csrv&c10=discovirtual&c11=enviando_arquivo&c12=300274%7C&c15=terra&c16=productos_y_servicios%7Cdiscovirtual%7Cterra&v19=More%20than%2030%20days&c20=http%3A//stf.terra.com.br/metrics/inc/br/201007300000b.js&v20=New&v44=D%3DpageName&v45=D%3Dg&c47=New&c48=flash%2010&c49=silverlight%203.0&c50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&v50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&s=1440x900&c=32&j=1.5&v=Y&k=Y&bw=945&bh=699&ct=lan&hp=N&pid=discovirtual.br/vd.cgi&pidt=1&oid=functiononclick%28%29%7Bjan%3Djan.open%28%27./vdexecup.html%27%2C%27Uploading%27%2C%27toolbar%3D0%2Clocation%3D0%2Cdirectories%3D0%2Cmen&oidt=2&ot=IMAGE&oi=125&AQE=1' 2010/08/06 10:23:56.810| ctx: exit level 5 2010/08/06 10:23:56.812| ctx: enter level 5: 'http://terra.112.2o7.net/b/ss/terrabr/1/H.20.3/s03451603567267?AQB=1&ndh=1&t=6/7/2010%2010%3A23%3A41%205%20180&ce=ISO-8859-1&ns=terra&cdp=3&pageName=discovirtual.br/vdexecup.html&g=http%3A//discovirtual.terra.com.br/vdexecup.html&cc=USD&ch=br.produtoseservicos.discovirtual%7C300274&server=discovirtual.terra.com.br&events=event1&v1=br.produtoseservicos.discovirtual%7C300274&h1=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c2=productos_y_servicios&v2=srv&h2=terra%7Cproductos_y_servicios%7Cdiscovirtual%7Cdiscovirtual%7C300274%7Csrv&c3=productos_y_servicios%7Cdiscovirtual&v3=300274&c4=300274&v4=discovirtual.br/vdexecup.html&c5=srv&c6=productos_y_servicios%7Cdiscovirtual%7Cdiscovirtual&c7=discovirtual&v7=terra&c8=discovirtual&c9=300274%7C%7Csrv&c10=discovirtual&c11=enviando_arquivo&c12=300274%7C&c15=terra&c16=productos_y_servicios%7Cdiscovirtual%7Cterra&v19=More%20than%2030%20days&c20=http%3A//stf.terra.com.br/metrics/inc/br/201007300000b.js&v20=New&v44=D%3DpageName&v45=D%3Dg&c47=New&c48=flash%2010&c49=silverlight%203.0&c50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&v50=Weekday%20%3A%20Friday%20%3A%2010%3A00AM&s=1440x900&c=32&j=1.5&v=Y&k=Y&bw=945&bh=699&ct=lan&hp=N&pid=discovirtual.br/vd.cgi&pidt=1&oid=functiononclick%28%29%7Bjan%3Djan.open%28%27./vdexecup.html%27%2C%27Uploading%27%2C%27toolbar%3D0%2Clocation%3D0%2Cdirectories%3D0%2Cmen&oidt=2&ot=IMAGE&oi=125&AQE=1' 2010/08/06 10:23:56.812| ctx: exit level 5 2010/08/06 10:23:57.892| ctx: enter level 5: 'http://b.scorecardresearch.com/p?c1=2&c2=6906546&c3=&c4=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c5=&c6=&c7=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c8=&c9=&c10=CERT&c15=' 2010/08/06 10:23:57.892| ctx: exit level 5 2010/08/06 10:23:57.894| ctx: enter level 5: 'http://b.scorecardresearch.com/p?c1=2&c2=6906546&c3=&c4=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c5=&c6=&c7=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c8=&c9=&c10=CERT&c15=' 2010/08/06 10:23:57.894| ctx: exit level 5 2010/08/06 10:23:57.896| ctx: enter level 5: 'http://b.scorecardresearch.com/p?c1=2&c2=6906546&c3=&c4=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c5=&c6=&c7=http%3A%2F%2Fdiscovirtual.terra.com.br%2Fvdexecup.html&c8=&c9=&c10=CERT&c15=' 2010/08/06 10:23:57.896| ctx: exit level 5 2010/08/06 10:24:10.281| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/apu/apu-0607g.xml.dif' 2010/08/06 10:24:10.281| ctx: exit level 5 2010/08/06 10:24:10.282| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/apu/apu-0607g.xml.dif' 2010/08/06 10:24:10.283| ctx: exit level 5 2010/08/06 10:24:11.957| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/as/pas/pas-0607g.xml.dif' 2010/08/06 10:24:11.957| ctx: exit level 5 2010/08/06 10:24:11.959| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/as/pas/pas-0607g.xml.dif' 2010/08/06 10:24:11.959| ctx: exit level 5 2010/08/06 10:24:12.866| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/av/emu/emu-0607g.xml.dif' 2010/08/06 10:24:12.866| ctx: exit level 5 2010/08/06 10:24:12.867| ctx: enter level 5: 'http://dnl-14.geo.kaspersky.com/bases/av/emu/emu-0607g.xml.dif' 2010/08/06 10:24:12.867| ctx: exit level 5 Additional info: * CentOS release 5.5 (Final), 32 bit * squid3-3.1.4-1.el5.pp.i386.rpm (from http://www.pramberger.at/peter/services/repository/) * No more than 5 simultaneous users * Intel Core 2 Duo E7600, 4 GB RAM, Intel DG31PR motherboard * Direct connections, without squid, always work. * Resolv.conf points to 127.0.0.1, which is bind-9.3.6-4.P1.el5_4.2 * Tried with and without "half_closed_clients off". * Already deleted and recreated /var/cache/squid. * One of the cache.log files seem to be truncated or with binary characters preventing it to be properly read from the console. * Found two occurrences of "Exception error:found data bewteen chunk end and CRLF" in cache.log. My guesses are: - It could be a hardware problem with the server specifically related to faulty NIC, I/O or bad memory, but there are no system wide errors being logged which would support this and all other server applications are working fine; - It could be a hardware problem with the wan circuit or provider, but without the proxy server, going directly to Internet, the problem never happens. - It could be a DNS problem. Unlikely, since the problem is only relates to upload (POST) operations to the same websites which were already resolved by its own named. - It coud be a DoS launched from an internal infected workstation. Unlikely, squid is not crashing and server load stays at 0.00. - It could be a squid bug or problem in face of an unknown condition? Unlikely, we have the same software setup (O.S., the same rpm and config of squid 3.1.4) in another remote office which works perfectly with these same upload websites. - It could be a problem with all the upload websites tried? REALLY unlikely. So I would like to kindly ask for any suggestions on diagnostics and troubleshooting of this problem. -------- squid.conf half_closed_clients off range_offset_limit -1 maximum_object_size 200 MB quick_abort_min -1 auth_param basic program /usr/libexec/squid/squid_radius_auth -f /etc/squid/radius.conf auth_param basic children 5 auth_param basic realm Web-Proxy auth_param basic credentialsttl 5 minute auth_param basic casesensitive off acl manager proto cache_object acl localhost src 127.0.0.1/32 acl localhost src ::1/128 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 acl to_localhost dst ::1/128 acl localnet src 192.168.2.0/24 acl localnet src fc00::/7 acl localnet src fe80::/10 acl windowsupdate dstdomain windowsupdate.microsoft.com acl windowsupdate dstdomain .update.microsoft.com acl windowsupdate dstdomain download.windowsupdate.com acl windowsupdate dstdomain redir.metaservices.microsoft.com acl windowsupdate dstdomain images.metaservices.microsoft.com acl windowsupdate dstdomain c.microsoft.com acl windowsupdate dstdomain www.download.windowsupdate.com acl windowsupdate dstdomain wustat.windows.com acl windowsupdate dstdomain crl.microsoft.com acl windowsupdate dstdomain sls.microsoft.com acl windowsupdate dstdomain productactivation.one.microsoft.com acl windowsupdate dstdomain ntservicepack.microsoft.com acl StreamingRequest1 req_mime_type -i ^video/x-ms-asf$ acl StreamingRequest2 req_mime_type -i ^application/vnd.ms.wms-hdr.asfv1$ acl StreamingRequest3 req_mime_type -i ^application/x-mms-framed$ acl StreamingRequest4 req_mime_type -i ^audio/x-pn-realaudio$ acl StreamingReply1 rep_mime_type -i ^video/x-ms-asf$ acl StreamingReply2 rep_mime_type -i ^application/vnd.ms.wms-hdr.asfv1$ acl StreamingReply3 rep_mime_type -i ^application/x-mms-framed$ acl StreamingReply4 rep_mime_type -i ^audio/x-pn-realaudio$ acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 acl CONNECT method CONNECT acl wuCONNECT dstdomain www.update.microsoft.com acl wuCONNECT dstdomain sls.microsoft.com acl localsites dstdomain .company-xxx.local acl sites_liberados url_regex -i "/etc/squid/sites.liberados" acl radius_users proxy_auth REQUIRED always_direct allow localsites cache deny nocache http_access allow manager localhost http_access allow CONNECT wuCONNECT localnet http_access allow CONNECT wuCONNECT localhost http_access allow windowsupdate localnet http_access allow windowsupdate localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny StreamingRequest1 all http_access deny StreamingRequest2 all http_access deny StreamingRequest3 all http_access deny StreamingRequest4 all http_reply_access deny StreamingReply1 all http_reply_access deny StreamingReply2 all http_reply_access deny StreamingReply3 all http_reply_access deny StreamingReply4 all http_access allow sites_liberados http_access allow radius_users http_access deny all http_port 3128 hierarchy_stoplist cgi-bin ? cache_dir ufs /var/cache/squid 2500 16 256 coredump_dir /tmp refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 Rodrigo Ferraz CONCEPT NETSERVICES www.conceptnetservices.com Qualidade. Do início ao fim. +55 (51) 3355-8000 http://twitter.com/ConceptNetBR
