Hi, i am using Squid as a reverse proxy. Squid tested is 3.5.20 and 3.5.28 (same result). OS of Squid box is Centos7 latest. The backend is some IIS-server with a software "Esko Webcenter" (used in printing industry). Everything is working, except large HTTP-POST uploads stall and time out. Small HTTP-POST uploads work fine. The amount of "large" or "small" varies and seems to be somehow dependent on connection speed. For some tests with fast connections, the limit seems to be around 800 MB, for a slower connection speed more like 50MB. The exact amount of data transfered is not consistent, each test shows a different value. The software wants to upload up to 3GB. The upload client is some Javascript/Ajax bloat, that shows a progress bar. That progess bar stalls at some point and then the Javascript client shows some generic error message. Everything works fine without Squid reverse proxy in between. Everything works fine with Apache as reverse proxy! In all tests, a simple firewall is in the path, that shows no error. No IPS, DPI or similar is present. No Selinux on Squid box. When the upload stalls, i am seeing the receive queue grow on the Squid box OS: netstat -anp Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name [...] tcp6 1443336 0 217.xx.xx.18:2443 77.7.88.123:43252 ESTABLISHED 10512/(squid-1) [...] At this point wireshark shows the TCP window at the client shrink to zero and after some time a tcp teardown. So squid seems to not read any more from the client tcp stream! Why? In cache.log i get no error. In access.log i get the following (each line one test): 1548315375.580 147346 77.7.88.123 TCP_MISS/502 5200 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html 1548315515.773 140151 77.7.88.123 TCP_MISS/502 5200 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html 1548316276.926 154505 77.7.88.123 TCP_MISS/502 5200 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html 1548318410.172 137555 77.7.88.123 TCP_MISS/502 5200 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html 1548318550.384 140082 77.7.88.123 TCP_MISS/502 5200 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html 1548352767.439 141549 77.7.88.123 TCP_MISS/502 5122 POST https://webcenter.example.com/WebCenter/asyncupload.wcr?action=DO_UPLOAD - FIRSTUP_PARENT/10.yy.yy.21 text/html Using some of Squids debug function, the HTTP-POST request of a testfile upload with size 268435456 bytes seems to look like this: POST /WebCenter/asyncupload.wcr?action=DO_UPLOAD HTTP/1.1^M Host: webcenter.example.com^M Connection: keep-alive^M Content-Length: 268436133^M Accept: application/json, text/javascript, */*; q=0.01^M Origin: https://webcenter.example.com^M X-Requested-With: XMLHttpRequest^M User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36^M DNT: 1^M Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryoYIJ0e8lVMdsNIh4^M Referer: https://webcenter.example.com/WebCenter/projdetailsdocsadddocapplet.jsp?projectID=00002_0000028952&folderID=00002_0000363216&menu_file=addnewdocument^M Accept-Encoding: gzip, deflate, br^M Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7^M Cookie: JSESSIONID=3B3B60B9E2EF7F74700AF4877C17F6A9C43E76DEF8F219839EEC3779B310124C15EEE4CF57CB8437E6FD5FFE8D76EF5BE79D2304D1023C4F6A539D0F619A245D; /WebCenter/projdetails.jsp=; /WebCenter/projdetailsdocs.jsp=; ASPSESSIONIDQERDBCDC=OPHNNBAADGNBKMBPGGKAFIOI^M Squid forwards it to the backend server like this: POST /WebCenter/asyncupload.wcr?action=DO_UPLOAD HTTP/1.1^M Content-Length: 268436133^M Accept: application/json, text/javascript, */*; q=0.01^M Origin: https://webcenter.example.com^M X-Requested-With: XMLHttpRequest^M User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36^M DNT: 1^M Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryoYIJ0e8lVMdsNIh4^M Referer: https://webcenter.example.com/WebCenter/projdetailsdocsadddocapplet.jsp?projectID=00002_0000028952&folderID=00002_0000363216&menu_file=addnewdocument^M Accept-Encoding: gzip, deflate, br^M Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7^M Cookie: JSESSIONID=3B3B60B9E2EF7F74700AF4877C17F6A9C43E76DEF8F219839EEC3779B310124C15EEE4CF57CB8437E6FD5FFE8D76EF5BE79D2304D1023C4F6A539D0F619A245D; /WebCenter/projdetails.jsp=; /WebCenter/projdetailsdocs.jsp=; ASPSESSIONIDQERDBCDC=OPHNNBAADGNBKMBPGGKAFIOI^M Host: webcenter.example.com^M Via: 1.1 HFKGN062.example.com (squid/3.5.28)^M Surrogate-Capability: HFKGN062.example.com="Surrogate/1.0 ESI/1.0"^M X-Forwarded-For: 77.7.88.123^M Cache-Control: max-age=259200^M Connection: keep-alive^M Front-End-Https: On^M ^M In the debug logs i find the following error message: X-Squid-Error : ERR_READ_ERROR 104 I tried to change various kernel parameters like tcp_rmem, rmem_max, etc. Results in no change to the problem. Any ideas what is causing this? How can i further track down the problem? Is anybody on this list able to do HTTP-POST with gigabyte sized files through squid? TIA Matthias P.S. here is my squid.conf: acl SSL_ports port 443 acl CONNECT method CONNECT acl trace method TRACE acl webcenter dstdomain webcenter.example.com acl my443 myport 443 acl my2443 myport 2443 http_access deny trace http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow webcenter http_access allow localhost http_access deny all icp_access allow all http_port 80 accel https_port 2443 accel vport=443 cert=/etc/squid/certs/webcenter.crt.pem key=/etc/squid/certs/webcenter.key.pem options=NO_SSLv2,NO_SSLv3 ssl_unclean_shutdown on cache_peer 10.yy.yy.21 parent 443 0 originserver no-digest front-end-https=on no-query ssl sslflags=DONT_VERIFY_PEER ssloptions=NO_SSLv2,NO_SSLv3 login=PASS cache_peer_access 10.yy.yy.21 allow webcenter cache_peer_access 10.yy.yy.21 deny all cache_mem 250 MB maximum_object_size_in_memory 1 MB cache_dir ufs /var/spool/squid 10000 48 256 maximum_object_size 40960 KB access_log /var/log/squid/access.log squid cache_store_log none debug_options ALL,1 strip_query_terms off redirect_program /etc/squid/urlmapper.pl redirect_children 10 redirect_rewrites_host_header on acl QUERY urlpath_regex cgi-bin \? cache deny QUERY refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 request_header_max_size 200 KB reply_header_max_size 200 KB request_body_max_size 3 GB peer_connect_timeout 10 seconds request_timeout 5 minutes persistent_request_timeout 5 minutes half_closed_clients on pconn_timeout 10 minute shutdown_lifetime 60 seconds cache_mgr hotline@xxxxxxxxxxx detect_broken_pconn on deny_info TCP_RESET trace always_direct deny all never_direct allow all hosts_file /etc/hosts coredump_dir /var/spool/squid _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
