Thanks Amo for your fast reply, I meant it ignores the POST requests, I can access the requested page normally but with no post parameters. So, If I want to navigate through the website using POST, I can't and I get only the first page. Actually I didn't change any configuration !! Squid Version and Configuration options: Squid Cache: Version 3.0.STABLE16 configure options: '--build=i386-redhat-linux-gnu' '--host=i386-redhat-linux-gnu' '--target=i386-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-digest-auth-helpers=password' '--enable-epoll' '--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group' '--enable-icap-client' '--enable-ident-lookups' '--with-large-files' '--enable-linux-netfilter' '--enable-ntlm-auth-helpers=SMB,fakeauth' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,null,ufs' '--enable-useragent-log' '--enable-wccpv2' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl=/usr/kerberos' '--with-pthreads' 'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu' 'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe -fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' 'LDFLAGS=-pie' 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' 'FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' But I don't know how to get the HTTP headers of the traffic between squid and the spider. Can squid log such details ? Finally, the spider is trying to access this website, navigate on it using post requests and parse the data. When I use the spider without squid, it works like a charm, once I use squid, the post problem occurs. Thanks again On Tue, May 29, 2012 at 2:29 AM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote: > On 29.05.2012 07:47, Mahmoud Abdel-Fattah wrote: >> >> Hello, >> >> I'm totally new to squid and using it as a proxy for a scrapy (python >> spider ). it works great with all get requests, but it doesn't work >> with large post requests like the requests is generated by ASP.net in >> the following site : http://www.e3050.com/ >> >> So, is there any configuration should I adjust to enable / enlarge >> post requests ? > > > Maybe yes, Maybe no. The answer to this question depends on your definition > of "doesn't work". > > What kind of limitation did you configure to prevent POST working like they > do out of the box? > > We need to know specifics about what is going wrong... > > * software version numbers. > -> check against the latest current release, if its old try with the > current one(s). > > * HTTP headers of the traffic between Squid and the spider if you can get > them. > -> what exactly is the spider trying to do? and how? > --> what does the site do? how is the spider reacting to the response? > > Amos >
