On 29/05/2012 13:33, Mahmoud Abdel-Fattah wrote:
Thanks Amo for your fast reply,
I meant it ignores the POST requests, I can access the requested page
normally but with no post parameters. So, If I want to navigate
through the website using POST, I can't and I get only the first page.
Actually I didn't change any configuration !!
Squid Version and Configuration options:
Squid Cache: Version 3.0.STABLE16
configure options: '--build=i386-redhat-linux-gnu'
'--host=i386-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--includedir=/usr/include' '--libdir=/usr/lib'
'--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--exec_prefix=/usr' '--bindir=/usr/sbin'
'--libexecdir=/usr/lib/squid' '--localstatedir=/var'
'--datadir=/usr/share' '--sysconfdir=/etc/squid'
'--disable-dependency-tracking' '--enable-arp-acl'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
'--enable-delay-pools' '--enable-digest-auth-helpers=password'
'--enable-epoll'
'--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group'
'--enable-icap-client' '--enable-ident-lookups' '--with-large-files'
'--enable-linux-netfilter' '--enable-ntlm-auth-helpers=SMB,fakeauth'
'--enable-referer-log' '--enable-removal-policies=heap,lru'
'--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,null,ufs'
'--enable-useragent-log' '--enable-wccpv2' '--with-aio'
'--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
'--with-openssl=/usr/kerberos' '--with-pthreads'
'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu'
'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
-fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
-mtune=generic -fasynchronous-unwind-tables' 'LDFLAGS=-pie'
'CXXFLAGS=-fPIE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32
-march=i386 -mtune=generic -fasynchronous-unwind-tables' 'FFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables'
But I don't know how to get the HTTP headers of the traffic between
squid and the spider. Can squid log such details ?
Finally, the spider is trying to access this website, navigate on it
using post requests and parse the data. When I use the spider without
squid, it works like a charm, once I use squid, the post problem
occurs.
Thanks again
you can use tcpdump on the squidbox to capture all the traffic between
the proxy and the client but you will need to put some patterns as for
the client name\ip and maybe destination ip\name.
you can use the POST method only for pages that are built for that.
what do you get in your access.log when the spider tries to access the
page with post?
Eliezer
On Tue, May 29, 2012 at 2:29 AM, Amos Jeffries<squid3@xxxxxxxxxxxxx> wrote:
On 29.05.2012 07:47, Mahmoud Abdel-Fattah wrote:
Hello,
I'm totally new to squid and using it as a proxy for a scrapy (python
spider ). it works great with all get requests, but it doesn't work
with large post requests like the requests is generated by ASP.net in
the following site : http://www.e3050.com/
So, is there any configuration should I adjust to enable / enlarge
post requests ?
Maybe yes, Maybe no. The answer to this question depends on your definition
of "doesn't work".
What kind of limitation did you configure to prevent POST working like they
do out of the box?
We need to know specifics about what is going wrong...
* software version numbers.
-> check against the latest current release, if its old try with the
current one(s).
* HTTP headers of the traffic between Squid and the spider if you can get
them.
-> what exactly is the spider trying to do? and how?
--> what does the site do? how is the spider reacting to the response?
Amos
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il
