RE: app to look/view/inspect http traffic...

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



mark...

after a thorough inspection of the headers... (finally!!).. the server
returns a 302 (object moved) along with the new location of the page...

so my question is, how can libwww be used to detect/process this
information. i assumed that the information should somehow be in the
request/response method of the useragent.. but i couldn't seem to get my
hands around how it should perform...

the headers are as follows:

========================================================
http://soc.our.psu.edu/soc/

GET /soc/ HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Cache-Control: max-age=0

HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:22:18 GMT
X-Powered-By: ASP.NET
Connection: close
Content-Type: text/html
Page-Completion-Status: Normal, Normal
----------------------------------------------------------
http://soc.our.psu.edu/soc/socheader.gif

GET /soc/socheader.gif HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
If-Modified-Since: Tue, 02 Oct 2001 20:42:12 GMT
If-None-Match: "70228fb6824bc11:13e4"
Cache-Control: max-age=0

HTTP/1.x 304 Not Modified
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:24:07 GMT
X-Powered-By: ASP.NET
Etag: "70228fb6824bc11:bd9"
Content-Length: 0
----------------------------------------------------------
http://soc.our.psu.edu/soc/main_search.cfm

POST /soc/main_search.cfm HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Content-Type: application/x-www-form-urlencoded
Content-Length: 91
Semester=FALL+2004&CrseLoc=AA%3A%3AAltoona+Campus&CECrseLoc=&CourseAbbrevAnd
Num=&CrseAlpha=
HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:24:13 GMT
X-Powered-By: ASP.NET
Connection: close
Content-Type: text/html
Page-Completion-Status: Normal, Normal
----------------------------------------------------------
http://soc.our.psu.edu/soc/main_search.cfm

POST /soc/main_search.cfm HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/main_search.cfm
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Content-Type: application/x-www-form-urlencoded
Content-Length: 115
Semester=FALL+2004&CrseLoc=AA%3A%3AAltoona+Campus&CECrseLoc=&CourseAbbrevAnd
Num=&CrseAlpha=a-c&Search=View+schedule
HTTP/1.x 302 Object Moved
Location:
act_main_search.cfm?Semester=FALL%202004&CrseLoc=AA%3A%3AAltoona%20Campus&CE
CrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum=0
Server: Microsoft-IIS/5.0
Content-Type: text/html
Content-Length: 270
----------------------------------------------------------
http://soc.our.psu.edu/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=
AA%3A%3AAltoona%20Campus&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum
=0

GET
/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=AA%3A%3AAltoona%20Camp
us&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum=0 HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/main_search.cfm
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
========================================================

++++>>>>>>>>HTTP/1.x 302 Object Moved   <<<<<<+++++++

thanks...

-bruce


tom...

the following url works in a browser window. you can select it, and the
appropriate page appears...

http://soc.our.psu.edu/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=
AA%3A%3AAltoona%20Campus&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum
=0

what i'm trying to get a better understanding of, is why when this is used
with libwww/wget/etc.. the correct page is not returned.. i've disabled
cookies on my test IE browser so it shouldn't be a cookie issue..

(and yeah.. you were correct when you pointed out that all this really does
is redirect to a basic tree structure!!! which kind of makes this effort
pointless.. but i need to get a better understanding as to what the
problem/issue is that i'm missing...)

if anybody has any insight, i'd appreciate it!

i took a quick look at the headers using windump... (i've cut out alot of
the garbage to give a feel as to what appears to be happening...)

flags [DF], length: 691, bad cksum 0 (->a17c)!) sys2.mesa.com.2327 >
soc.our.psu.edu.80: P 366:1017(651) ack 165 win 17356
	0x0000:  4500 02b3 6ad1 4000 8006 0000 c0a8 0103  E...j.@.........
	0x0010:  92ba 9791 0917 0050 4c75 f500 da9b f421  .......PLu.....!
	0x0020:  5018 43cc ee9c 0000 504f 5354 202f 736f  P.C.....POST./so
	0x0030:  632f 6d61 696e 5f73 6561 7263 682e 6366  c/main_search.cf
	0x0040:  6d20 4854 5450 2f31 2e31 0d0a 4163 6365  m.HTTP/1.1..Acce
	0x0050:  7074                                     pt
   .
   .
   .
	0x0000:  4500 0028 6ad5 4000 8006 0000 c0a8 0103  E..(j.@.........
	0x0010:  92ba 9791 0917 0050 4c75 f78b da9b f491  .......PLu......
	0x0020:  5010 435c ec11 0000                      P.C\....
12:21:17.747572 IP (tos 0x0, ttl 128, id 27351, offset 0, flags [DF],
length: 617, bad cksum 0 (->a1c0)!) sys2.mesa.com.2327 > soc.our.psu.edu.80:
P 1017:1594(577) ack 788 win 16733
	0x0000:  4500 0269 6ad7 4000 8006 0000 c0a8 0103  E..ij.@.........
	0x0010:  92ba 9791 0917 0050 4c75 f78b da9b f690  .......PLu......
	0x0020:  5018 415d ee52 0000 4745 5420 2f73 6f63  P.A].R..GET./soc
	0x0030:  2f61 6374 5f6d 6169 6e5f 7365 6172 6368  /act_main_search
	0x0040:  2e63 666d 3f53 656d 6573 7465 723d 4641  .cfm?Semester=FA
	0x0050:  4c4c                                     LL
12:21:17.747619 IP (tos 0x0, ttl 128, id 27351, offset 0, flags [DF],
length: 617, bad cksum 0 (->a1c0)!) sys2.mesa.com.2327 > soc.our.psu.edu.80:
P 1017:1594(577) ack 788 win 16733
	0x0000:  4500 0269 6ad7 4000 8006 0000 c0a8 0103  E..ij.@.........
	0x0010:  92ba 9791 0917 0050 4c75 f78b da9b f690  .......PLu......
	0x0020:  5018 415d ee52 0000 4745 5420 2f73 6f63  P.A].R..GET./soc
	0x0030:  2f61 6374 5f6d 6169 6e5f 7365 6172 6368  /act_main_search
	0x0040:  2e63 666d 3f53 656d 6573 7465 723d 4641  .cfm?Semester=FA
	0x0050:  4c4c                                     LL
   .
   .
   .
12:21:18.270385 IP (tos 0x0, ttl 128, id 27360, offset 0, flags [DF],
length: 40, bad cksum 0 (->a3f8)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
. [bad tcp cksum ec11 (->e378)!] 0:0(0) ack 1 win 17520
	0x0000:  4500 0028 6ae0 4000 8006 0000 c0a8 0103  E..(j.@.........
	0x0010:  92ba 9791 091a 0050 4c95 8117 dab9 ea23  .......PL......#
	0x0020:  5010 4470 ec11 0000                      P.Dp....
12:21:18.272296 IP (tos 0x0, ttl 128, id 27361, offset 0, flags [DF],
length: 522, bad cksum 0 (->a215)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
P 0:482(482) ack 1 win 17520
	0x0000:  4500 020a 6ae1 4000 8006 0000 c0a8 0103  E...j.@.........
	0x0010:  92ba 9791 091a 0050 4c95 8117 dab9 ea23  .......PL......#
	0x0020:  5018 4470 edf3 0000 4745 5420 2f73 6f63  P.Dp....GET./soc
	0x0030:  2f66 616c 6c2f 6161 2f61 2d63 2f69 6e64  /fall/aa/a-c/ind
	0x0040:  6578 2e68 746d 6c20 4854 5450 2f31 2e31  ex.html.HTTP/1.1
	0x0050:  0d0a                                     ..
12:21:18.272342 IP (tos 0x0, ttl 128, id 27361, offset 0, flags [DF],
length: 522, bad cksum 0 (->a215)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
P 0:482(482) ack 1 win 17520
	0x0000:  4500 020a 6ae1 4000 8006 0000 c0a8 0103  E...j.@.........
	0x0010:  92ba 9791 091a 0050 4c95 8117 dab9 ea23  .......PL......#
	0x0020:  5018 4470 edf3 0000 4745 5420 2f73 6f63  P.Dp....GET./soc
	0x0030:  2f66 616c 6c2f 6161 2f61 2d63 2f69 6e64  /fall/aa/a-c/ind
	0x0040:  6578 2e68 746d 6c20 4854 5450 2f31 2e31  ex.html.HTTP/1.1
	0x0050:  0d0a                                     ..

as you can see... the client initially sends a "..main_search.cfm.. which is
redirected towards ..act_main_search.cfm...to finally
.../fall/aa/a-c/index.html.."

the overall issue i have is how to set this up using libwww/mechanize/etc...

thanks...

-bruce


-----Original Message-----
From: perl-win32-users-bounces@xxxxxxxxxxxxxxxxxxxxxxxx
[mailto:perl-win32-users-bounces@xxxxxxxxxxxxxxxxxxxxxxxx]On Behalf Of
Thomas, Mark - BLS CTR
Sent: Wednesday, June 09, 2004 11:25 AM
To: 'bedouglas@xxxxxxxxxxxxx'; 'General Red Hat Linux discussion list';
perl-win32-users@xxxxxxxxxxxxxxxxxxxxxxxx
Subject: RE: app to look/view/inspect http traffic...


Bruce,

I suggest looking at the 'Live HTTP headers' plugin for Mozilla/Firefox. It
may be just what you're looking for. It allows you to see the HTTP headers
in real time, and lets you edit request headers.

The extension is one of many available here:
http://extensionroom.mozdev.org/




--
Mark Thomas                    Thomas.Mark@xxxxxxx
Internet Systems Architect     DigitalNet, Inc.

$_=q;KvtuyboopuifeyQQfeemyibdlfee;; y.e.s. ;y+B-x+A-w+s; ;y;y; ;;print;;




_______________________________________________
Perl-Win32-Users mailing list
Perl-Win32-Users@xxxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs


-- 
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

[Index of Archives]     [CentOS]     [Kernel Development]     [PAM]     [Fedora Users]     [Red Hat Development]     [Big List of Linux Books]     [Linux Admin]     [Gimp]     [Asterisk PBX]     [Yosemite News]     [Red Hat Crash Utility]


  Powered by Linux