a loop constructing the URLs and make PHP to fetch up to 10 thousand pages

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



hello dear list - good morning!


I am trying to figure out a method how to retrieve on the same URLs [see  below] with different query arguments, and i am wondering if this is doable with PHP!? 

on a sidenote:  Well - i guess that we can do this with  LWP::UserAgent -  guess that this provides a way for us to loop through the query arguments: I am not sure that LWP::UserAgent has a method for us to do that. I tried to figure it out. And i digged deeper in the Manpages and Howtos. we can have a loop constructing the URLs and use 
LWP::UserAgent repeatedly:


see the Code:
 
for my $id (0 .. 100000)

{

 $ua->get($url."?id=21&extern_eid=".(0-$id))

 //rest of the code

} 



Well, alternatively we can add a request_prepare handler that  computes and add the query arguments before we send out the request. Do you think that this fits the needs? 

But wait - i want to do this with PHP! 

What is aimed: Here on this following site we find a list of many  schools: [see the page with the subsequent results - approx more than 1000 sites] 

see this site:  http://www-db.sn.schule.de/index.php?id=25

i want to fetch the sites that are listet on this page - and therefore i  want to use PHP for this [job] - and subesquently
 i want to parse them.

the sites can be reached directly - by constructing in other words the subsites of the overview can be reached via direct
 links... see the following. 


http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1543

http://www-db.sn.schule.de/index.php?]id=21&extern_eid=709

http://www-db.sn.schule.de/index.php?]id=21&extern_eid=789

http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1297



Well - i want to fetch all those.... And i try to do it with PHP and a 
mentioned loop. Does this work!?
___________________________________________________________
GRATIS! Movie-FLAT mit Ãber 300 Videos. 
Jetzt freischalten unter http://movieflat.web.de

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php




[Index of Archives]     [PHP Home]     [Apache Users]     [PHP on Windows]     [Kernel Newbies]     [PHP Install]     [PHP Classes]     [Pear]     [Postgresql]     [Postgresql PHP]     [PHP on Windows]     [PHP Database Programming]     [PHP SOAP]

  Powered by Linux