Re: PHP Data Mining/Data Scraping

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Tijnema wrote:
On 5/22/07, Myron Turner <turnermm02@xxxxxxx> wrote:

> On Sat, May 19, 2007 10:22 pm, Shannon Whitty wrote:
>
>> I'm looking for a piece of software or coding that will let me post a
>> form
>> to another URL, accept the response, search it for a specific
>> "success"
>> string and then let me continue processing the rest of my program.
>>
>
> http://php.net/curl
>
>
>> I want to accept queries on behalf of my supplier, forward it to them
>> behind
>> the scenes, accept their response and display it within my website.
>>
>> Has anyone had any experience with this?  Is there a simple, basic
>> utility
>> to let me do this?
>>
>> I was kind of hoping I could avoid developing it myself.
>>
As I understand this, you want to create a web page of your own which
accepts requests for customers who are going to order products from your
supplier.  You want to have a form on your page which accepts their
requests, then forward the form data on to your supplier's web site,
where presumably it will be processed.  Then you want to retrieve the
response from your supplier's page, and display the result on your own
web page.  You suggest that the response string for "success" is
relatively stable and that this string is this what you want to search
for in the response.

This doesn't sound like a very complicated problem.  You can do this
either using Ajax or not.  The basic solution is the same.  You have a
script on the server which accepts the form data from your page and
re-sends it to the supplier's site.  If your supplier's site accepts
form data using GET, then you can simply create a url with the form data
attached in a query string:

    http://my.supplier.com?fdata_1=data1&fdata_2=data2

Send this url to your suppler using file_get_contents:

         $return_string =
file_get_contents("http://my.supplier.com?fdata_1=data1&fdata_2=data2";);

This will return the html file as a string which you can then parse with
preg_match() for the 'success' string.

The problem is more involved if your supplier doesn't accept GET but
only accepts POST.  Then you  have to use either curl or fsockopen to
post your data.   I've tested the following fockopen script and it
worked for me:

<?php
$fp = fsockopen("my.supplier.com", 80, $errno, $errstr, 30);
if (!$fp) {
   echo "$errstr ($errno)<br />\n";
} else {
   $out = "POST http://my.supplier.com/form_page.html / HTTP/1.1\r\n";
   $out .= "Host: my.supplier.com\r\n";

   $post = "form_data_1=data_1&formdata_2=data_2";
   $len = strlen($post);
   $post .= "\r\n";

   $out .="Content-Length: $len\r\n";
   $out .= "Connection: Close\r\n\r\n";

   $out .= $post;

   fwrite($fp, $out);

   $result= "";
   while (!feof($fp)) {
       $result .=  fgets($fp, 128);
   }
   fclose($fp);
   echo $result;


}
?>

You have to adhere to the above sequence.  The posted data comes last
and it is preceded by a content-length header which tells the receiving
server how long the posted data is.  The returned result is the html
page returned from your posted request.


It's a nice script, but you're way better off using cURL, you can
simply pass a PHP array as POST form data.

Tijnema

Thanks. That's good to know. My experience with these kinds of things is with Perl, with which I've done an awful lot of screen-scraping. But I haven't had any actual practical experience with cURL. I've looked at it, but that's all. Also, I was under the impression that cURL is an extension which is not always installed, whereas fsockopen() is a built-in. There is one server I sometimes use, a shared server, where it isn't installed. The other servers I use are all independent machines.

--

_____________________
Myron Turner
http://www.room535.org
http://www.bstatzero.org
http://www.mturner.org/XML_PullParser/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php


[Index of Archives]     [PHP Home]     [Apache Users]     [PHP on Windows]     [Kernel Newbies]     [PHP Install]     [PHP Classes]     [Pear]     [Postgresql]     [Postgresql PHP]     [PHP on Windows]     [PHP Database Programming]     [PHP SOAP]

  Powered by Linux