I had a spider written in PHP long ago. I had similar problems.
because there were millions of rows of urls and I was fetching them in
one single query. See inline, could this modification help you. Please
test.
On Fri, Oct 28, 2011 at 10:38 PM, Jim Long <php@xxxxxxxxxxxxx> wrote:
>
> I'm running PHP 5.3.8 on FreeBSD 8.2 with MySQL 5.1.55.
>
> The script below is designed to be able to WHILE it's way through
> a MySQL query result set, and process each row.
>
> However, it runs out of memory a little after a quarter million
> rows. The schema fields total to about 200 bytes per row, so
> the row size doesn't seem very large.
>
> Why is this running out of memory?
>
> Thank you!
>
> Jim
>
> <?php
>
> $test_db_host = "localhost";
> $test_db_user = "foo";
> $test_db_pwd = "bar";
> $test_db_name = "farkle";
>
> $db_host = $test_db_host;
> $db_user = $test_db_user;
> $db_name = $test_db_name;
> $db_pwd = $test_db_pwd;
>
> if (!($db_conn = mysql_connect( $db_host, $db_user, $db_pwd )))
> die( "Can't connect to MySQL server\n" );
>
> if (!mysql_select_db( $db_name, $db_conn ))
> die( "Can't connect to database $db_name\n" );
>
$limit=10;
$offset=0;
while(1){
> $qry = "select * from test_table order by contract $offset, $limit";
>
> if ($result = mysql_query( $qry, $db_conn )) {
>
> $n = 0;
> while ($row = mysql_fetch_assoc( $result )) {
> // process row here
> $n++;
> } // while
>
> mysql_free_result($result);
> echo "$n\n";
>
> } else {
>
> die( mysql_error() . "\n" );
// break the loop
break;
>
> }
$offset+=$limit;
}
> ?>
>
>
Its the same thing but you are fetching data in chunks.
Now this portion "order by contract" on quarter million rows is not a
good practice. It will slow down your query time and make the script
severely slow.
I had about 100 millions of rows in my table in the url and I was
sorting on last-visit column. Later I removed the order by and it was
much faster.
Try it and let us know.
Thanks
--
Shiplu Mokadd.im
Follow me, http://twitter.com/shiplu
Innovation distinguishes between follower and leader
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
