Peter Hoskin wrote:
Jochem Maas wrote:
how much memory do you have set as the max?
512mb
should be enough. :-)
just to give it perspective I have a text file I
import on at least a weekly basis (someone else does it actually but
i wrote the code) that contains 500,000+ lines representing
customers each of which is imported/updated into firebird DB
(mostly not as fast as mysql) using a transaction. the code works and only
chews up about 100Mbs (which is mostly due to the garbage collector not
being able to release memory belonging to objects that have circular
references)
do you have control over this value? (i.e. is it not locked down by an
admin)
yes
what are you doing with each row? (I imagine that effectively you
From each row, I determine what XML files to fetch and parse... the
parser then inserts them into SQL.
This is really the second part of the parsing, the first pass fetches
the concerned table as XML and parses it.
Each XML file I fetch can be between 15-50 rows.
I get the feeling you should be unsetting variables more actively inside
your loops. also avoid objects (specifically/especially ones with circular
'references'** - i.e. parent<-->child type 'references'**)
** - I use 'references' to mean in the general english language
sense not in the php sense. although if you're using php4 then most probably
your object variables should actually be php-references!
only need on row in memory at a time - in which case the memory
requirements are
probably quite conservative)
show us your code if you can :-)
It came down to my code I think... wasn't using it appropriately. I read
the post 'pcntl functions and database' from this list a few days ago,
and it had a good example... so far I've come up with:
<?
require_once('../private/sql.php');
$pgconnectstring='dbname=' . returndbname() . ' user=' . returndbuser()
. ' password=' . returndbpass() . ' host=redback.10mbit.biz';
$database = pg_connect($pgconnectstring);
$query = "SELECT count(key) AS count FROM tvguide_channels WHERE
crawl=true;";
$result = pg_query($database,$query);
$row = pg_fetch_object($result,0);
$childcount = 0;
$childrencount = $row->count;
while($childcount < $childrencount) {
$pid = pcntl_fork();
if ( $pid == -1 ) {
die("Unable to fork\n");
} elseif ($pid == 0) {
$childcount++;
pcntl_wait($status);
continue;
} else {
echo $childcount ." ". posix_getpid() ." ". posix_getppid() ."\n";
exit();
}
}
?>
Each fork seems to be able to see the value of $childcount... so I just
need to create a function to work with a particular row. Having a
counter is precisely what I needed as I can then use
pg_fetch_object($result,$childcount)
moving forward, always nice to make progress :-)
Regards,
Peter Hoskin
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
