----- Original Message -----
From: "flint" <fmillion@xxxxxxxxx>
To: "דניאל דנון" <danondaniel@xxxxxxxxx>
Sent: Sunday, May 31, 2009 9:21 AM
Subject: Re: mysql_query takes long time...
have you actually tried running the script against your data?
i'm running MySQL/PHP on an old P3/933Mhz box with only 384M memory... I
had a situation just like yours. I had a text file of 600,000 lines, the
script reads a line, does regex matches/replacements on it, and inserts to
SQL. it took the script about 15 mins to process the entire file and put
all the data into MySQL.
and remember that's on a P3/933 so a faster system will obviously speed
that up significantly
don't worry about the time MySQL reports for an individual query. it is
dependent on many factors. but once you open your SQL connection and start
streaming data in it will actually go much faster than that. just run the
script and give it a try... even add some debugging output, somethign like
this maybe:
$counter = 0;
while (!feof($fhandle)) {
$line = fgets($fhandle);
process_line($line);
insert_into_sql($line);
$counter++;
if ($counter % 100 = 0) { echo $counter . " records done\n"; }
}
that will give you a printout every 100 records processed... a nice way to
track progress.
FM
----- Original Message -----
From: "דניאל דנון" <danondaniel@xxxxxxxxx>
To: "PHP General List" <php-general@xxxxxxxxxxxxx>
Sent: Sunday, May 31, 2009 7:18 AM
Subject: mysql_query takes long time...
I've a file of about 500,000 lines, each line contains a string in
variety
of lengths, but no less then 3 characters and usually no more then....
120.
average of about 80, and maximum of about 250.
I made a PHP script to fetch the data (using fgets), process it and
insert
it to a MySQL database.
The problem is inserting to MySQL takes about 0.02 seconds, which looks
like
nothing - but when you have 500,000 lines to insert...
The while goes like that:
fgets from file
x1 = some function about the string
x2 = some other function about the string
x3 = the string
insert into table (field1, field2, field3) VALUES (x1, x2, x3)
(pseudo-code)
I was wondering, is there any faster way to perform it, assuming I have
to
do it with PHP?
also, if it matters - the MySQL table got id in auto increment.
Yours, Daniel.
--
Use ROT26 for best security
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
