One other thing to keep in mind. Connecting to a database and disconnecting takes some amount of time. If you do it in batches of 10, 100, whatever then make sure you only connect and disconnect once or else you won't have any time savings at all (and will burden the server unnecessarily). Sucks that you don't have more control over the incoming data. Good luck in finding a solution. -TG = = = Original message = = = On Sat, October 22, 2005 5:04 pm, tg-php@xxxxxxxxxxxxxxxxxxxxxx wrote: > Does it make a difference if instead of one record at a time, you pull > 10, 20, 100... might speed up your process a little and if it > doesn't use any more memory, then why not? I could make a buffer of N lines and read them, I guess. I kinda figured at 1 line a second, PHP and the OS would have no problem buffering from the disk. > Another thought.. can you use database replication here? So instead > of dumping all the records and processing and inserting the ones that > are new, you'd only be acting on what's changed since the last backup. > With real replication, you can do this realtime (or at least > semi-realtime), but you could simulate replication in your PHP script > as well. This data's coming from an external source. Replication is probably not gonna happen. Though I am going to ask for a timestamp column so I can ignore rows that haven't changed since my last import. That should kill about 90% of the MySQL processing, which seems to be the biggest CPU sink here. If that happens, I can probably change the sleep to usleep and a quarter second or so. ___________________________________________________________ Sent by ePrompter, the premier email notification software. Free download at http://www.ePrompter.com. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
