On 09/06/07, Angelo Bourik <jusanotherangel@xxxxxxxxx> wrote:
Hello all,
Hi, If I understood your question correctly, you've written a spider script that runs via apache. That script connects to a remote site, fetches a page by id and parses out the data you need, once per request to your own server (or does it run all 100,000 requests per single request to your server?). My first question is, why are you running your script under apache, rather than as a command line process? I don't see any advantage to having apache parent these scripts for you. As regards having more than one script running at the same time, that's perhaps a matter of cpu usage, though it does sound odd that your server is not at least starting to run the others. Perhaps your script opens and locks your database in such a way that the others are blocked? I'd recommend running your spider on the command line, outside of the server. If you really want to speed it up, split the spider in to a fetching thread group that is network bound, and a parsing thread group that munges the queued responses. Or, just write your script in a faster language :-) -- noodl --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|