On 6/9/07, Angelo Bourik <jusanotherangel@xxxxxxxxx> wrote:
The problem is that I am able to get only two of these scripts running at the same time. So it would still take me at least 5 days to get all the results. The rest of the scripts just sit there in the server's backlog. This is definitely due to Apache's limitation to handle multiple processes. The server I am using to run this script as well the target webserver both run on Apache2. I am sure it's not a problem with the receiving server. It has to be my Apache web server which is running the scripts. I have tried using mpm_winnt (on a windows server) as well as the prefork and worker modules (on a linux server) without any luck. Has any of you ever faced the same situation?
Lots of stuff there that doesn't make any sense to me. What does an apache server on your end have to do with it? You are talking about using a web CLIENT to access some else's server. So why would apache have any role to play on the client end? As for why you can only get two connections at a time, it would not at all be unusual for there to be a two-connection-per-ip limit in a firewall on the receiving end or in a client-side http library.
PS: For those concerned about the legitimacy of this work, rest assured, this is absolutely legit. There's nothing in the website's use policy which restricts somebody from doing this. Moreover, my client hired me to do this only because the website owners were not able to hand over the data he required. They gave the stupid reason that they are helpless in providing the data because they don't have a system in place which would allow them to do a search restriction!
Regardless, unless you have explicit permission to do this kind of thing, many people would find it abusive. You have no idea what kind of load you are going to be putting on their databases, etc. Joshua. --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|