Jeff Trawick wrote:
On Sat, Jul 12, 2014 at 1:25 PM, Miles Fidelman <mfidelman@xxxxxxxxxxxxxxxx <mailto:mfidelman@xxxxxxxxxxxxxxxx>> wrote:Hi Folks, Ever once in a while, a crawler comes along and starts indexing our site - and in the process pushes our server's load average through the roof. Short of blocking the crawlers, can anybody suggest some quick tuning adjustments to make, to reduce load (setting the max. number of servers and/or requests, renicing processes)?Use robots.txt to block access to dynamically generated resources which areexpensive to generate and not necessary for search hits? Is it using a lot of concurrent requests, or is the main load issue due to the cost of the requests it is making?
a bit of both -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|