On Mon, 2010-04-26 at 12:05 +0900, ioannes@xxxxxxxxxxxxxx wrote:
> >>
> >> Just to eliminate all possibilities, are you to open the same URL/URI in
> > the
> >> web pages repeatedly? Also, what happens when you fake the user agent in
> >> the web browser? The target site may have some anti bot mechanism in
> >> place to reduce stress/load on the server(s).
> >>
> >> Regards,
> >> Tommy
> >
> > One more thing, check it with cookies enabled/disabled in the web browser
> > too.
> >
> >
>
> Having deleted cookies on the browser and disabled them, it still does
> not like various user agents:
>
> $useragent = array('Mozilla','Opera','Microsoft Internet
> Explorer','ia_archiver');
> $os = array('Windows','Windows XP','Linux','Windows NT','Windows
> 2000','OSX');
> //random user agent code
> $agent = $useragent[rand(0,3)].'/'.rand(1,8).'.'.rand(0,9).'
> ('.$os[rand(0,5)].' '.rand(1,7).'.'.rand(0,9).'; en-US;)';
> //would give something like Mozilla/3.5 (Windows 5.4; en-US;)
>
> -- OR --
>
> //$useragent='Google Image - Googlebot-Image/1.0 (
> http://www.googlebot.com/bot.html)';
> //$useragent="MSN Live - msnbot-Products/1.0
> (+http://search.msn.com/msnbot.htm)";
>
> -- OR --
> //$agent = "DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0)
> Gecko/20020804";
>
> I am just calling the page manually, once at a time. It is probable
> that there is some anti-bot measures. Page would probably not want to
> be indexed as it is providing ever changing content. How to use this
> for normal level of use for real user just in a different site?
>
> John
>
How frequently do you request the page? Maybe playing about with that
would resolve it? Is it possible to randomise the request frequency a
bit?
Thanks,
Ash
http://www.ashleysheridan.co.uk
